Tech Blog

For most cases I prefer JUnit over TestNG, the fact that it the de-facto standard for unit testing make integration with eclipse, CI servers and build systems easier. However, TestNG has its merits, it is more flexible than JUnit, grouping is highly configurable and the DataProvider feature is also good.

Cobertura is one of the key static analysis tool every developer should work with. I’ve seen numerous project increase their code coverage using this tool and which in turn can and will reduce QA cycles and overall bug counts any given system.

I recently had to integrate Cobertura code coverage with TestNG into the build system and CI server and found very little on the net on how to do it. EclEmma has a great coverage plugin for eclipse which is a must for every developer writing tests (a small nit that I had with TestNG and this plugin is that you have to run the tests first using the TestNG plugin and only then I was successful in running them again in coverage mode). But the plugin is not enough, it is also important to run coverage reports on a nightly build and continuously monitor the coverage percentage of your application as the project evolves.

In order to run the Cobertura coverage you need to:

1. instrument your classes
2. execute your tests with the instrumented classes
3. use the cobertura generated execution file created at step 2 to generate an html report.

For keeping things DRY, you’re better off not copying your test ant targets but reuse the existing ones. In order to do this let’s make the classpathref attribute of the TestNG task parametric (I used a var so I can override it from other builds, you need antcontrib to use var).

   <property name="cb.file" location="${basedir}/cobertura.bin" />
   <var name="cb.arg" value="none" />

   <!-- normally this would contain your standard test classpath -->
   <var name="test.cp" value="test.cp" />
   <target name="test">
      <testng classpathref="${test.cp}"
              haltOnFailure="true">
         <xmlfileset dir="${basedir}/src/test" includes="tests.xml" />
         <sysproperty key="${cb.arg}" value="${cb.file}" />
      </testng>
   </target>

The remainder of the build file looks roughly something like this:

   <!-- define the cobertura jars classpath -->
   <path id="cb.cp">
      <!--
		Includes the following jars:
		asm-all 3.2
		cobertura 1.9.4.1
		log4j 1.2.14
		oro 2.0.8
		-->
   </path>

   <target name="cb.init">
      <!-- output directory -->
      <property name="cb.report.dir" value="${basedir}/cobertura" />
      <!-- instrumentation directory -->
      <property name="cb.instrument.dir"
                value="${cb.report.dir}/.cobertura-instrumented-classes" />

      <!-- your main java source files -->
      <property name="cb.src.dir" value="${basedir}/src/main" />
      <!-- your main class files files -->
      <property name="cb.classes.dir" value="${basedir}/bin/main" />
      <!-- your test class files -->
      <property name="cb.test.classes.dir" value="${basedir}/bin/test" />

      <taskdef classpathref="cb.cp" resource="tasks.properties" />
   </target>

   <target name="cb" depends="cb.gen, cb.xml, cb.html">
      <delete file="${cb.file}" />
      <delete dir="${cb.instrument.dir}" />
   </target>

   <target name="cb.instrument">
      <mkdir dir="${cb.report.dir}" />
      <cobertura-instrument todir="${cb.instrument.dir}/"
                            maxmemory="512M"
                            datafile="${cb.file}">
         <fileset dir="${cb.classes.dir}">
            <include name="**/*.class" />
         </fileset>
      </cobertura-instrument>
   </target>

   <target name="init.cb.path">
      <var name="test.cp" value="cb.test.cp" />
      <var name="cb.sysarg" value="net.sourceforge.cobertura.datafile" />
      <path id="cb.test.cp">
         <pathelement path="${cb.instrument.dir}" />
         <pathelement path="${cb.classes.dir}" />
         <pathelement path="${cb.test.classes.dir}" />
         <path refid="test.cp" />
         <path refid="cb.cp" />
      </path>
   </target>
   <target name="cb.run" depends="init.cb.path, test" />
   <target name="cb.gen" depends="cb.init, cb.instrument, cb.run" />
   <target name="cb.html">
      <cobertura-report format="html"
                        destdir="${cb.report.dir}"
                        maxmemory="512M"
                        datafile="${cb.file}">
         <fileset dir="${cb.src.dir}">
            <include name="**/*.java" />
         </fileset>
      </cobertura-report>
   </target>

   <target name="cb.xml">
      <cobertura-report format="xml"
                        destdir="${cb.report.dir}"
                        maxmemory="512M"
                        datafile="${cb.file}">
         <fileset dir="${cb.src.dir}">
            <include name="**/*.java" />
         </fileset>
      </cobertura-report>
   </target>

Make sure you remember to initialize your test classpath properly as part of the Cobertura flow.

Once you get this working (had to overcome a couple of obstacles with Cobertura report not finding the datafile and tweak the classpath a bit to get it to work) it is really easy to connect this to your CI build. If you’re using Hudson then download the Cobertura Plugin and point it to all of your projects output cobertura.xml files. These CI servers are great in aggregating these reports and providing a time-lapse view of your code coverage: