• Ma├»kel Vandorpe

  • Functional Lead

Recently, I've talked about our test-setup. What I didn't mention was that we also check our code coverage to see how well our unit tests are covering the production code. This way, we can easily see how well we are testing  what we are delivering. There are plenty of tools around the block to do this. All these tools have unit tests in mind. What they do is run all the unit tests and see for each module and each and every method and even every code unit within every method if the unit tests get there. And so, after running all these tests, it is easy to see how much code is covered by all the unit tests.

You could easily do the same with acceptance tests, you'd think. See how much code you're covering when you run all the acceptance tests. Only these quality management tools are not built for acceptance tests but for unit tests. So we were curious ...

Our Experiment

 In a Java project, where we used JUnit, Cucumber Web Bridge, Gradle, SonarQube and Atlassian ...

  • Unit tests written in JUnit
  • Functional tests written with Cucumber Web Bridge
  • Atlassian Suite as continuous integration platform. Bamboo being the most important in this story.

In short, we’ve used jacoco to merge the JUnit coverage and the Cucumber Web Bridge coverage in SonarQube.

Here’s how exactly:

  • Run the JUnit tests with a Jacoco destination
  • Generate a Jacoco file when running acceptance tests
  • Merge the 2 Jacoco files
  • Kick off SonarQube with the merged Jacoco files

1.     Run the JUnit tests with a jacoco destination

Make sure you run the JUnit tests with a Jacoco destination in build.gradle. We use "fronted-site" to represent the site where we want JUnit coverage and cucumber coverage to be merged.

Apply plugin: 'jacoco' 

test() { 

   finalizedBy 'copyTestJacoco' 

   jacoco { 

        destinationFile = file("$buildDir/jacoco/jacoco.exec") 

   } 

} 



task copyTestJacoco() << {

    copy {

        from "frontend-site/build/jacoco/"

        into "build/jacoco/"

        include "jacoco.exec"

        rename "jacoco.exec", "jacoco.junit.exec"

    }

}

This will define a jacoco-file used for unit tests and at the end of this execution, a backup copy is put in the general "build/jacoco" dir for later use.

2.     Generate a jacoco file when running acceptance tests

Run the Cucumber scenarios inside an embedded Tomcat server. We use the Tomcat Gradle plugin from https://github.com/bmuschko/gradle-tomcat-plugin. You can find relevant information for the setup and config there.

Following is the code block we use to execute the in Bamboo using GRADLE_OPTS="-Xmx1024m -XX:MaxPermSize=256m -javaagent:jacoco/jacocoagent.jar=destfile=build/jacoco/jacoco.cucumer.exec,includes=net.packages.*" gradlew acceptanceTest

task acceptanceTest() { 

    dependsOn assemble, testClasses, 'integrationTomcatRun' 

    finalizedBy 'integrationTomcatStop' 

    doFirst { 

        if (!System.properties['environment.type']) { 

            throw new GradleException('You must specify environment.type system property: -Denvironment.type=<ENV>') 

        } 

    }

    doLast {

        javaexec {

            systemProperties = ['environment': System.properties['environment.type']]

            main = 'cucumber.api.cli.Main'

            classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output

            args = ['--format', 'com.foreach.cuke.core.formatter.ConsoleReporter',

                    '--format', "junit:${project.buildDir}/test-results/cucumber-report.xml",

                    '--format', "json:${project.buildDir}/test-results/cucumber-report.json",

                    '--glue', 'com.foreach.cuke',

                    '--tags', '~@ignore',

                    'src/test/resources/features']

        }

    }

}

Note how we pass -javaagent to gradle, which will instrument all classes run by the cucumber scenario. You may need to change the "includes" to match your packages from the "frontend-module".

The jacoco file will be available in build/jacoco/jacoco.cucumer.exec after the Tomcat JVM has shutdown.

3.     Merge the 2 jacoco files

Now, we have two jacoco files inside "build/jacoco": jacoco.cucumber.exec and jacoco.junit.exec. SonarQube will only handle one file, so we will merge the two files and drop them in the "frontend-module" directory for SonarQube to pick up before it runs. 

apply plugin: 'sonar-runner'

task jacocoMergeFiles(type:JacocoMerge) {

    def executionFiles = [ file( 'build/jacoco/jacoco.cucumber.exec' ), file( 'build/jacoco/jacoco.junit.exec' ) ]

    executionData( executionFiles )

    destinationFile file('frontend-moduyle/build/jacoco/jacoco.exec')

    jacocoClasspath = files( "jacoco/org.jacoco.core-0.7.1.201405082137.jar", "jacoco/org.jacoco.ant-0.7.1.201405082137.jar" )

}

sonarRunner {

    sonarProperties {

        property "sonar.host.url", "http://sonar.website.com/"

        property "sonar.login", "login"

        property "sonar.password", "password"

        property "sonar.jdbc.url", "jdbc:jtds:sqlserver://localhost:1431;databaseName=Sonar"

        property "sonar.jdbc.driverClassName", "net.sourceforge.jtds.jdbc.Driver"

        property "sonar.jdbc.username", "username"

        property "sonar.jdbc.password", "password"

        property "sonar.jacoco.reportPath", "build/jacoco/jacoco.merged.exec"

        property "sonar.language", "java"

    }

}

tasks['sonarRunner'].dependsOn jacocoMergeFiles

4.     Run SonarQube

Finally when sonar runs, the coverage of both test instances will be included in SonarQube. 

Note that the executions for the Junit tests, Cucumber tests and Sonarrunner happen in seperate JVMs. This is done so we can collect the jacoco file from Tomcat after the JVM shuts down.

The command lines  used in the Bamboo job are:

$ gradlew clean

$ gradlew test

$ GRADLE_OPTS="-Xmx1024m -XX:MaxPermSize=256m -javaagent:jacoco/jacocoagent.jar=destfile=build/jacoco/jacoco.cucumber.exec,

includes=net.packages.*" gradlew acceptanceTest -D environment.type=mem

$ gradlew sonarRunner

Clean is only used once at the start from the build and in a clean build directory.

Conclusion

It can be done. Now let's read those SonarQube reports and see what interesting facts we can deduct from all of this.

Verwante Artikels