How to test TPTP with line level code coverage enabled

Background

EMMA is an open-source toolkit written in Java for measuring and reporting line level code coverage statistics for Java applications.

TPTP is currently conducting a Proof of Concept (PoC) to evaluate EMMA for generating and reporting line level code coverage statistics when testing TPTP.

Download

The PoC requires EMMA v2.0.5312.

Instrumentation

To generate line level code coverage statistics, the TPTP Software Development Kit (SDK) must be first statically instrumented using EMMA. The Release Engineering Team has integrated this instrumentation with the TPTP build process to generate a statically instrumented TPTP SDK ZIP for each candidate build. The instrumented TPTP SDK ZIP is available from the TPTP download site (see the 'EMMA Instrumented' link for the candidate build).

In addition, you can manually generate statically instrumented TPTP plug-ins:

• JARed plug-ins:

   java -cp emma.jar emma instr -instrpath org.eclipse.tptp.*.jar -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*

• JARs in plug-ins:

   java -cp %EMMA_JAR% emma instr -instrpath org.eclipse.tptp.*\*.jar  -outfile EMMA_Coverage.metadata -merge y -outmode overwrite -filter +org.eclipse.hyades.*,+org.eclipse.tptp.*

where

• org.eclipse.tptp.*.jar: One or more TPTP JARed plug-in paths.
• org.eclipse.tptp.*\*.jar: One or more TPTP JAR paths.
• EMMA_Coverage.metadata: EMMA metadata file path.

When manually generating statically instrumented TPTP plug-ins, an EMMA metadata file (EMMA_Coverage.metadata) is generated. This file describes the instruments classes, used when generating EMMA coverage reports. The EMMA metadata file for an instrumented TPTP SDK ZIP is available from the TPTP download site (see the 'EMMA Metadata' link for the candidate build).

For more information, see the EMMA instructions.

Running

Before running the statically instrumented TPTP SDK, the EMMA JAR (emma.jar) must be added to the default environment of both the Agent Controller and Integrated Agent Controller. This is easily done by adding a <Variable> to the <AgentControllerEnvironment> in <Agent Controller installation directory>\config\serviceconfig.xml (Agent Controller) and <Eclipse installation directory>\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_<build ID>\agent_controller\config\serviceconfig.xml (Integrated Agent Controller). For example:

   <Variable name="CLASSPATH" position="append" value="emma.jar"/>

where

• emma.jar: EMMA JAR file path.

When running the statically instrumented TPTP SDK, the EMMA JAR (emma.jar) must be added to the runtime classpath of each plug-in. This is easily done by starting Eclipse in development mode (-dev). For example:

   eclipse -dev emma.jar

where

• emma.jar: EMMA JAR file path.

After running the statically instrumented TPTP SDK and/or Integrated Agent Controller, one or more EMMA coverage files (coverage.ec) are generated. These files contain the raw line level code coverage statistics used when generating EMMA coverage reports. The default location of these EMMA coverage files is:

• Eclipse: Eclipse home directory (ECLIPSE_HOME).
• Integrated Agent Controller: User's temporary directory (user.dir JVM system property).

Note: Currently, the Agent Controller is not statically instrumented so it does not generate a EMMA coverage file.

Each time Eclipse and/or the Integrated Agent Controller is started, the line level code coverage statistics for that instance are appended to the associated EMMA coverage file.

Tip: Delete all of the EMMA coverage files before starting a new testing session.

To override the default location of the EMMA coverage file generated by Eclipse, Eclipse can be started with the following environment variable:

• emma.coverage.out.file=EMMA_Coverage.data

where

• EMMA_Coverage.data: EMMA coverage file path.

This environment variable can be passed to the Eclipse executable as VM argument (-vmargs), using the -D command line option. For example:

   eclipse -dev emma.jar -vmargs -Demma.coverage.out.file=EMMA_Coverage.data

At the end of each test pass, the EMMA coverage file(s) are emailed to Paul Slauenwhite to be merged using the EMMA aggregator and reported for that test pass.

Reporting

To view the line level code coverage statistics, the EMMA metadata file and EMMA coverage file(s) created during instrumenting and running Eclipse and/or the Integrated Agent Controller are merged and reported using EMMA.

The Test Tools Project/Release Engineering Team generates EMMA coverage reports during each test pass. The EMMA coverage reports are available from the TPTP test report site for the specific release. For example:

TPTP 4.4.0.3 EMMA Coverage Report

Note: The line level code coverage statistics in this report are generated from executing only a fraction of the test cases for the TPTP 4.4.0.3 smoke test pass.

To manually generate a EMMA coverage report:

   java -cp emma.jar emma report -input EMMA_Coverage.metadata,coverage.ec -report html -Dreport.out.file=EMMA_Coverage_Report.html

where

• EMMA_Coverage.metadata: One or more EMMA metadata file paths.
• coverage.ec: One or more EMMA coverage file paths.
• EMMA_Coverage_Report.html: EMMA coverage report (HTML) file path.

For more information, see the EMMA instructions.

Issues

1. When running automated tests (AGR/JUnit/JUnit Plug-in tests) with EMMA, test execution hangs. The test runner code needs to be modified to allow testers to add the EMMA JAR to the class path and the emma.rt.control=false system property to the session JVM.
2. The Agent Controller is not statically instrumented so it does not generate a EMMA coverage file.
3. Usability since the tester has to add the EMMA JAR to the Eclipse classpath and locate/retrieve the generated code coverage files (see Running).

Demo

To demo this PoC, we created a Windows batch script that instruments the TPTP plug-ins (org.eclipse.tptp.* and org.eclipse.hyades.* plug-ins), starts Eclipse with the necessary configuration, and generates a coverage report.

Instructions:

• Save the Windows batch script in the Eclipse home directory (ECLIPSE_HOME).
• Save the EMMA JAR (emma.jar) in the Eclipse home directory (ECLIPSE_HOME).
• Run the Windows batch script.

After execution, the <Eclipse home directory>/emma<code> directory contains:

• EMMA metadata file (EMMA_Coverage.metadata).
• EMMA coverage file (EMMA_Coverage.data).
• EMMA coverage report (EMMA_Coverage_Report.html).
• Instrumentation log (EMMA_Instrumentation.log).
• Reporting log (EMMA_Report.log).