Eclipse/Testing/JUnit4 Changes

Introduction

JUnit version 4.0 was released in March 2006, and further maintenance on JUnit 3.x stopped around the same time with the 3.8.2 release. Due to its dependency on Java SE 5, JUnit 4 was not adopted by the platform test framework, but there has been strong community interest in running the Eclipse test framework with JUnit4. This page describes the enhancement of the Eclipse Test Framework (ETF) to support JUnit version 4.0. This change is being made in the Helios M4 milestone build.

This will be a transparent change for many users of the test framework, but some tests may be affected by this and will need to react accordingly. This document outlines the steps required to either transition to JUnit4 or to continue using JUnit3 with the Eclipse Test Framework.

The org.apache.ant bundle contains the ant.jar as well as jars for optional tasks, like JUnit. Unfortunately, the manifest file for this bundle doesn't have JUnit and other in its dependency list. To enable the Ant JUnit tasks to work, the Eclipse Test Framework adds a fragment to org.apache.ant which adds the JUnit bundle to the classpath of the Ant bundle. However, since the fragment will only be bound to a single version of JUnit by OSGi, the test framework can only run one version of JUnit for any given invocation of the test framework. As a result of this limitation, clients must select either JUnit3 or JUnit4 to run their tests against.

Clients of the ETF that wish to transition to using JUnit4 may require some migration in their tests due to the change in version number, and some minor incompatibilities between versions of JUnit. Clients that wish to remain on JUnit3 will need to either remove JUnit4 from their test environment, or run with JDK 1.4 which will cause JUnit4 to be ignored by the OSGi runtime.

Test framework changes

JUnit4 Bundle Rename

Because we wanted to retain backwards compatibility of the Framework, having the org.eclipse.test bundle depend on org.junit4 simply wasn't an option. This would mean that users running with a JVM <1.5 would not be able to resolve the org.eclipse.test bundle. So it was decided that by renaming the JUnit4 bundle to "org.junit" (same as the JUnit3 bundle name), the EclipseTestRunner (ETR) could just run with the highest version of org.junit that gets resolved.

org.junit4 is now "empty", and simply re-exports the packages exported by the org.junit version 4.x bundle.

Widened test framework version ranges

Test framework bundles have had their dependency on JUnit widened to include both JUnit3 and JUnit4. This will allow the framework to run with either version of JUnit (but not both at once).

Making org.junit a singleton

Having multiple versions of JUnit running in the same framework instance can easily result in problems when some bundles resolve against JUnit3 and others resolve against JUnit4. We are considering making org.junit a singleton to avoid this kind of runtime error from happening. See bug 296104 for more details and discussion on this proposed change. This change isn't strictly required for the test framework to add support for JUnit4.

Update: It was decided not to make these bundles singleton, as it could potentially cause other issues in future, as discussed in in bug 296104.

Moving to JUnit4

This section describes steps required by clients who want to switch to using JUnit4 in their tests.

Some test bundles will run just fine with either JUnit3 or JUnit4. However in some cases tests have created dependencies on one version of JUnit or another (either deliberately or not). Here are some of the kinds of changes that tests currently running on JUnit3 may have to make in order to run on JUnit4. Feel free to add entries to this section if you encounter other issues when migrating from JUnit3 to JUnit4.

Version ranges

Some test bundles express a narrow version range on org.junit 3.x, such as "[3.8.0,4.0.0)". Such test bundles won't resolve against JUnit4, and hence require updating the version range. Since JUnit is not an eclipse.org project and doesn't necessarily follow our version evolution semantics, the safest approach is to only specify the lower bound dependency. If you still want to be able to run with JUnit3, use a range of "3.8.0". If you only want to run on JUnit4, you can use a range of "4.8.1".

Dependencies on the shape of the JDT feature

Some tests make assumptions about the contents of the JDT feature - the number of bundles, the version number of the bundle "org.junit", etc. Such tests need to be updated due to the changes in the JDT feature (new bundle "org.junit" with a 4.x version). Typically these tests are in PDE and JDT and are unlikely to be an issue with tests outside the Eclipse SDK.

Custom subclasses of junit.framework.TestSuite

Some of the test runners in JUnit4 no longer call TestSuite.tests() to obtain the list of tests. If you have a subclass of TestSuite that overrides the tests() method, you will likely be broken by the move to JUnit4. The test runner in JUnit4 instead calls TestSuite.run(TestResult). If you were previously overriding TestSuite.tests(), you will now also need to override TestSuite.run(TestResult) to make sure your extra tests are called.

Custom subclasses of junit.framework.TestResult

The JUnit class TestResult changed several of its protected fields from Vector to List. This is a breaking change for any subclass of TestResult that accessed those fields. Such classes will need to update from calling Vector methods to calling List methods. Luckily since Vector implements List, after you make this change you will be able to run on both JUnit3 and JUnit4.

JUnit-4-style test suites

If you start converting your JUnit-3-style test suites with a suite() method into JUnit-4-style @RunWith(Suite.class) classes, you may run into issue 1189 if one of the included suite classes is a JUnit-3-style class with a suite() method.

The problem is that e.g. the EclipseTestRunner wraps tests in a JUnit4TestAdapter to run them as JUnit-3-style tests. To work around the cited bug, add the @RunWith(org.junit.runners.AllTests.class) annotation to the class that declares the suite() method.

Remaining on JUnit3

Clients that wish to remain with JUnit3 for their tests may do so. However, such clients will need to ensure JUnit4 is not present in their test environment while their tests are running. There are three options for continuing to run tests with JUnit 3:

1. Run with a version of the Eclipse test framework from Helios M3 or earlier. Earlier releases of the test framework supported only JUnit3 so clients can continue to use these older versions. This should continue to work for testing throughout the Helios release cycle, but old versions of the test framework may not continue to work with future releases of the platform.
2. Run tests with JDK 1.4. This will cause the JUnit4 bundle to be ignored by OSGi and all bundles will be bound to JUnit3.
3. Remove the org.junit version 4.x bundle from your test environment completely. This will cause all bundles to resolve and run against JUnit3.

Clients wishing to continue using JUnit3 should also restrict their org.junit version range to exclude JUnit 4.0.0, to avoid accidentally referencing JUnit4 API in their tests.

References

• Complete details on the move to JUnit 4 can be found in bug 153429.
• JUnit releases