Performance Test Fingerprint Guidelines

This page describes the guidelines for tests that show up in the global or component performance fingerprint (aka summary). The fingerprints have to be targeted at users that want to get a quick overview of what became faster or slower when using Eclipse. We will have to improve the support for developers, e.g. allow to get the bar charts for tests that aren't in the fingerprint but that's outside the 3.3 scope.

Component Fingerprint

1. the test must cover a common scenario that the user sees/experiences e.g. startup, full build, opening a concrete wizard or part that exists in the workbench, perspective switch, etc.
2. the test must be relevant for that component i.e. if it turns red you must not say, e.g. "the test is not really significant because..."
3. the test should at least take 100ms but longer is of course better
4. the test must be stable i.e. don't take one that looks like a sawtooth
5. give the test a name that fits the scenario and mention the amount of iterations e.g. use "Activate 30 Java Editors" instead of just "Activating an Editor"
6. at the end of a release a degraded test must not be red but gray with an explanation why it degraded
7. don't put more than 20 tests into the fingerprint as it should give an overview that shows the big picture

Global Fingerprints

Each component contributes up to 2 most common component fingerprints to the global list.