4.5 Test Automation Initiative

Members

• Paul Slauenwhite (IBM)
• Alan Haggarty (IBM)
• Joel Cayne (IBM)
• Jonathan West (IBM)
• Joanna Kubasta (IBM)
• Kiryl Kazakevich (Intel)

Goals

• Consolidation of our testing process and infrastructure.
• Consolidation of custom test frameworks (Test Dashboard, AC/Profiler test server, variants).
• Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.).
• Automatic test execution and report generation with every build (BVT).
• Integrate EMMA with the build and test process for generating code coverage reports for all manual and automated testing.

Benefits

• Specialization of testing skills (e.g. builds).
• Provide an infrastructure that makes it easy for committers to contribute automated test cases/suites.
• Manual labor savings by using automated tests, which are automatically executed with every build.
• Decrease time/cost to resolve defects since uncovered earlier in the code-test-build cycle.
• Decrease testing overlap by a) localizing test executing and reporting to one group and b) using code coverage statistics.

Delivery dates

• November 14:
  • First draft of the first testing process document for review.
  • Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds.
  • Status update for the PMC.
  • Measure cost savings.

• November 21:
  • Review of the first testing process document.
  • First draft of the remaining testing process documents for review
  • Identify and resolve bugs.

• November 28:
  • Final version of the the first testing process document.
  • Review of the remaining testing process documents for review
  • Status update including cost savings for the PMC.
  • Presentation to TPTP committers.
  • Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.

Testing Process

• Summary:
  • A very lightweight testing process for TPTP.
  • Extends the existing TPTP Testing Strategy. The existing technical content will remain unchanged, aside from review changes.
  • Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
  • Sections include
    • Getting tests.
    • Creating tests.
    • Running tests.
    • Save test results.
    • Reporting results.
    • BVTs.
  • Two (or more) documents:

1. High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s).
2. Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
3. Additional documents detailing the specifics of a testing topics.

• Draft Documents:
  • Part 1:
    • Draft 1: Comments
    • Draft 2: Comments
  • Part 2:
    • Draft 1: Comments

• Final Documents:
  • Part 1:
    • Final
  • Part 2:
    • Final

Common Test Infrastructure

• Summary:
  • Reference platform:
    • Developers are expected to unit test on reference platform before checking-in code to CVS.
    • Target: IBM Java 1.5 (latest SR) and Windows XP/x86.
    • Goal to expand to a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
    • IBM JREs for Windows and Linux (see IBM Development Package for Eclipse) are publicly available from developerWorks.
    • Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.
  • ASF integrated with the TPTP builds by invoking the automated tests on reference platform via ANT:
    • Invoke tests.
    • Interrogate results.
    • Generate test reports.
  • TPTP test types:
    • AGR for functional UI testing.
    • TPTP JUnit and TPTP JUnit Plug-in test for unit testing.
  • TPTP build infrastructure:
    • Extract test suites from CVS.
    • Provision reference platform (Agent Controller and Eclipse/TPTP).
    • Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
    • Clean-up reference platform.
    • Post test reports (test and code coverage) on TPTP web site.
    • Email notifications to component leads for failing test suites.
  • Limitations/Issues:
    • Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
    • Intel needs to inform IBM that their portion of the build is complete (see defect 200351).

• Architecture: Comments

• Demonstration:
  • Viewlet.

Metrics

In order to evaluate this initiative, are measuring the labor costs (as accurately as possible) in Person Weeks (PW) to run:

1. full test pass
2. smoke test pass

• Before Initiative:

Project	Full Test Pass Cost (PW)	Smoke Test Pass Cost (PW)
Platform	5	2.5
Test	5	2.5
Trace	3	1.5
Monitoring	3	1.5
Total	16	8

• With Test Automation:

Project	Full Test Pass Cost (PW)	Smoke Test Pass Cost (PW)
Platform	4	2
Test	4	1.5
Trace	3	1.5
Monitoring	2.5	1
Total	13.5	6

References

• ASF
• ASF Tutorial
• TPTP Testing Strategy
• Test Dashboard (see enhancement 112928)
• Test Dashboard Plug-in

Meeting Minutes

• September 24, 2007
• October 22, 2007
• November 6, 2007
• November 12, 2007
• November 19, 2007
• November 26, 2007
• November 29, 2007
• December 3, 2007