Consolidation of our testing process, and infrastructure.
Consolidation of custom test frameworks (Test Dashboard, AC/Profiler test server, variants).
Reuse of Test Project framework (ASF, reporting, etc.) and tools (AGR, JUnit, etc.).
Automatic test execution and report generation with every build (BVT).
Benefits
Specialization of testing skills (e.g. builds).
Provide an infrastructure that makes it easy for committers to contribute automated test cases/suites.
Manual labor savings by using automated tests, which are automatically executed with every build.
Decrease time/cost to resolve defects since uncovered earlier in the code-test-build cycle.
Decrease testing overlap by a) localizing test executing and reporting to one group and b) using code coverage statistics.
Reference Platform
Target: IBM Java 5* and XP/x86.
Developers are expected to unit test on reference platform before check-in.
Nightly builds invoke automated test execution on reference platform.
Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
Move to Java 6 once a complete test pass is completed to expose initial defects.
Deliver dates
October 17:
Define the testing process, extending the existing TPTP Testing Strategy.
Define the common test infrastructure and its constraints.
Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval.
Profiler test suite automation.
Determine the machine for the target platform.
October 31:
Fully integrate PoC with TPTP builds.
Resolve infrastructure limitations.
Presentation to TPTP committers.
Measure cost savings.
November 14:
Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.
Measure cost savings.
Report cost savings to PMC.
Action Items
Alan: Review and revamp the existing TPTP Testing Strategy as the new test process.
Jonathan/Kendric/Paul: Port existing profiler automated tests to used the Test Project framework.
Joel: Locate a dedicate test machine for the target platform.
Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard..
Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
Paul: Encourage Project Leads to automate existing manual test suites.
Paul: Book another meeting in two weeks (October 15).
