TPTP-Test-Automation-20071106

Attendees

• Present:
  • Paul
  • Joanna
  • Kiryl
  • Alan
  • Joel

• Absent:
  • Jonathan

Testing Process

• • A very lightweight testing process for TPTP.
  • Extends the existing TPTP Testing Strategy.
  • Possibly two documents:
    • 1) High steps to use the testing infrastructure intended for first time users. This document will reference the second document.
    • 2) Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.

Reference Platform

• Target: IBM Java 1.5 (latest SR) and XP/x86.
• Developers are expected to unit test on reference platform before check-in.
• Nightly builds invoke automated test execution on reference platform.
• Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).
• Move to Java 6 once a complete test pass is completed to expose initial defects.

• Update:
  • The dedicate test machine for the reference platform has been clean and does not need to be re-imaged.
  • IBM JREs for Windows and Linux (see IBM Development Package for Eclipse) are publicly available from developerWorks.
  • Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.

Common Test Infrastructure

• ASF integrated with the TPTP builds (invoked via ANT):
  • Invoke tests.
  • Interrogate results.
  • Generate test reports.

• TPTP test types:
  • AGR for functional UI testing.
  • TPTP JUnit and TPTP JUnit Plug-in test for unit testing.

• TPTP build infrastructure:
  • Extract test suites from CVS.
  • Provision reference platform (Agent Controller and Eclipse/TPTP).
  • Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
  • Clean-up reference platform.
  • Post test reports (test and code coverage) on TPTP web site.

• Update:
  • Joel and Paul discussed the detailed design of this infrastructure last week.
    • Joel will attach a diagram of this infrastructure to the WIKI.
  • Joel will document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
  • This infrastructure will not allow testing with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
  • Intel will inform IBM that the Intel portion of the build is complete (see defect https://bugs.eclipse.org/bugs/show_bug.cgi?id=200351 200351]).

Deliverables

• Alan:
  • No update.
  • Waiting for the PoC to be completed.

• Jonathan:
  • n/a

• Joel:
  • Eclipse/TPTP and the Agent Controller is now deployed and expanded on the target platform.
  • Jonathan will assist Joel with configuring and running the Agent Controller.
  • A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting.
  • Since there is defect when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests.
    • See the Common Base Event JUnit tests.
  • Joel will work on automatically posting the Test Report to /tptp/test/reports/<release>.

Delivery dates

• November 14:
  • Complete the first testing process document.
  • Proof of concept (PoC) for integration of ASF and TPTP builds for PMC approval and integrate with TPTP builds.
  • Measure cost savings.

• November 21:
  • Identify and resolve bugs.

• November 28:
  • Each project automates their manual test suites and converge existing automated test suites over time to leverage the test infrastructure.
  • Presentation to TPTP committers.
  • Report cost savings to PMC.

Action Items

• Alan: Complete the new test process.
• Jonathan: Assist Joel with configuring and running the Agent Controller.
• Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard.
• Joel: Attach a diagram of this infrastructure to the WIKI.
• Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
• Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
• Paul: Encourage Project Leads to automate existing manual test suites.
• Paul: Book another meeting for next week (November 12, 2007).
• Paul: Fix defect when deploying TPTP JUnit tests remotely.

References

• ASF
• ASF Tutorial
• TPTP Testing Strategy
• Test Dashboard (see enhancement 112928)
• Test Dashboard Plug-in