TPTP-Test-Automation-20071112

Attendees

• Present:
  • Paul
  • Joel
  • Alan

• Absent:
  • Joanna
  • Kiryl
  • Jonathan

Testing Process

• A very lightweight testing process for TPTP.
• Extends the existing TPTP Testing Strategy.
• Considered as a instruction manual (e.g. step-by-step) for testing TPTP for each type of testing scenario.
• Two (or more) documents:
  • 1) High steps to use the testing infrastructure intended for first time users. This document will reference the other document(s).
  • 2) Low level detailed discussion on the motivation and design of the infrastructure intended for TPTP adopters or extenders.
  • 3) Additional documents detailing the specifics of a testing topics.

• Discussion:
  • Naming conventions for root level test suites and execution results should include test plug-in ID.
  • Execution results need to be saved in a directory with a meaningful name (e.g. OS, JRE, type (full, smoke, BVT)).
  • Test suites for specific JREs and OSes (e.g. JVMPI versus JVMTI) will not always run on the reference platform(s).
  • The existing technical content will remain unchanged, aside from review changes.
  • Need sections for:
    • Getting tests.
    • Creating tests.
    • Running tests.
    • Save test results.
    • Reporting results.
    • BVTs.
  • The Agent Controller and profiler test suites need to migrated to use the TPTP Test Tools instead of requiring a custom configuration.
  • The average tester/reader does not need to know about the technical details, but the steps to perform a certain testing task.

Reference Platform

• Target: IBM Java 1.5 (latest SR) and Windows XP/x86.

• Developers are expected to unit test on reference platform before check-in.

• Nightly builds invoke automated test execution on reference platform.

• Goal to use a second reference platform for better test coverage (target: Sun Java 5 and Linux/x86).

• Move to Java 6 once a complete test pass is completed to expose initial defects.

• The dedicate test machine for the reference platform has been clean and does not need to be re-imaged.

• IBM JREs for Windows and Linux (see IBM Development Package for Eclipse) are publicly available from developerWorks.

• Once the PoC is complete, Intel will focus on integrating the testing for the native components (e.g. JVMTI profiler) with the test infrastructure.

Common Test Infrastructure

• ASF integrated with the TPTP builds (invoked via ANT):
  • Invoke tests.
  • Interrogate results.
  • Generate test reports.

• TPTP test types:
  • AGR for functional UI testing.
  • TPTP JUnit and TPTP JUnit Plug-in test for unit testing.

• TPTP build infrastructure:
  • Extract test suites from CVS.
  • Provision reference platform (Agent Controller and Eclipse/TPTP).
  • Install and configure Agent Controller and Eclipse/TPTP on the reference platform.
  • Clean-up reference platform.
  • Post test reports (test and code coverage) on TPTP web site.

• See the diagram of this infrastructure.

• Limitations/Issues:
  • Can not test with the Integrated Agent Controller (IAC) since it cannot coexist with the Agent Controller running on the reference platform.
  • Intel needs to inform IBM that their portion of the build is complete (see defect 200351).

• Discussion:
  • The test infrastructure will execute each test plug-in's AllTest.testsuite, as referenced by the org.eclipse.hyades.tests/All<test type>Tests.testsuite.
  • Need to check out the test plug-ins in the /test-results/<sub-project>/ directory.

Deliverables

• Alan:
  • Reviewed Test Strategy document.

• Joel:
  • Eclipse/TPTP and the Agent Controller is now deployed and expanded on the target platform.
  • Jonathan will assist Joel with configuring and running the Agent Controller.
    • Jonathan is waiting on legal approval for using SSH service and a decompression application for the reference platform.
      • Joel to work with Jonathan to use Cygwin.
  • A sample ASF script is working, which handles TPTP JUnit test execution, results interrogation and BIRT reporting.
  • Since there is defect when deploying TPTP JUnit tests remotely, Joel will continue with using TPTP JUnit tests.
  • Executes the Common Base Event JUnit tests.
  • Joel will work on automatically posting the Test Report to /tptp/test/reports/<release>.
  • Joel will work on documenting the test infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
  • Outstanding/Issues:
    • Need to import test plug-ins into workbench before running the tests. Joel will work with Joe to determine how to do this with an ASF service.
    • Exception when generating execution results when the test suite path has a tailing slash. Joel will confirm with Joe if this is a defect.

• Paul:
  • Working on the defect when deploying TPTP JUnit tests remotely.
  • No project leads have replied to a request to estimate the current testing costs.
  • Paul will remind the TPTP Project Leads to provide a sizing.

Action Items

• Alan: Complete the first draft of the first testing process document for review by November 14, 2007.
• Alan: Complete the new test process.

• Jonathan: Assist Joel with configuring and running the Agent Controller.

• Joel: Prepare a small discussion on the architecture of the test infrastructure for next week's call.
• Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.
• Joel: Create a small slide deck for demoing the PoC to the PMC by November 14, 2007.
• Joel: Integrate the ASF with the TPTP builds to check-out a 'Hello World' TPTP JUnit test suite from CVS, run the test suite on an Eclipse instance running on the target platform, and generate a BIRT report on the test execution using existing sample code, possibly reuse some function from the Test Dashboard.
• Joel: Document this infrastructure for use in the second testing process document, which will be required by TPTP adopters or extenders.

• Paul: Determine labor cost (PW) for a) full test pass and b) smoke test pass from project leads.
• Paul: Encourage Project Leads to automate existing manual test suites.
• Paul: Book another meeting for next week (November 19, 2007).
• Paul: Fix defect when deploying TPTP JUnit tests remotely.

Upcoming Discussion

• Test framework reuse:
  • See defect 171827.
  • Existing test frameworks include:
    • Test Server for the Agent Controller/Profiler
    • Test Server for the JVMTI Profiler.
  • Proposed test frameworks include:
    • Test Dashboard (see enhancement 112928 and Test Dashboard Plug-in).
    • Test automation service (see defect 87285)

References

• ASF
• ASF Tutorial
• TPTP Testing Strategy
• Test Dashboard (see enhancement 112928)
• Test Dashboard Plug-in