Testing Process Part 1 Draft 2 Comments

Paul Slauenwhite

• Can we change the organization to (note new sections):
  • 1.0 Preparation
    • 1.1 Install the TPTP Test Tools
    • 1.2 Extracting the Test Resources
    • 1.3 Structure of the Test Resources
  • 2.0 Creating Tests
    • 2.1 Manual Test Suites
    • 2.2 JUnit and JUnit Plug-in Test Suites
    • 2.3 AGR Test Suites
  • 3.0 Executing Tests
    • 3.1 Manual Test Suites
    • 3.2 JUnit and JUnit Plug-in Test Suites
    • 3.3 AGR Test Suites
  • 4.0 Common Tasks
    • 4.1 Creating Test Suites
    • 4.2 Launching Test Suites
    • 4.3 Viewing Test Logs
    • 4.4 Report Test Results
  • 5.0 Help
• We need to document the BVT process:
  • The root-level test suites for BVTs.
  • The test infrastructure will execute the org.eclipse.hyades.tests/AllBVT<test type>Tests.testsuite test suites.
  • We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results.
  • Each developer can rerun the automated tests to reproduce a failure.
• We need to add the new structure in CVS, which now to three sets of root level test suites in the TP1, TP2, BVT directories.
• 1.1:
  • "1. Open the TPTP download site (embedded link to http://www.eclipse.org/tptp/home/downloads/)."
  • "2. Select the release of TPTP to be tested and download...". Capitalize Remember in the note. AGR is now an as-is component (not Technology Preview).
• 1.2:
  • i) "for that subproject".
  • 2) "beginning of each test pass, test report generation, etc.".
• 1.3:
  • Remove steps 1, 2, and 4.
  • Move step 3 to the new 4.2 section.
  • Remove white space after step 6.
  • In step 5, change Client to View.
  • In the verdict table, add an note on adding defects to the test log (see http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/t_submitting_defects.html), remove the second sentence in the fail verdict explanation, and add a note to provide comments explaining the verdict (e.g. fail and inconclusive) if applicable.
• 1.4:
  • Change plugin to plug-in.
  • Remove steps 1 and 2.
  • Move step 3 to the new 4.2 section.
  • Remove white space after step 4.
  • In the verdict table, add an note on adding defects to the test log (see http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/t_submitting_defects.html).
• 1.5:
  • Remove steps 1 - 3.
  • Move steps 4 and 5 to the new 4.2 section.
  • Remove white space after step 6.
  • In the verdict table, add an note on adding defects to the test log (see http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/t_submitting_defects.html).
• 1.6:
  • In step 1, change _results to results and give an example of the structure.
  • In step 2, change id to identifier.
• 2.0:
  • Move to new 4.1 section.
  • In steps 2 and 7, move new section 1.3.
  • In step 6, move section from part 2 to here.
• 2.1, 2.2 and 2.3 should contain instructions for creating the specific type of tests.
• 2.3:
  • Move 1 a, c, d to new 1.3 section.
  • In step b, append ", if applicable".
• 2.4:
  • Need more content here about what is expected from developers for BVT (e.g. all tests in the root level BVT test suites are ran every build and it is expected that developers will test locally before checking into CVS).
• 3.0 and 4.0:
  • Integrate into one Help section with:
    • • TPTP web site
      • Point to TPTP Test Tools documentation (Help >> Help Contents >> Testing Applications).
      • AGR documentation.
      • TPTP consumers documentation (http://www.eclipse.org/tptp/home/documents/extenders.php).
      • Mailing list and news group (http://www.eclipse.org/tptp/home/project_info/general/mailnews.php).
• New 4.2 section:
  • The follow steps are common for launching all types of test suites
    • Create a new local test deployment:
      • (add steps - see http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/tdeploymentintro.htm, note the distinction between locations and workbench locations).
    • Create a new remote test deployment:
      • (add steps - see http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/tdeploymentintro.htm, note the distinction between locations and workbench locations).
    • Create a test launch configuration:
      • (add steps - see http://help.eclipse.org/help33/topic/org.eclipse.hyades.test.doc.user/tasks/trunningmanualtestsuite.htm)
      • Choose the results folder associated with the selected test suite and deployment platform

Note: The name and directory structure containing the generated execution history file MUST be the same as it will be in CVS in order for generated reports to work correctly.