STEM Releng

STEM Release Engineering

This page details how to operate the STEM builder to create binary, platform-specific builds of STEM. This information is for advanced users and committers only.

Background

The STEM build process is built on several foundational pieces of the Eclipse Platform. These pieces include:

• Eclipse Products
• Eclipse Features
• PDE Build Process
• P2 Provisioning

These fundamental aspects of the Eclipse Platform are orchestrated to build platform-specific binaries of STEM.

Adding New Code to the Builder

This section describes how to add new plug-ins, features, and third-party dependencies to the STEM Build process.

Important Note About Feature Cross-Dependencies

Plug-in dependencies across features are not as simple as they appear. For a feature (Feature A) containing plug-ins built from source (e.g. not binary), then any feature(s) (Feature B) it depends on must either be built completely first or the plug-ins in Feature B that Feature A depends on must also be included directly in Feature A.

Example:

• STEM has Features org.eclipse.stem.core.feature and org.eclipse.stem.models.epidemiology.feature
• Plug-in org.eclipse.stem.ui is in the core feature but depends on the org.eclipse.stem.diseasemodels plug-in, which is (logically) in the epidemiology feature
• Plug-in org.eclipse.stem.diseasemodels depends on the org.eclipse.stem.core plug-in, which is in the core feature
• The epidemiology feature cannot be built completely first because it depends on plug-ins in the core feature
• The core feature cannot be built completely first because it depends on plug-ins in the epidemiology feature
• Therefore, to solve this cross-dependency, the org.eclipse.stem.diseasemodels plug-in must be in both core and epidemiology features

Adding a new plug-in to an existing feature

STEM is built from Eclipse features and plug-ins. This section describes how to add one or more new plug-ins to the STEM build process to an existing feature. If you want to create a new feature, see the next section.

• Determine which Feature(s) the plug-in(s) should be added to

  See note above about feature cross-dependencies to help determine which features you should add the plug-in to

  1. In Eclipse, expand the feature project the plug-in(s) should be added to and double-click feature.xml
  2. In the Feature Editor, select the Plug-ins tab
  3. On the Plug-ins and Fragments page, click the Add... button
  4. In the Plug-in Selection dialog, type the name of the plug-in you wish to add. When it shows up in the list, select it and click OK

     Repeat this step for all plug-ins you wish to add to this feature

  5. Save feature.xml (File-Save or Control-S)
  6. Commit feature.xml (Right-click Team-Commit)

• Add the plug-in to Git
  1. In Eclipse, select the plug-in project you wish to add.
  2. Right click on the project, highlight Team and select Share Project
  3. Follow the steps to add it to the repository.

     Note: The STEM Source Repository has a specific layout. Please make sure to place your new plug-in in an appropriate location. Plug-ins are grouped in folders with other plug-ins of similar functionality (such as disease models ). Remember this location, as it's needed for the build.map step below.

• Add the plug-in to the build map
  1. In Eclipse, expand the org.eclipse.stem.releng project
  2. Expand the maps folder and open stem.map in the text editor
  3. Add the plug-in to the build map. The syntax is as follows:

     plugin@[PLUGIN_NAME]=COPY,${checkoutDir}/[REPOSITORY_PATH],[PLUGIN_NAME]

     Where:

     [PLUGIN_NAME] should be replaced with the plug-in name (Example: org.eclipse.stem.core)

     [REPOSITORY_PATH] should be replaced with the path in SVN where the plug-in was added, relative to trunk (Example: plug-ins added to /trunk/core would use core for the [REPOSITORY_PATH])

     Important Note: Plug-ins are grouped in the build map for project set generation. To ensure that a plug-in gets checked out, make sure it's grouped appropriately with similar plug-ins

  4. Save and commit build.map

Adding a new feature

If the new functionality you're adding is substantially different from existing functionality in STEM (such as a new type of modeling, like climate modeling ), then it may be appropriate to add it as a new feature. These steps will help you add a new Eclipse Feature to STEM's runtime and build process.

Important Note: These steps add a feature to the default build process (which, in turn, adds the feature to the main STEM distribution). If you do not wish for the feature to be part of the STEM distribution, only follow the first step below (Creating the Feature Project).

• Create the Feature Project
  1. In Eclipse, select the File menu and click New
  2. In the New wizard, expand the Plug-in Development category and select Feature Project . Click Next
  3. On the New Feature page, populate the fields
     • Project Name should be fully qualified and end with .feature (Example: org.eclipse.stem.new.feature)
     • Feature ID should be the same as Project Name
     • Feature Name should be a short description of what functionality the feature provides
     • Feature Provider should be your organization's name (Example: Eclipse Foundation)
  4. Click Next
  5. On the Referenced Plug-ins and Fragments page, select the plug-ins to be part of this feature.

     Note: If the plug-ins aren't created yet, use the steps from the previous section to add them to the feature later

  6. Click Finish
  7. Add the new feature project to SVN

     Note: In the STEM Source repository, feature projects are placed in the /trunk/features folder.

• Add the Feature to the STEM Product Feature (optional)
  1. In Eclipse, expand the org.eclipse.stem.product.feature project. Open feature.xml
  2. In the Feature Editor , select the Included Features tab
  3. On the Included Features page, click Add
  4. In the Feature Selection dialog, type in your Feature Project's name. When found, select it and click OK
  5. Save and commit feature.xml

• Add the Feature to the build map
  1. In Eclipse, expand the org.eclipse.stem.releng project
  2. Expand the maps folder and open stem.map in the text editor
  3. Add the feature to the build map at the end of the rest of the features. The syntax is as follows:

     feature@[FEATURE_NAME]=COPY,${checkoutDir}/features,[FEATURE_NAME]

     Where:

     [FEATURE_NAME] should be replaced with the feature project name (Example: org.eclipse.stem.core.feature)

  4. Save and commit build.map

Add a new dependency

Adding a dependency from another Eclipse project

See next section

Adding a third-party dependency from a P2 Repository (including Orbit bundles)

The easiest way to include a third-party (non-Eclipse) dependency is to select one from the Eclipse Orbit project. Orbit is a formal process by which non-Eclipse, third-party libraries are vetted by Eclipse Legal and approved for general use and redistribution by Eclipse projects. This significantly reduces the amount of work required for both committers and Eclipse Legal to get libraries approved for redistribution.

1. Select the library that you wish to include from the latest Orbit release (See here for releases)

   Note: Keep note of the IPZilla CQ number from the release table

2. Committers Only In the Eclipse Committer Portal, fill out the use a third-party library from Orbit form to request approval to use the library
3. Add the library to the Target Platform
   1. In Eclipse, expand the org.eclipse.stem.releng.targetplatforms project
   2. Open the STEM2.target target platform
      • If the existing Orbit release repository is in the target platform , select the repository, click Edit , and choose the library from the list to include, and click Finish
      • If the existing Orbit release repository is NOT in the target platform , click Add , select Software Site , enter the Orbit Release Repository URL, select the library you wish to use, and click Finish

      Note: The Orbit repository URL is http://download.eclipse.org/tools/orbit/downloads/drops/[ReleaseVersionNumber]/repository/

   3. Save the Target Platform
4. Add the Library to the Build Script Mirror List
   1. In Eclipse, expand the org.eclipse.stem.releng project
   2. Open build.xml
   3. Search for the <p2.mirror> Ant task
   4. Add the P2 repository URL to the <repository> list (see examples in file)
   5. Add the plug-in's P2 Installable Unit (IU) entry to the <iu> list
      • The feature/plug-in id and version come from the P2 repository metadata.
      • For Orbit bundles, the id is the plug-in and the version is the full version as listed on the Orbit page
   6. Save and commit build.xml

Running the STEM Builder

The section describes how to setup and run the Eclipse PDE Builder to create a STEM binary.

Setting up your environment

• Environment with a compatible shell (bash)
  • Windows users are recommended to run the builder from cygwin or a similar interpreter
• Subversion command-line client (svn)
• Java J2SE Development Kit (JDK) 6.0 (Oracle or IBM)
• Eclipse Classic SDK with the following features installed:
  • Eclipse Modeling Framework (EMF)
  • Eclipse Business Intelligence and Reporting Tools (BIRT) Framework
  • Eclipse Zest Framework
  • Eclipse RCP Delta Pack (for Eclipse 3.7.0, see here)
  • Eclipse Releng Tools (For Eclipse 3.7.0, see here)

Checking out the source from Git

The STEM Builder looks for projects based on the Git Repository layout, not in the flat structure of an Eclipse workspace. Therefore, you should clone and use the repository AS IS using the Git command line client.

This git clone command clones the STEM source repository to your local machine with the proper directory structure.

git clone git://git.eclipse.org/stem/org.eclipse.stem.git

To checkout from a different branch, use the git checkout -b command after clone. Example that checks out the STEM_1_4_1 branch:

cd org.eclipse.stem

List all branches: git branch -a

Checkout branch: git checkout -b STEM_1_4_1

Running the STEM Builder

The STEM builder is built on top of the Eclipse PDE build process. For STEM, it's three steps that are controlled by build properties:

1. Build the STEM Features
2. Create and Package the STEM Product
3. Run JUnit Tests

Each step can be enabled or disabled using appropriate property switches

Build Properties

The following are the build properties that should be defined before invoking the STEM build script.

Important Note: The org.eclipse.stem.releng/build.properties file defines these properties. They can also be passed in as -D parameters to the ant script.

Variable Name	Required	Example	Description
buildFeature	yes	true	Build the STEM Features. In most cases, this should be true
buildProduct	yes	true	Create and package the STEM Product. In most cases, this should be true
runTests	yes	true	Run the JUnit tests after the features and product are created.
skipData	yes	true	Skip building the denominator data. True for most cases. (default true)
eclipse.home	yes	/usr/local/eclipse	Path to the root folder of your Eclipse SDK installation. The eclipse plugins folder should be located at /plugins relative to this path
eclipse.deltapack.home	no	/usr/local/eclipse-dp	Path to the Eclipse RCP Delta Pack.
majorVersion	yes	1.2.1	The release train this build is member of. Should be in X.Y.Z form. Should not include other information, such as a qualifier.
buildTypeFull	yes	nightly	The type of release. One of three options: nightly - unstable builds created automatically integration - weekly or semi-weekly that are mostly stable stable - pre-release stable builds, such as milestones and release candidates release - official, stable releases
buildQualifier	no	RC1	Version qualifier for stable builds only. Examples are M1, M2, RC1, RC2, etc.
publishBuild	no	false	Whether a build should be prepared for publishing to Eclipse. Only valid for STEM project release engineering.
publishDir	no	/opt/stem/builds	Path that builds should be published to. Only valid for STEM project release engineering.
cleanupFirst	no	true	Removes all files from build type's publishDir before copying new build. Only valid for STEM project release engineering.

Launching the Builder from Command Line

The builder scripts are located in the releng/org.eclipse.stem.releng folder.

The STEM Builder uses Apache Ant to run.

1. Download and install Apache Ant. STEM Release Engineering uses Ant 1.8.x

   http://archive.apache.org/dist/ant/binaries/

2. Clone the STEM Git Repository to a local path

   git clone git://git.eclipse.org/stem/org.eclipse.stem.git

3. From a command line or terminal, change into the org.eclipse.stem.releng directory

   cd [STEM_CHECKOUT_PATH]/releng/org.eclipse.stem.releng

4. Edit the build.properties file to configure the builder for your environment (see above for help)
5. Invoke ant

   ant

6. Wait while STEM builder compiles and packages the binaries

   Depending on your system speed and options specified, this step could take from 30 minutes to 1 hour to finish

7. After it's finished, the zip files containing the binaries are located in the build folder:

   cd build/I.WeeklyBuild

Launching the Builder from Hudson/Jenkins

If you wish to setup STEM builds using Hudson or Jenkins, you can easily do so. In Hudson/Jenkins, you can configure a job to define the build properties listed above and then configure a job to Invoke Ant with the Build Script path of releng/org.eclipse.stem.releng/build.xml. This will invoke the ant builder. See section on Continuous Integration below for more information.

Running the JUnit Tests

JUnit tests are run as part of the standard build process. Whether or not they are run is controlled by the RUNTESTS environment variable. To run JUnits as part of the build process, simply set runTests=true in build.properties before invoking ant.

Test results are aggregated and collected in the releng/org.eclipse.stem.releng/tests/ folder.

Important Note: Running JUnit tests requires a windowing environment to be available to the terminal invoking the build script (e.g. $DISPLAY must be defined). There are several options available for all-text environments, such as running Xvnc. Most users won't be affected by this.