Platform-releng/Platform Build Automated

This page is a small addition to Platform-releng/Platform_Build. Be sure you have read that first and are familiar with the concepts (and preferably have done "local builds" as described there).

The CBI-based automated builds for, Platform Releng, essentially automates the steps described in Platform-releng/Platform_Build and does a few additional things meant to be convenient for Platform committers, who are used to the procedures in the PDE based builds. This page gives the briefest overview, and describes what's different. [Note: the scripts/code used to do these things might well someday make their way to some "common tools" repository, but most are "platform specific" and not meant as "API" or to be re-used as-is ... but, naturally, if others find them useful as a starting point are free to make a copy, etc., under EPL constraints.]

Contributing to a build

Primary Method

Typically, all a committer has to do is commit and push to the branch that code belongs too, and the automated build will fetch the most recent version of that branch, tag it (with buildId, such as I20130709-0800), and then build it.

But which branch? The branches that the automated build pulls from, for each repository, are listed in repositories.txt. That file is in the "streams" directory of the aggregator repository. For the master version, see

https://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/cje-production/streams/repositories_master.txt

Tip on making sure qualifiers increment correctly

See Tycho/Reproducible_Version_Qualifiers for Tycho documentation on this topic.

Bundles

Normally, for I-builds, the qualifier of bundles is computed automatically, from the most recent commit to the source tree of that bundle. One exception to this rule is if the only the POM file changes. POM changes are ignored in computing the qualifier. Most of the time, this is what's desired (that is, the POM change doesn't really effect what'd built) ... but, many times, a change in the POM file will effect what is built, in which case the bundle has to be "touched" to force the qualifier to increase. Our project's convention is to use a file named 'forceQualifierUpdate.txt' at the root of bundle. In the event the qualifier needs to be "forced" to increment, it is customary to added a line of text to that file, such as the bug number that lead to it needing to be touched, and committing that to Git. (The file is not otherwise "exported" with the bundle binary or source).

Features

Features are similar to bundles, described above, except it's not only the feature that is considered, but is also all the bundles it contains. It's qualifier is basically determined by the most recent commit of that whole collection. (The same "exclusion of POM files) applies to features too.).

One well known exception is "third party jars" such as from Orbit. Since they don't have a "commit" like the rest of the source tree, they won't cause a qualifier to increase, even if it is a new or different version that is added to a feature. In these cases, the "forceQualifierUpdate.txt" is the mechanism to use to get it to update, if 3rd party bundles change.

There is another, more problematic case that's been mentioned in several places, such as bug 436039 and bug 416619. In short, the qualifier is determined by "commit date" ... nothing like "push date". So it's fairly easy, especially when we do I-build every day, for a feature (or other bundle) to have "the latest commit date" if some other bundle was committed, but not pushed right away. That is, the feature qualifier would have already been increment for "most recent" bundle ... and the new "push" doesn't look any newer, since it's commit is older than what's already being used.

Therefore, especially during critical periods, such as "right before a milestone or release", if you change a bundle, you will want to strongly consider proactively "touching" the feature (or, features, if contained in several) to make sure they are correctly incremented, instead of depending on the automatic computation. Ironically, this problem is worse, when we do frequent I-builds (since that serves as the basis for the comparator "swapping out" items whose qualifier has not changed) since then it is more likely "the most recent" commit has already incremented a feature's qualifier in that repo -- and an older "commit", even though more recently pushed, will not be "picked up" in already qualified feature.

Trigger builds manually

The builds can be triggered manually or scheduled to run at a specified time. The build jobs are available at releng JIPP instance. We use I-build for integration builds. These jobs can be directly triggered after logging in or can be scheduled by modifying the configuration of the specific job.

Syntax of cron consists of 5 fields separated by TAB or whitespace as below:

   MINUTE HOUR DOM MONTH DOW

   MINUTE	Minutes within the hour (0–59)
   HOUR		The hour of the day (0–23)
   DOM		The day of the month (1–31)
   MONTH	The month (1–12)
   DOW		The day of the week (0–7) where 0 and 7 are Sunday, 1 is Monday, 2 is Tuesday, 3 is Wednesday, 4 is Thursday, 5 is Friday and 6 is Saturday

Other tasks of the automated build

The automated build also creates summary pages of the build, and starts jobs on Hudson to do the Unit testing and then creates summaries of test results, as before. Much of this is controlled by scripts and cron jobs that are in a special directory called "production", and some in resources in "eclipse.platform.releng.tychoeclipsebuilder" which conceptually corresponds to the previously used "eclipse.platform.releng.eclipsebuilder". Some of this may be "mavenized" eventually, but for now, bash scripts and ant tasks.

Routine release engineering tasks for builds

There are some routine tasks that release engineers have to do, with help of committers, to keep the build "current" (and these usually effect "local, casual builds" as well as production builds). These routine tasks are described in the sections that follow.

Update JDT Compiler Version

Typically, the JDT compiler is updated after each milestone and then at end, the RC2 (final release) version is created with RC1 version of compiler (if not in an earlier RC). This is done partially to make sure our code is generated with the best possible compiler, but also to help test our compiler, on our own code.

The version of the compiler used is controlled by two elements in the eclipse-platform-parent. Current examples are as follows:

     <cbi-jdt-repo.url>https://repo.eclipse.org/content/repositories/eclipse-staging/</cbi-jdt-repo.url>
     <cbi-jdt-version>3.9.0.v20130604-1421</cbi-jdt-version>

The URL is pretty much constant, and is a special "maven repository" where we can put unreleased versions of Eclipse bundles (and remove them later, as desired). The method of adding a bundle to that maven repo is outlined as follows:

1. Place the org.eclipse.jdt.core_(version).jar in /shared/eclipse/jdtstaging/
2. Update the pom.xml file in that directory to refer to the correct version of the bundle (artifact).
3. Run the Hudson job named deployJDTCompiler. That is a parameterized Hudson job which requires the version of the JDT core bundle and the location where the bundle and pom.xml file are located. (Default value should be correct for location, but version has to be update for each new version). [Note: this is a Hudson job that is set up with special security tokens, to make it harder for someone to accidentally (or, maliciously) put something bad in the maven repository, since it is used by others.]

Remember to watch the comparator logs after a compiler update, as some bundles may have to be manually "touched" to force a qualifier change, or else new byte codes, if any, will not be picked up for distribution.

NOTE: We typically will leave a "milestone compiler" in the staging repo for only 2 or 3 milestones, and then remove it, without announcement. So if anyone else is "following along", and doing frequent updates of the compiler, don't forget you too will need to update every milestone or two.

Once a compiler is part of an official release, it will be moved to Eclipse Foundations maven repository at

https://repo.eclipse.org/content/repositories/releases

(And, by "compiler", is meant the org.eclipse.jdt.core_(version).jar OSGi bundle, which, of course, can be used as a stand-alone compiler (see Eclipse Help for details. Thus far, we've not had any requests to put a mavenized version of ecj.jar there, which is the same thing ... just a little smaller since doesn't contain all the function that is in org.eclipse.jdt.core_(version).jar).

Update Eclipse version for eclipserun-plugin (API Description and JavaDoc Generation)

The automated platform build generates some extra content. API descriptions (.api_description files at the root of a bundle) provide information on API restrictions for use with PDE API Tools. JavaDoc provides documentation for API packages. To build this extra content, the automated build must depend on an existing version of Eclipse as used by the tycho-eclipserun-plugin.

The version of Eclipse used will lag behind the current release. This means any changes to the generation code will not be reflected in the build until the build script is updated (similar to how we update the JDT compiler every milestone).

To update:

1. The instance of Eclipse used, as a "helper" for these tasks, is created by the Tycho eclipse-run plugin and makes use of whatever repository is defined in the eclipse-platform-parent. See the <eclipserun-repo> element in the global properties section, and update it, any time there is new API descriptor function or in the few functions used to help create documentation.
2. Remember, in such cases, where the "build mechanisms" change, some bundles may have to be manually "touched" for them to change qualifiers, and hence pick up the new changes (such as new .api_descriptors) rather than an older version (with same qualifier) being picked by comparator to include in distribution. If someone forgets, though, these "differences" should show up in comparator logs.

Other releng tasks

See also Platform-releng/How_to_do_miscellaneous_releng_tasks for other, miscellaneous releng tasks.