Making Builds Reproducible

Background

In order to build non-SDK projects offline -- a requirement of our buildsystem for security purposes -- we must do the following:

1. Generate a source drop
   1. check out releng plugin
   2. add a fetch target
   3. run the fetch target with the Eclipse antrunner
   4. tar up the results
2. Run the standard PDE build (ie. java -cp $SDK/startup.jar org.eclipse.core.launcher.Main

Common Pain Points

The last step -- the actual building with PDE -- is often different between projects. Some common variable differences:

• javacFailOnError - why isn't this always set?
• mapsLocal
• pde.build.scripts
• skipFetch
• buildLabel
• buildAlias
• baseExists
• eclipse.builder.fetch
• baseLocation vs. base.location
• pde.builder.path
• mapVersionTag
• buildDirectory

• a build.xml that is essentially a shell for PDE's build.xml - why not just use PDE build.xml directly?
• ant build files that are shells around the "standard" (essentially empty) build.xml that set obscure variables that are necessary but non-standard
• build scripts (bash, DOS) that set obscure variables that are necessary but non-standard
• build scripts that assume relative/absolute paths
• customTargets.xml attempts to fetch dependencies with no flags to override this behaviour
  • ex. getBaseBuilder is in postFetch unconditionally - we have to patch this out
• build scripts that assume certain usernames/passwords -- as mentioned in Eclipse Build Best Practices, a common cause of this is the lack of separation between fetching, building, and publishing

Solutions to these common gotchas

• separate fetching of your project's source code from fetching basebuilder/SDK and dependencies
• wrap fetching of basebuilder/SDK and dependencies with flags that can be overridden (ie. dontFetch)
• use consistent flags/variables
  • derive values from other variables (ie. pde.build.scripts/path from baseLocation
• do not assume use of basebuilder as it is almost always perfectly fine to use the SDK -- especially for I- or M-buildsand releases -- and we do this :)
• do not hard-code usernames, paths, etc. that aren't either universal or at the very least overridable

An Ideal Future

In an ideal world, all it would take to build a project on top of the SDK would be something like this:

1. assume SDK is available and have a standard way of passing its path during the build
2. download source drop from project -- no generated files (including build.xmls), no binary code
3. run the PDE build.xml with consistent options
   1. I envision us using: baseLocation, dontUnzip, dontFetch -- everything else should be derived from these values