Buckminster Galileo Builder

Introduction

The Buckminster Galileo/Helios Builder was created to enable a fast but still very accurate build of the Galileo release train. It's provided as a single feature to be installed either into an IDE or into some headless environment. The builder is intended to be used on any computer, not just on the build server at Eclipse.org.

Note that this page is about the builder itself. For instruction on how to contribute to the build, see Contributing to Galileo Build or Contributing to Helios Build.

The build will perform the following steps:

1. Generate a full build model from the all contributions.
2. Create a composite repositories that includes all contributed repositories.
3. Create a special category repository that includes categories declared in the model.
4. Create a feature that includes all contributed features and bundles.
5. Verify that the feature can be installed into all configurations declared in the model.
6. Create a mirror that includes everything. The mirroring is selective and will only bring in the transitive closure of all contributions. No categories are copied from the contributions. This last step is optional.

Steps 1-5 usually completes within 3-6 minutes on my machine (located in Sweden). The time is very much dependent on bandwidth and the current mode of download.eclipse.org. These steps will probably reveal about 95% of the problems. During the mirroring in step 6, it's verified that all artifacts exists and that packed files can be unpacked.

The builder should make a fairly good job of determining where to send emails when things go wrong. So far, this functionality has been tested using mock emails only. I think we should continue in that vein until everything is found to be working properly and the builder can be used in production. I.e. do not pass the -production flag to the builder unless you really know what you're doing.

How to get started

1. Start an Eclipse 3.5 installation, I20090407-1430 or newer. If you don't have one available, download from here http://download.eclipse.org/eclipse/downloads/
2. Use the P2 update manager and add the update site http://download.eclipse.org/tools/buckminster/tools-galileo/
3. Install the Galileo Builder feature found under the Buckminster Tools category
4. Check out the project org.eclipse.galileo.build from the CVS repository at :pserver:anonymous@dev.eclipse.org:/cvsroot/callisto. This is the project that contains all the contributions.

NOTES:

• Unless you specify an output directory for the build using the -buildRoot option (see options below), the builder will use the buildRoot element of the galileo.build file to determine where to put its files. Currently this is defined as /shared/galileo/build. This directory is created if it doesn't exist and will need to be read/writable by Eclipse for the build to work.
• The heap allocated for the build Java VM needs to be increased to at least 256 megabytes for the build to run. In the Run Configuration->Arguments->VM arguments add -Xmx256m to increase the max heap size.

Running the build

From the command line

Just type:

eclipse -nosplash -application org.eclipse.buckminster.galileo.builder.app <options>

From within the IDE

Create a Launch Configuration of type Eclipse Application, check Run an application, select org.eclipse.buckminster.galileo.builder.app, go to the Arguments tab and enter the options, then click Run (or Debug).

Hint: If you want to point to the contributions you have checked out into your workspace, use:

-buildModel "${project_loc:org.eclipse.galileo.build}/galileo.build"

Command line options

Option	Value	Default	Description
-production	N/A	N/A	Do absolutely not use this option unless the builder is used in real production.. This option will enable sending of emails to proper addresses. If you want to test email sending, use mock emails (see -mockEmailTo and -mockEmailCC). Do not use this option unless you really really know what you are doing.
-buildModel	<path to file>	This value is required	An absolute path to the galileo.build file (the one found adjacent to all contributions in the org.eclipse.galileo.build project)
-verifyOnly	N/A	N/A	Only verify all meta-data. Do not create a mirror
-updateOnly	N/A	N/A	Reuse the results of the previous run, i.e. update metadata and, if applicable, the mirror
-logLevel	ERROR, WARNING, INFO, or DEBUG	INFO	Control the verbosity of the trace output. Using DEBUG forces the logger to print out stack-traces for logged exceptions.
-logURL	<url>		A URL that should point to the log that captures the standard and error output from the build.
-buildRoot	<path to folder>	The buildRoot declared in the build model	A path denoting the folder where all generated artifacts will end up. This setting overrides the attribute with the same name in the build model.
-buildId	<build id>	build-<timestamp in the format yyyyMMddHHmm>	The ID of the current build.
-packedStrategy	<strategy>	copy	Controls how mirroring is done of packed artifacts found in the source repository. Can be one of: copy  If the source contains packed artifacts, copy and store them verbatim. No further action verify  Same as copy but unpack the artifact and then discard the result unpackAsSibling  Same as copy but unpack the artifact and store the result as a sibling unpack  Use the packed artifact for data transfer but store it in unpacked form skip  Don't consider packed artifacts. This option will require unpacked artifacts in the source
-mirrorReferences	N/A	false	Mirror all meta-data repository references from the contributed repositories.
-referenceIncludePattern	<regexp>	.*	Include only those references that matches the given regular expression pattern.
-referenceExcludePattern	<regexp>	empty	Exclude all references that matches the given regular expression pattern. An exclusion takes precedence over an inclusion in case both patterns match a reference.
-brandingContribution	<name>	empty	The name of the contribution that contains the branding feature. This feature must be listed in a map present in the <buildRoot>/maps folder. The feature will be built from source and published in the final repository. A typical example is -brandingContribution Galileo.
-noBrandingBuild	N/A	false	Prevents the source build of the branding contribution. Useful if you want to verify all but branding. But remember, even if not actually build it, you must still specify the branding contribution, if the model has one, or there will be error because the builder will simply think there is a "loose" feature with no repository specified.
-trustedContributions	<comma separated list of names>	empty	A list of contributions with repositories that will be referenced directly (through a composite repository) rather then mirrored into the final repository. Trusted repositories will NOT have their categories exposed, but instead the Galileo defined categories are used. A typical example is -trustedContributions Eclipse.
-emailFrom	<email address>	Address of build master	The email address to use in as the sender of emails.
-emailFromName	<name>	The name of the build master	The name to associate with the senders email address
-subjectPrefix	<string>	The label for the build	A prefix for the email subject. Will be placed within brackets.
-smtpHost	<host name>	localhost	The name of the SMTP host, something like mail.example.com.
-smtpPort	<port number>	25	The port that the SMTP host will listen to.
-mockEmailTo	<email address>	Use contact addresses provided with the contributions	An email address that will be used instead of the addresses listed with the contributions. Useful when testing the email sending function. NOTE: The log will indicate the real address, not the mock address, but no email is sent to the real address.
-mockEmailCC	<email address>	Use the build masters address	An email address that will be used instead of the declared build-masters address. Useful when testing the email sending function.

Theory and Practice

Choice of technology

The builder is written in Java to get fine grained control over things like the P2 planner and the P2 mirroring functionality. The interface to the builder is through the standard org.eclipse.equinox.app.IApplication interface. This means that the builder can be installed and run from any equinox based environment. It does not need to be an Eclipse IDE. A much smaller version of the builder can be configured directly on top of a headless runtime (although at the time of writing, I'm awaiting the removal of some UI dependencies from the m2m team).

Contributing projects

Technology from the following projects is used by the builder:

• Amalgam: Contributes the build model and templates used by the builder.
• EMF: Contributes the modeling framework which includes parsers, transformers, and generators.
• Equinox: Contributes the P2 provisioning framework and the OSGi platform
• Buckminster: Contributes the 'create repository from source features' functionality and of course, the Galileo Builder itself.
• Platform: Contributes the Eclipse runtime.

The builder also uses org.apache.ant.tools.mail.MailMessage to send emails.

Flow of execution

1. After parsing the command line options, the builder will run a transformation on the Amalgam Build model that we use for all contributions. The transformation is performed by EMF. When the transformation is complete, the resulting model is loaded into memory. Since it's an EMF model, all parts of the model can now be accessed using real Java interfaces.
2. The builder uses the model to define certain defaults unless they have been overridden by command line options.
3. The P2 meta-data of all repositories from all contributions is loaded. One contribution at a time. This stage checks that each contributed feature can be found in the repository that it claims that it resides in. All loaded repositories are added to a composite repository (henceforth referred to as the global repository).
4. Next, the builder creates a new meta-data repository (let's call it the category repository). It generates each one category for each of the category definitions in the master build model. The category content is based on the category affinity declared in all contributed features. All generated categories are added to the category repository.
5. A feature named all.contributed.content is generated. It contains all features and bundles listed in the build model. This feature is added to the category repository.
6. The category repository is added to the global repository composite.
7. A P2 profile is setup for each of the os/ws/arch configurations that are listed in the build model and the P2 planner is asked to create a provisioning plan for installing the all.contributed.content feature into each of those profiles. If the planner succeeds, then all of Galileo can be installed into a runtime on those platforms. During this step, the planner will generate a list of operations to perform. This list is used when determining what the builder needs to copy from each contributing repository during the final mirroring step.
8. Finally, the builder will ask P2 to mirror all artifacts and all meta-data into a new repository. During this step, the builder will verify that all artifacts can be copied (since they actually are) and that packed content can be unpacked.

When things go wrong

If something cannot be built, a repository or feature missing, some misconfiguration etc., the builder will find the culprit contribution and send out emails to the contacts listed there. The email will be fairly specific and just contain the messages relevant for that contribution. The builder uses the standard javax.mail classes to send emails through SMTP.

Source for the Builder

The easy way, just import the source

Use this approach if your objective is to look at the code or just debug through it. Install the Source bundles for Galileo Builder feature. Once that is installed, just Import the org.eclipse.buckminster.galileo.builder into your workspace:

1. Click Import... to start the Import wizard
2. Choose Plug-in Development -> Plug-ins and Fragments
3. Click Next
4. Check Projects with source folders
5. Click Next
6. Select the org.eclipse.buckminster.galileo.builder project and click Add
7. Click Finish

Getting source from our SVN

You can of course obtain the source from our SVN repository too. This is necessary if you plan to provide patches to the code.

Install Subversive

1. Add these two sites (see: #Useful Bookmarks)

   http://download.eclipse.org/technology/subversive/0.7/update-site/

   http://polarion.org/projects/subversive/download/eclipse/2.0/update-site/

2. Select the following features:
   • Everything below the Subversive SVN Team Provider Plugin (Incubation) category
   • These two features below the Subversive SVN Connectors category:

     Subversive SVN Connectors

     SVNKit 1.2.2 Implementation (Optional)

3. Click next, agree to licenses, and install.
4. Follow the recommendation and restart your IDE

Checking out the source

1. Switch to the SVN Repository exploring perspective
2. Add the SVN repository http://dev.eclipse.org/svnroot/tools/org.eclipse.buckminster
3. Open trunk and check-out the org.eclipse.buckminster.galileo.builder project.

Getting it all with Buckminster

An alternative to just checking out the project is to use a Buckminster CQUERY.The CQUERY will not only obtain the needed source but also all of the dependent plug-ins. In other words, it will obtain everything necessary to compile and run the source, regardless of what you already have installed in your IDE. Everything will end up in your workspace (binary bundles are imported) so you don't need to worry about messing up your installation. If it's your first time, just follow these instructions.

Install Buckminster into your IDE

1. Make sure Subversive is installed
2. Add this site (see: #Useful Bookmarks)

   http://download.eclipse.org/tools/buckminster/updates-3.5

3. Select the following features:
   • Everything below the Buckminster category
   • Everything below the Buckminster SVN Support (Subversive) category
4. Click next, agree to licenses, and install.
5. Follow the recommendation and restart your IDE

Run the CQUERY

1. Click the File menu and select Open a Component Query
2. Enter the following URL http://www.eclipse.org/buckminster/samples/queries/builder.cquery
3. In the query that opens, click Resolve and Materialize

The expected result is that your workspace gets populated with the source for the Galileo builder and that a number of amalgan, emf, m2m, and ocl bundles are imported into your workspace. Those bundles will not have any source.

Useful Bookmarks

Instead of adding the update sites manually, you can store these bookmarks in a file and import them:

<?xml version="1.0" encoding="UTF-8"?>
<bookmarks>
  <site url="http://download.eclipse.org/tools/buckminster/updates-3.5" selected="true" name=""/>
  <site url="http://download.eclipse.org/technology/subversive/0.7/update-site/" selected="true" name=""/>
  <site url="http://polarion.org/projects/subversive/download/eclipse/2.0/update-site/" selected="true" name=""/>
</bookmarks>