WTP/Build/How to promote a build

To "promote a build", means to copy it from the build machine to the download machine. This happens, of course, after a build has been deemed promotable ... passed smoke tests or whatever other criteria there are for that build.

Background

Why doesn't the build script just put every build on the download server? There are a couple of reasons. First, the build machine's files are not mirrored, which turns out to be much more efficient for day-to-day continuous build. Since not many people download those, we would trigger a great deal of of unnecessary activity in the Eclipse.org infrastructure if we put every build on download server. The second reason is that this makes it a little more apparent which build consumers and early testers can pick up. We produce many builds throughout a given week, some of them have never even been downloaded or looked at by anyone so we don't want innocent users or testers downloading a very bad build.

Mechanics

There's basically two things that have to be done. One, copy the files. In our current setup, this is accomplished by copying a directory from the build machine somewhere in the /shared/webtools/committers directory to its download location, somewhere on ~/downloads/webtools/downloads/drops. The other thing that has to be done is that a few files have to have some URLs updated to reflect the new location (most URLs are relative, though, so are valid no matter what the top level directory is). Besides these main things, there's also cleanup to do. On the build machine, any builds during the week that are not promoted should be deleted. It is a good idea, though, to leave the build directory that is being in place, partially to give time for any downloads from users or testers to complete, if they happen to be getting the latest at the same time you promote. Plus, downloads from the build machine are faster or more available for many committers, since once copied to the download server, it can take 10 to 30 minutes (or more) to "show up" by the time it replicates. To actually start showing up on mirrors takes several hours (and usually overnight). There is another type of cleaning not really addressed here (that is, by the automation described below) and that is cleaning up old builds on the download server itself .... this happens when we "change types" of builds. For example, when we promote a Milestone build, then shortly after that we should remove the I-builds that led up to that milestone.

The Promote Automated Script

There is a script (a bash script) in the releng.control project called promote.sh that automates the tasks mentioned above: copies the most recent build in what ever project is named, fixes up the URLs, and removes all but the most recent build.

As of this writing, the script can be ran from an SSH console on the build.eclipse.org machine, from the /shared/webtools/releng.control directory.

Note that "most recent" is something to be careful of. While it does not happen often, occasionally we'll do a "final" build, only to discover it didn't go well, and decide to promote some previous build instead. In those cases, you would have to "manually" delete any build directories that were more recent than the one you wanted to promote before using the scripts.

The script can be ran with something like

./promote.sh  -[vcsd]p build-project

where build-project is one of the projects such as wtp-R3.0-M, wtp-R3.1-R, etc.

The options are all singe letters and technically optional (though not much is done if there are no options.

v verbose: 
  provides more output about what it's doing 
 
c copy: 
  actually does copy the most recent version from the specified project. 
  If 'c' is not provided, no copy is done but the most recent one is listed to the console. 
  This is one way to 'sanity check' that the one that will be copied matches the one you think it is. 

s send:
  a mail notification is sent to wtp-dev@eclipse.org announcing the build has been "declared". 
  Note: If sent at the same time the copy is done, the 'download' may not actually be available when people
  first read the mail ... as the download replication can take from 10 to 60 minutes. You can run the promote
  script once, to do the copy (only) and then run it again to do the send mail (only) once you've confirmed 
  the download is available. That is, you can do this as long as there have been no more recent builds done 
  for this project. [An obvious area for automation improvement :) ]

d delete: 
  deletes the previous, older builds from the same project that is being promoted. 

p project: 
  the 3-part name associated with any CC projects, such as wtp-R3.0-M, wtp-R3.1-R, etc. 

So, for example, the command to promote a weekly M build on the 3.0 maintenance branch would be as follows (ran from the /releng/control directory.

./promote.sh -dcsp wtp-R3.0-M

The script will prompt you for your password twice, once at the beginning and once at the end. To setup the proper id for the password:

1. Make a copy of the /shared/webtools/releng.control/commonVariations.shsource file to your home bin directory /home/data/users/<your id>/bin
2. Edit the commonVariations.shsource file in your home bin dir and update line with REMOTE_SSH_COMMAND, such as

export REMOTE_SSH_COMMAND="ssh <yourid>@build.eclipse.org"

Issue: This should work (without password prompt) if you just set the variable to empty. That is

export REMOTE_SSH_COMMAND=

I've not personally tested (and would be hard for me to, given my ids and settings, but if someone confirms, please update this section ... changing it to the recommended action, no longer "an issue".

Theory: Interested in why this is this way? There are some commands that are ran sometimes from "wtpBuild" ID but that ID can't doesn't exist on the 'download' servers, so it's required to SSH with an id that does exist there. But, if you are using your "real" eclipse foundation id then commands should work on both build.eclipse.org and download.eclipse.org.

Issue: Technically, the ~/bin/commonVariations.shsource file should only have variables in it that you know you need to customize for your ID, such as the REMOTE_SSH_COMMAND. That way, if the "main" commonVariations.shsource file changes, you won't end up with stale values. (I document this as an "Issue" for now, since not sure of the best way to write this up, to make it easy. Not sure if there's any other variables in there that will likely have to be modified by others, when on the build.eclipse.org machine. By the way, many have to be modified if you are actually running our build scripts on a local machine).

Requirements for using the script

• a "shell account" on the build server
  • a "shell account" can be requested on the Eclipse Portal under the [tools] link.
• webtoolsadmin privilege
  • the "webtoolsadmin" group membership can be acquired by asking the releng team lead to send out a note to the webmaster.

What if things change?

The process (and script) makes a few assumptions about the structure of the build machines directory structure and the webtools downloads directory structure. Hence, if any of those things change the script would have to be redone. Similarly, if and when there are new projects added, say "xml-4.0-R" then some tweaks might be required.