Platform-releng-signedbuild

Signing platform builds

You'll find below the latest copy of the ant targets that we use in our buildAll.xml script to complete the signing portion of the build. Please note

During our build, we build a build-time feature that results in file called eclipse-master-${buildId}.zip. This file includes all the plugins and features that are used in the build, excluding test plugins. This file is used later on by pde build's packaging process to assemble the all the downloads available on the platform download page. By taking this approach, we ensure that each jar is only sent to the signing process once. That being said, the entire signing process still takes about 45 minutes to complete given the size of our build.

We scp the zips to and from the dev.eclipse.org server, instead of build.eclipse.org. The dev.eclipse.org server has QoS rules associated with it, and therefore the copy takes less time.

We add a pack.properties file to the eclipse-master-${buildId}.zip to overcome the the bugs described in this message For more information on including jar processor options in your build, refer to this document prepared by the core team. This overall jar signing document is also a good reference.

The process we use can be summarized as follows:

1. Use scp to copy the eclipse-master-${buildId}.zip to the eclipse.org signing staging area using pserver.
2. Invoke the signing script.
3. Poll the server for the signed file in the output directory. Poll from the same server that you will be copying from. I initially polled build.eclipse.org but copied the file from dev.eclipse.org to take advantage of its QoS riles. This caused problems because of the lag in nfs synchronization across the nodes. So I changed it to poll dev.eclipse.org and copy from dev.eclipse.org.
4. When available, scp the signed eclipse-master-${buildId}.zip back to the server.
5. Delete the build specific signing files from build.eclipse.org.
6. Invoke the packager against the eclipse-master-${buildId}.zip to create the the drops available for download, and continue with the remainder of the build.

<target name="compareAttribs">
     <!--poll file for change in attributes-->
     <exec dir="${buildDirectory}" executable="ssh" outputProperty="polledAttribs">
         <arg line="${sshline}"/>
     </exec>
     <echo message="original:  ${originalAttribs}" />
     <condition property="attribChanged">
          <not>
	       <contains string="${polledAttribs}" substring="No such file or directory" casesensitive="no" />
	  </not>
     </condition>
          <echo message="polled:  ${polledAttribs}" />
	 	<antcall target="writeDiffResult" />
	 	<sleep seconds="120" />
	 		<available property="attribs.changed" file="${buildDirectory}/attribDiff.txt" />
	 	<antcall target="waitForChangedAttribs" />
</target>

<target name="writeDiffResult" if="attribChanged">
     <echo message="original: ${originalAttribs}" file="${buildDirectory}/attribDiff.txt" />
     <echo message="new: ${polledAttribs}" file="${buildDirectory}/attribDiff.txt" append="true" />
</target>

<target name="signMasterFeature" if="sign">		 		 		 		 
	<property name="archiveName" value="eclipse-master-${buildId}.zip" />
	<property name="packtmp" value="${buildDirectory}/packtmp" />
	<property name="stagingDirectoryOutput" value="/our/staging/directory/${buildId}-out"/>
	<property name="stagingDirectory" value="/our/staging/directory" />
	<property name="outputFile"  value="${stagingDirectoryOutput}/${archiveName}"/>

	<mkdir dir="${packtmp}" />		 		  
	<move file="${buildDirectory}/${buildLabel}/${archiveName}" tofile="${packtmp}/${archiveName}"/>
	<!-- add pack.properties file that specifies effort level -->
	<exec dir="${eclipse.build.configs}/../../extras" executable="zip">
		<arg line="-r ${packtmp}/${archiveName} pack.properties" />
	</exec>
	<!--push drop to staging directory-->
 	<echo message="push drop to staging directory"/>
	<exec dir="${packtmp}" executable="scp" output="signing.txt">
  		  <arg line="${archiveName} dev.eclipse.org:${stagingDirectory}"/>
	</exec>
 	<exec dir="${buildDirectory}" executable="ssh" output="signing.txt" append="true">
	<arg line="build.eclipse.org /bin/chmod ugo+rw ${stagingDirectory}/${archiveName} "/>
	</exec> 

	<!--invoke sign script and wait-->
 	<echo message="invoke sign script and wait"/>
		<exec dir="." executable="ssh" output="signing.txt" append="true">
 	  		<arg line="build.eclipse.org "cd ${stagingDirectory}; /usr/bin/sign ${stagingDirectory}/${archiveName} mail ${stagingDirectoryOutput}""/>
		</exec>
		
	<!--Wait for signed build to be available -->
	<antcall target="waitForChangedAttribs">
               <param name="sshline" value="dev.eclipse.org "cd ${stagingDirectoryOutput};ls ${archiveName}"" />
        </antcall>

	<!--copy zip back to build machine -->
	<echo message="copy zip back to build machine"/>
	<exec dir="." executable="scp" output="signing.txt" append="true">
		<arg line="dev.eclipse.org:${stagingDirectory}/${buildId}-out/${archiveName} ${buildDirectory}/${buildLabel}"/>
        </exec> 

        <!--delete files on build.eclipse.org-->
 	<echo message="delete temp files on build.eclipse.org"/>
 	<exec dir="." executable="ssh" output="signing.txt" append="true">
		<arg line="build.eclipse.org "/bin/rm -rf ${stagingDirectory}/${buildId}-out ${stagingDirectory}/${archiveName}""/>
	</exec> 
	</target>		 		

After signing the master feature, we call the jar processor again to create the pack.gz files. This step could be completed on the build.eclipse.org server. However, this would mean that the eclipse-master-${buildId}.zip would be much larger file to copy back to our build machine and subsequently take longer. Consequently, we decided to run this task on our local build machine.

<target name="packMasterFeature">
     <property name="archiveName" value="eclipse-master-${buildId}.zip" />
     <property name="packtmp" value="${buildDirectory}/packtmp" />
     <mkdir dir="${packtmp}" />	
     <move file="${buildDirectory}/${buildLabel}/${archiveName}" tofile="${packtmp}/${archiveName}"/>
		
		
    <!-- update location of jvm arguments --> 
    <replace file="${eclipse.build.configs}/../../extras/pack200" token="@pack200@" value="${java15-home}/bin/pack200"/>
    <chmod file="${eclipse.build.configs}/../../extras/pack200" perm="755"/>
		
    <!--condition jar if it is not pushed to eclipse.org for signing-->
    <condition property="repack" value="-repack">
	<not>
		<isset property="sign" />
	</not>
     </condition>
     <property name="repack" value="" />

     <!--pack200-->
     <java jar="${eclipse.home}/startup.jar"
          fork="true"
	  timeout="10800000"
	  jvm="${java15-home}/bin/java"
	  failonerror="true"
	  maxmemory="512m"
	  dir="${buildDirectory}">
          <jvmarg value="-Dorg.eclipse.update.jarprocessor.pack200=${eclipse.build.configs}/../../extras"/>
			<arg line="-application org.eclipse.update.core.siteOptimizer"/>
			<arg line="-jarProcessor -outputDir ${buildLabel} -processAll -pack ${repack} ${packtmp}/${archiveName}"/>
       </java>
	
	<delete dir="${packtmp}" />
</target>