Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

COSMOS RE AND BUILD

Introduction

This page provide information and links for COSMOS Release Engineering.


Quick Start

export LANG=C
export emailadr=<your@email.address>
ssh <user>@build.eclipse.org
cd /shared/technology/cosmos/bin
./start-build.sh

Build Overview


The COSMOS build is based on Eclipse PDE (Plug-in Development Environment). The Plug-in Development Environment (PDE) provides tools to create, develop, test, debug, build and deploy Eclipse plug-ins, fragments, features, update sites and RCP products. PDE is built atop the Platform and JDT, and ships as part of the Eclipse SDK. The goal of PDE Build is to facilitate the automation of plug-in build processes [3][4][5] . Essentially, PDE Build produces Ant scripts based on development-time information provided by, for example, the plugin.xml and build.properties files. The generated Ant scripts, can fetch the relevant projects from a CVS repository, build jars, Javadoc, source zips, put everything together in a format ready to ship and send it out to a remote location (e.g., a local network or a downloads server It runs in the eclipse server.

An automated nightly build runs everyday, the build can also be manually started. The COSMOS build status (Failed/Completed/Progress), build logs, build downloads, dependencies etc can be accessed through a web based dashboard. The builds dashboard is available at http://build.eclipse.org/technology/cosmos/downloads/builds.php

2a) Build Environment
COSMOS build is performed on the eclipse server. The build initially fetch the build module from CVS to the workspace directory. The fetched build module contains various Bash/Ant scripts to perform the complete build. These Bash/Ant scripts broadly perform following functions:

1) Fetch plugins/features from CVS
2) Compile the fetched code to .class files
3) Assemble the .class files to .jar file
4) Package the jars as .zipped file
5) Publish the zipped files to COSMOS download page.

2b) COSMOS Deliverables
Following are the deliverables, which are available at the cosmos downloads site: http://www.eclipse.org/cosmos/downloads/

1. SML validator and SML-IF Editor (Resource Modeling)
This package consists of an SML validator that validates SML and SML-IF files, an editor to create and validate SML-IF documents used to exchange information between systems management applications, and tools for importing and exporting SML data to and from a flat file repository.

2. COSMOS Demo and Runtimes
This download contains an application to demonstrate some functions provided by the COSMOS project. It also includes the COSMOS runtimes.

3. SDD Tooling
This package includes a build time generator for creating and merging Solution Deployment Descriptor (SDD) files as well as a component called the change analyzer that will parse SDDs and display what will be done by that SDD in human readable HTML format.

4. SDD Runtime
The SDD runtime project includes a driver for parsing SDDs, resolving deployment related tasks defined in the SDD and performing the associated operations.


Build Process

Build starts automatically and the manual build has to be initiated by a person. If a build is in progress, another build cannot be started. A ".busy" file is created when the build starts and is removed automatically once it is over. The build output contains the deliverables, log files and reports.

The Release Engineering Team will be responsible for running the builds and tests to completion. In the event of problems with the build & test system, they are expected to be resolved as soon as possible and get the build and test restarted. In the event of problems with the build caused by developer checkins it is followed up with developers to ensure fixes are made quickly, and then the build and test are restarted. Any build failure will be reported through bugzilla to the build component for tracking the actual issue as well as trend analysis, etc. All committers are responsible for clean builds and tests. Anytime a check-in is made, the respective person is responsible for monitoring the builds and tests until they pass.

At the end of a successful build, the build output is published to the eclipse server as development build, as part of the build process.

Once the build progress through the different phases, it evolves into stable or release builds. Build type can be changed by editing the file "www/cosmos/downloads/1.1.0/publish_1.1.0.txt" in eclipse repository "dev.eclipse.org:/cvsroot/org.eclipse". In the publish file "publish_1.1.0.txt", build type is displayed as {development/candidate/stable}|{build id}|{comments}, for example stable|COSMOS-1.1.0-200708171630|i5 stable build (Release 0.5)


3a) Build Schedule
3a.1) Daily Builds
An automated build runs every 24 hours. The state of the head branch of the CVS repository will be labeled with a time stamp. The time stamp will be encoded with the release number, date, time, and serial build number, eg: COSMOS-RMm-YYMMDDHHMMSS. A module list will be maintained (in CVS) that lists all the modules that are part of the current build, labeling and extracting will be restricted to those modules.


3a.2) Manual Builds
Apart from the daily automated builds, a manual build can be started by those who are authorized to do so.
Starting the manual build on eclipse server:
- Login to eclipse build server (build.eclipse.org) console
- build directory on eclipse server is located at "/shared/technology/cosmos"
- build is started by executing the ./startbuild.sh in "/shared/technology/cosmos/bin" directory

Refer section 'Porting Build' below for more details on folder structure etc .
Refer section 'Access to Eclipse Server' to configure password less key based access to eclipse server.

Currently the dashboard doesn't provide any option to start a manual build. This might be added later.

3b) Build Types

Following are various build types


3b.1) Development Builds
All successful daily builds are published to the eclipse server as development builds. Development builds are produced from whatever has been released into the HEAD stream of the CVS repository. They are completely untested and may contain major problems. These drops are normally only useful to developers actually working on this project.


3b.2) Candidate Builds
Selected daily builds are promoted as candidate builds for testing, based on the iteration schedule. They can be promoted to either milestone or release builds.


3b.3) Stable Builds
Candidate builds which pass all the test phases & bugs fixed are flagged as stable builds. Stable builds are tested drivers delivered at the end of each iteration of the development cycle. The latest stable build is the correct build for people who want to stay up to date with what is going on in the latest development stream, and don't mind putting up with a few problems in order to get the latest greatest features and bug fixes. Stable builds are also known as milestone builds


3b.4) Release Builds
According to the release schedule the stable builds are released to the community. Releases are the right builds for people who want to be on a stable, tested release.


Note: Tagged vs Untagged Builds
When a build is tagged, the artifacts used in build are tagged in CVS using the build ID. This is for easily reproducing the build if needed. All nightly builds are tagged builds. Manual builds can be run either as tagged or untagged. Usually all candidate builds are run as tagged build and others as untagged one.


Build Setup

The build is setup to run on the public eclipse server (http://build.eclipse.org/technology/cosmos/). The build status, logs and build builds can be accessed at the dashboard (http://build.eclipse.org/technology/cosmos/downloads/builds.php). The build is performed at RedHat Enterprise Linux release 4 (Nahant Update 5) and IBM Java 1.5.

The Build is based on following dependencies

1)Eclipse SDK
2)Dojo Toolkit
3)Orbit Bundles
4)Equinox Bundles
5)Web tools Bundles
6)Axis2-1.3 binaries

Adding a new dependency or modifying an existing one is done by changing the following files:
a) build.properties - in dev.eclipse.org:/cvsroot/technology/org.eclipse.cosmos/releng/org.eclipse.cosmos.releng.builder/runtime
b) customTargets.xml - in dev.eclipse.org:/cvsroot/technology/org.eclipse.cosmos/releng/org.eclipse.cosmos.releng.builder/runtime


Build Repositories (CVS)
1) dev.eclipse.org:/cvsroot/org.eclipse - www/cosmos (eclipse server web interface)
The web pages, displayed at cosmos download page (http://eclipse.org/cosmos) are located here. These html/php pages can be added or edited

2) dev.eclipse.org:/cvsroot/technology - org.eclipse.cosmos (build artifacts & rel engg)
The build scripts, are located here and can be modified to reflect the change in builds. To setup build you should get the latest files from here. The entry point for build is make.sh.


Porting Build

The release engineering team has completed the process of transferring COSMOS build environment to the eclipse server. The build directory on eclipse server is located at "/shared/technology/cosmos".

At above location, there are following directories:
1) downloads (on build completion, folder named COSMOS-$Buildld is created and drivers, console/CVS change logs are saved in this folder)
2) workspace (various plugin/features are first fetched to this workspace directory, which are then compiled and packaged)
3) bin (contain various shell scripts used in build)
4) config (contain local configuration files, which specify the build parameters local to build)
5) dependencies (contain various dependencies, based on which the build is constructed)
6) logs (contain logs, used for debugging the build)

The build is started by executing the ./startbuild.sh in bin directory. A cronjob is set on eclipse server to start the nightly builds.

Steps to make build portable on another machine:
1) Make the directory structure similar to what described above
2) Copy all the required files in directories
3) Set parameters in /config/local-build-config.sf file, tuned to local environment setting. For example set the correct JAVA_HOME, ECLIPSE_BUILDER etc

Additional required steps are:
1) comment out the J2SE-1.4 and J2SE-1.5 at the org.eclipse.cosmos.releng.builder/runtime/build.properties and uncomment the J2SE-1.4 and J2SE-1.5, which points to /shared/common/..
2) comment out the cvsloc=cosmos:/cvsroot/technology and uncomment cvsloc=:pserver:anonymous@dev.eclipse.org:/cvsroot/technology org.eclipse.cosmos.releng.builder/make.sh


Access to Eclipse Server
Only authorized users have access to eclipse server. To make password-less connection to eclipse server, such that server do not prompt for password, the eclipse server should be configured to trust the local machine. Here are the steps to make that happen:

On your client machine: 

mkdir ~/.ssh
chmod 700 ~/.ssh
cd ~/.ssh
ssh-keygen -t rsa {leave the passphrase empty}
chmod 400 id_rsa
scp id_rsa.pub dev.eclipse.org:

On dev.eclipse.org: 

mkdir ~/.ssh
chmod 700 ~/.ssh
cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
rm ~/id_rsa.pub

Connecting CVS with SSH
The build machine includes an SSH client for accessing the CVS server. Different methods can be used to authenticate, depending on the level of functionality and security. Authentication methods used by the client by default are, in the following order: public-key, Keyboard-Interactive, and password authentication.

Public-key authentication allows the client to connect to a remote CVS server without sending the password. This is a more secure authentication method than password authentication. Public-key authentication uses two keys, a private key that it should be kept in a secure place and protected with a password. And the public key, is placed on the server (CVS), one wish to gain access to.

Moving build to a different server

The objective here is to create a fully working COSMOS build on a machine which replicates the build on the IBM eclipse server.

Requirements

Red Hat Enterprise Linux release 4 is used on the Eclipse server, but you can use Centos Linux 4.6 as it is functionally equivalent to RHEL 4 and its free whereas RHEL is not.

Access to a local CVS repository containing a replica of the the Cosmos CVS repository org.eclipse.cosmos

IBM Java 1.5. This also needs to eventually work with the Sun JDK 1.5. (TBD)

Download eclipse-SDK-3.3.2-linux-gtk.tar.gz

(Optional) A local Apache Server or any locally accessible web server. The purpose of this is to have the build dependencies accessible locally. This is not essential and is only necessary if you have difficulty bypassing a firewall do retrieve the dependencies during the build process. This will reduce the download time during the build process.

Machine setup

1)      Install Linux distribution, add default Apache server from distribution if desired.

2)      Log in to root and create the required folder structure with following commands:

mkdir -p /shared/technology/cosmos/

mkdir /shared/technology/cosmos/bin

mkdir /shared/technology/cosmos/config

mkdir /shared/technology/cosmos/downloads

mkdir /shared/technology/cosmos/tmp

mkdir /shared/technology/cosmos/logs

mkdir /shared/technology/cosmos/workspace

mkdir /shared/technology/cosmos/dependencies

3)      Unpack Eclipse 3.3 into /shared/technology/cosmos/

Eg.

cd /shared/technology/cosmos/

tar xvfz <download location>\eclipse-SDK-3.3.2-linux-gtk.tar.gz

4)      Install the IBM Java and record where it is installed. There are many ways to do this. The version I have was packaged in an RPM and its default install location is /opt/ibm/java2-i386-50.

Build dependencies

The build depends on some third party software, which it downloads during the build process if necessary. If you are behind a firewall it the scripts may not be able to connect externally to download these dependencies so the easiest solution is to download them and store them separately on a local web server and then configure the build to access that server .

If you are able to access the dependencies without proxy or firewall configuration, skip to the Build Configuration step.

Install and start a default Apache server instance from the Linux distribution.

Download all of the following dependencies:

Copy all of these downloaded files to the /var/www/html folder.

The location of these dependencies is specified in build.properties, which is explained in the following Build Configuration section.

Build Configuration

The build relies on customisations in property files, map files and the scripts. Before the build can be started some changes have to be made to the CVS repository where all the build scripts are located.

Make sure you know the connection method and a username and password to connect to CVS. A cvs connection string is used in many places so I will define it here as follows:

cvsconnstring =:<connection type>:<username>:<password>@<server-host>:<repository-path>

for example if cvs is at server-host cvsserver and path /var/cvs and your connection type is pserver and username/password is cvs/cosmos then your cvs connection string would be:

cvsconnstring=:pserver:cvs:cosmos@cvsserver:/var/cvs

Make a note of this as it will be used in most file edits performed in this step.

In cvs, edit the following highlighted files in the releng/org.eclipse.cosmos.releng.builder project:

bash/make.sh

Make sure CVS_RSH and cvsloc are set with the correct CVS connection details using this syntax:

export CVS_RSH=<connection type>

cvsloc=<cvsconnstring>

bin_scripts/cvs-bash.sh

Make sure CVS_RSH is set with the correct CVS connection type used in bash/make.sh you defined above. Like below:

export CVS_RSH=<connection type>

bin_scripts/start-build.sh

Search for a line that starts with “./cvs-bash.sh” and perform these changes.

Change the –cvscmd parameter to point to your connection string

Change the –emailadr parameter to an email address to receive the build results

bash/local-build-config.sf

Make sure CVS_RSH is set with the correct CVS connection type used in bash/make.sh you defined above. Like below:

export CVS_RSH=<connection type>

Make sure JAVA_HOME is set to location of your Java 1.5 JDK

export JAVA_HOME=/opt/ibm/java2-i386-50

Make sure the entries for ECLIPSE_BUILDER and DEP_PATH are set as follows

export ECLIPSE_BUILDER=/shared/technology/cosmos/eclipse

export DEP_PATH=/shared/technology/cosmos/dependencies

runtime/build.properties

Find the J2SE-1.5 property and change all entries to point to the location of IBM java. It should look like this:

J2SE-1.5=/opt/ibm/java2-i386-50/jre/lib/BD.jar;/opt/ibm/java2-i386-50/jre/lib/charsets.jar;/opt/ibm/java2-i386-50/jre/lib/core.jar;/opt/ibm/java2-i386-50/jre/lib/deploy.jar;/opt/ibm/java2-i386-50/jre/lib/graphics.jar;/opt/ibm/java2-i386-50/jre/lib/ibmcertpathprovider.jar;/opt/ibm/java2-i386-50/jre/lib/ibmcfw.jar;/opt/ibm/java2-i386-50/jre/lib/ibmjaaslm.jar;/opt/ibm/java2-i386-50/jre/lib/ibmjcefw.jar;/opt/ibm/java2-i386-50/jre/lib/ibmjgssprovider.jar;/opt/ibm/java2-i386-50/jre/lib/ibmjsseprovider2.jar;/opt/ibm/java2-i386-50/jre/lib/ibmorb.jar;/opt/ibm/java2-i386-50/jre/lib/ibmorbapi.jar;/opt/ibm/java2-i386-50/jre/lib/ibmpkcs.jar;/opt/ibm/java2-i386-50/jre/lib/javaplugin.jar;/opt/ibm/java2-i386-50/jre/lib/javaws.jar;/opt/ibm/java2-i386-50/jre/lib/security.jar;/opt/ibm/java2-i386-50/jre/lib/server.jar;/opt/ibm/java2-i386-50/jre/lib/vm.jar;/opt/ibm/java2-i386-50/jre/lib/xml.jar

If you are using Sun JDK 1.5, the J2SE-1.5 entry will be different to that above. That entry has not been defined yet and will appear here when known.

Each dependency’s location is stored in this file. If you are using a local web server to store the dependencies then you should change the entries for eclipseSdkUrl, birtURL, orbitUrl, WTPUrl, EMFUrl, gefUrl and equinoxUrl to your local web instance, in my case this is <a href="http://localhost">http://localhost</a>.

runtime/customTargets.xml

It is only necessary to change this file if you cannot download through a firewall.

Find the postPackage target.

Change the references to http://eclipse.org/legal/epl-v10.html and http://eclipse.org/legal/epl/notice.html to http://localhost/epl-v10.html and http://localhost/notice.html.

maps/cosmos.map

Each line in this file is comma delimited list of parameters.

Change the second parameter on each line to your connection string defined earlier.

maps/sdd.map

Each line in this file is comma delimited list of parameters.

Change the second parameter on each line to your connection string defined earlier.

Performing Build

When all the above steps are complete, make sure all changes are commited to CVS. Copy all scripts from the bin_scripts folder in the releng/org.eclipse.cosmos.releng.builder project into /shared/technology/cosmos/bin folder.

Place the bash/local-build-config.sf in the /shared/technology/cosmos/config folder

To perform build:

cd /shared/technology/cosmos/bin

./start-build.sh

Results and monitoring

The build can take a few minutes to complete.

On completion a mail is sent to the address specified in the start-build.sh script, indicating success or failure.

All the build objects are placed in folders /shared/technology/cosmos/downloads/COSMOS/1.1.0/COSMOS-1.1.0-YYYYMMDDhhmm. Where YYYYMMDDhhmm represents the start date and time of the process.

The log file console.txt is created in /shared/technology/cosmos/downloads/COSMOS/1.1.0/COSMOS-1.1.0-YYYYMMDDhhmm on completion.

Tagging the Builds

Tagging PDE-Based Builds

It is assumed that a tagged COSMOS build has been performed and that it will be the root of the new tag. Here are the following steps to perform tagging

1) Login into eclipse build server and change directory to "/shared/technology/cosmos/tmp". Create new directory in this by name of "v1_0_0_i13" and do cd to it.

2) Execute following command

cvs -d cosmos:/cvsroot/technology ex -r <tagged-build-id> org.eclipse.cosmos/releng/org.eclipse.cosmos.releng.builder/maps

Replace <tagged-build-id> with id (for example COSMOS-1_0_0-200809110902).
There are three map files. Tagging need to be done with all three map files.

3) Copy "tag.sh" file to this directory from "/org.eclipse.cosmos/releng/org.eclipse.cosmos.releng.builder/bash".

4) Change "tagname" and "tagged_buildid" variable values in this file and then execute this shell script. Once execution is over, check the log files (log01, log02, log03, log04, log05) created in the same directory for any errors.

Note: test folder is tagged from the HEAD stream


References

[1] Eclipse SDK help
[2] http://www.sshkeychain.org/mirrors/SSH-with-Keys-HOWTO/SSH-with-Keys-HOWTO-4.html
[3] http://www.eclipse.org/pde/pde-build/
[4] http://www.eclipse.org/articles/Article-PDE-Automation/automation.html
[5] http://help.eclipse.org/help32/index.jsp?topic=/org.eclipse.pde.doc.user/guide/tasks/pde_feature_build.htm
[6] http://wiki.eclipse.org/CVS_Howto


Build Contacts

The release engineering team is responsible for starting the build. In case of any build failure, please contact the release engg team. Following are the release engineering team members:

CA, Inc.:
Bradley Beck (bradley.beck at ca.com)
IBM:
Saurabh Dravid (sadravid at in.ibm.com)

Links to Bugzilla

Open bugs against Release Engineering team

COSMOS Test-automation process

http://wiki.eclipse.org/COSMOS/COSMOS_TEST_AUTO

Retrieved from "https://wiki.eclipse.org/index.php?title=COSMOS_RE_AND_BUILD&oldid=185686"

This page was last modified 17:59, 19 January 2010 by Eclipsepedia anonymous user Unnamed Poltroon.

Back to the top