WTP/Releng/Tools/addRepoProperties

addRepoProperties

Introduction and Overview

There is a tool in wtp releng that can modify p2 repository properties. This is especially useful for modifying these properties "after the fact", such as to give a final name to a repo, change or remove the p2.mirrorsURL value after a repository has been moved, etc.

While it is in the WTP code repository, it has been used for various things in Orbit and the yearly Simultaneous Release as well, so moderately reliable.

The tool can be used (or copied) by any project, but if you do, test well, on local test repos first, and early in dev cycle, and "use at your own risk" as it is not a fully supported API or tool with unit tests, etc. It uses only p2 APIs so should work with about any version of Eclipse SDK.

Getting or installing

Getting Source

The source code itself is in the cvs repository /cvsroot/webtools in a module under webtools.releng/plugins/, the module is named org.eclipse.wtp.releng.tools (that is, that's what you would check out as your Eclipse IDE project).

The feature used when building it is similar, but under webtools.releng/features/, and named org.eclipse.wtp.releng.tools.feature.

Installing from releng repo

To install an already built version into your IDE or SDK, you can use the repository at

http://download.eclipse.org/webtools/releng/repository/

And install the "Releng Tools Feature", its ID is org.eclipse.wtp.releng.tools.feature and at the time of this writing at the 1.2.0 level.

In install with p2 Director, you could use something similar to

 eclipse/eclipse \
    -application org.eclipse.equinox.p2.director \
    -noSplash -consolelog -debug -console \
    -repository \
 http://download.eclipse.org/webtools/releng/repository/ \
    -installIUs \
    org.eclipse.wtp.releng.tools.feature.feature.group

General Usage and Examples

In general, this tool operates directly on the artifacts repository, typically the "leaf" version (i.e. simple repository, not composite) where your "artifacts.jar" file is. You must have write access to that repository and use the absolute file path (i.e. not accessed over http or anything).

The parameters are all specified with -D system properties passed to the VM (currently).

The repository location is the only parameter that required. For most of the rest, if not specified at all, then any existing values are not changed. If specified with an empty string, this is taken as an indication to remove the property (i.e. doesn't just change the value to an empty string).

General Setup

After actually installing the tools into Eclipse SDK, the tool is typically invoked with a script such as the following ... though you should be able to run it from your IDE for local testing. I give this example script here, just for illustration, then we'll focus on the properties and different examples.

#!/usr/bin/env bash
BUILD_HOME=${HOME}/testbuildhome
JAVA_6_HOME=${HOME}/jdks/ibm-java-x86_64-60
APP_NAME=org.eclipse.wtp.releng.tools.addRepoProperties
devworkspace="${BUILD_HOME}"/addRepoPropertiesWorkspace
devJRE=${JAVA_6_HOME}/jre/bin/java
# remember, the '&' should NOT be unescaped here ... the p2 api (or underlying xml) will escape it. 
devArgs=" -DartifactRepoDirectory=/tmp/testartifactrepo/ \
    -Dp2ArtifactRepositoryName=ServiceRelease2Artifacts \
    -Dp2MirrorsURL=http://www.eclipse.org/downloads/download.php?format=xml&file=/project/releases/repository/ \
    "

echo "   cmd:          " $0
echo "   APP_NAME:     " $APP_NAME
echo "   devworkspace: " $devworkspace
echo "   devJRE:       " $devJRE
echo "   devArgs:      " $devArgs
echo
$devJRE -version
echo

eclipse/eclipse  -debug -nosplash -consolelog -console \
   -data $devworkspace --launcher.suppressErrors -application ${APP_NAME} \
   -vm $devJRE -vmargs $devArgs

Example of changing repo name and p2.mirrorsURL

From the example above, note the three properties specified for this run, via -D:

artifactRepoDirectory=/tmp/testartifactrepo/ 
p2ArtifactRepositoryName=ServiceRelease2Artifacts 
p2MirrorsURL=http://www.eclipse.org/downloads/download.php?format=xml&file=/project/releases/repository/

Assuming there is a valid artifact repo in /tmp/testartifactrepo the output of the run would look something like the following:

INFO: artifact repository directory set from 'artifactRepoDirectory': /tmp/testartifactrepo/
INFO: metadata repository directory set from 'metadataRepoDirectory': null
INFO: No metadata repository was specified. If desired, set the system property 'metadataRepoDirectory'.
INFO: the artifact repository had existing name: Indigo artifacts
INFO: p2ArtifactRepositoryName value set from 'p2ArtifactRepositoryName': ServiceRelease2Artifacts
INFO: the repository had existing mirrorURL: http://www.eclipse.org/downloads/download.php?format=xml&file=/releases/indigo/201202240900/aggregate/
INFO: mirror URL value set from 'p2MirrorsURL': http://www.eclipse.org/downloads/download.php?format=xml&file=/project/releases/repository/
INFO: the repository had existing statsURI: http://download.eclipse.org/stats/releases/indigo
INFO: stats URI value set from 'p2StatsURI': null
INFO: p2StatsURI was not specified. Any existing values not changed.
INFO: No artifacts requested to be tracked
INFO: repository rewritten

Notice the "INFO" is telling you what it found and did, what existing values were found, and what they were changed to. At the end, it says "repository rewritten" to make explicit it found changes to make and did re-write the repository file.

If the exact same job were run again, on the already changed repository, the output would be as below Notice the application knows now sometimes prints the "found" and "existing" values "were the same". So, it didn't change anything, so does not "rewrite" the repository. It calls that a "warning" under the assumption you ran a job to "change properties" so you probably expected something to happen.

INFO: artifact repository directory set from 'artifactRepoDirectory': /tmp/testartifactrepo/
INFO: metadata repository directory set from 'metadataRepoDirectory': null
INFO: No metadata repository was specified. If desired, set the system property 'metadataRepoDirectory'.
INFO: the artifact repository had existing name: ServiceRelease2Artifacts
INFO: p2ArtifactRepositoryName value set from 'p2ArtifactRepositoryName': ServiceRelease2Artifacts
INFO: new value and existing value of artifact repository name are the same
INFO: the repository had existing mirrorURL: http://www.eclipse.org/downloads/download.php?format=xml&file=/project/releases/repository/
INFO: mirror URL value set from 'p2MirrorsURL': http://www.eclipse.org/downloads/download.php?format=xml&file=/project/releases/repository/
INFO: new value and existing value of mirrorURL are the same
INFO: the repository had existing statsURI: http://download.eclipse.org/stats/releases/indigo
INFO: stats URI value set from 'p2StatsURI': null
INFO: p2StatsURI was not specified. Any existing values not changed.
INFO: No artifacts requested to be tracked
WARNING: repository was not rewritten

Example of removing p2.mirrorsURL property

Occasionally, people make a "mirror" of an Eclipse mirror to their local network, and when they do, they typically do not want p2 to try to fetch things from other mirrors, but only from their local network. This application allows removing the property if the value for p2MirrorsURL is empty.

Continuing the above example, if I now run the job with

devArgs=" -DartifactRepoDirectory=/tmp/testartifactrepo/ -Dp2MirrorsURL="

The output would be

INFO: artifact repository directory set from 'artifactRepoDirectory': /tmp/testartifactrepo/
INFO: metadata repository directory set from 'metadataRepoDirectory': null
INFO: No metadata repository was specified. If desired, set the system property 'metadataRepoDirectory'.
INFO: the artifact repository had existing name: ServiceRelease2Artifacts
INFO: p2ArtifactRepositoryName value set from 'p2ArtifactRepositoryName': null
INFO: p2ArtifactRepositoryName was not specified. Any existing values not changed.
INFO: the repository had existing mirrorURL: http://www.eclipse.org/downloads/download.php?format=xml&file=/project/releases/repository/
INFO: mirror URL value set from 'p2MirrorsURL': 
INFO: the repository had existing statsURI: http://download.eclipse.org/stats/releases/indigo
INFO: stats URI value set from 'p2StatsURI': null
INFO: p2StatsURI was not specified. Any existing values not changed.
INFO: No artifacts requested to be tracked
INFO: repository rewritten

Note that the repo name was unchanged, since we didn't specify it at all, but the p2.mirrorsURL property would be completely removed. You could check the artifacts.jar file and see the property was actually gone, not just set to empty string.

Save XML version if you want to compare
If your artifact repo is jarred (as most should be) and you unzip it to take a look, and thus have artifacts.xml file there in that same directory, then these operations to change your properties and write a new artifacts.jar file will also delete the artifacts.xml file it finds there. So, if you unzip your artifacts.jar, to artifacts.xml, and later want to compare it to your changed artifacts.xml, you should change your original to another location or name.

Examples of changing "stats" properties

If you've read the above examples closely, you'll see a bunch of info about "stats" and related. There are two aspects to "keeping stats" from a p2 repository. First, a general property of the repo must enable it for that repository. Also, specific features (or, any IU) must be marked as being desired to be tracked (remember, only a few things from each repo should be "tracked" ... else the download server would be overwhelmed with stats messages.)

Enabling a repository

Assuming IUs have already been marked for tracking, only the repo must be "enabled". This is what we do with the simultaneious release repositories. When they are "promoted" to be released the p2StatsURI is given a value such as following (again, as -D parameter):

p2StatsURI=http://download.eclipse.org/stats/releases/juno

Even in "staging" we give it a value, though mostly for tests (its is of little practical value):

p2StatsURI=http://download.eclipse.org/stats/releases/staging

For common repo, we assume every project is responsible themselves for marking their own IUs for tracking, how ever they see fit (and, those of course purposely get "copied over" from contributed repositories, to the final, common repository).

Marking features to be tracked

For Webtools Project, we use this application to mark 4 IUs for "tracking", in addition to "enabling" our own project repository. So, for Webtools we use (again, as -D properties):

p2StatsURI=http://download.eclipse.org/stats/webtools/repository${STATS_TAG_VERSIONINDICATOR} 
statsArtifactsSuffix="${STATS_TAG_SUFFIX}" 
statsTrackedArtifacts=org.eclipse.wst.jsdt.feature,org.eclipse.wst.xml_ui.feature, \
   org.eclipse.wst.web_ui.feature,org.eclipse.jst.enterprise_ui.feature

The most important, new value in this example is the statsTrackedArtifacts, which lists the artifacts to be tracked. I think we don't even use the statsArtifactsSuffix any longer, but think what was added on to the tracking property in each IU, so we could, if desired, mark easy to find releases such as if from SR0, SR1, SR2, etc.

The ${STATS_TAG_VERSIONINDICATOR} at the end of p2StatsURI property, for webtools, is just the main release, indigo, juno, etc.

See Equinox_p2_download_stats for more information about tracking your p2 downloads (but, it is purely optional ... you are not required to collect stats on p2 downloads, and as far as I know, not too many projects do).

Tracking artifacts always rewrites repo
The repo will always be rewritten if statsTrackedArtifacts is specified (and the IUs found) even if it has already been marked and written once with exact same values. This doesn't really hurt anything except changing time stamps, but just wanted clarify since it is different from the other cases.

References

IT_Infrastructure_Doc

Equinox/p2/p2.mirrorsURL

Equinox_p2_download_stats

bug 310132 To give credit and thanks, the inspiration and starting point for my "repository manipulation application" efforts were inspired by the applications attached to this bug (and probably others I've forgotten). Thanks to all who have come before. May my own application inspire even greater ones!