Equinox/p2/p2.mirrorsURL

Disclaimer: this page is a community maintained page, not part of the p2 project, per se, and is fairly specific to the Eclipse.org infrastructure. The p2.mirrorsURL concept was implemented by p2 team primarily for the Eclipse.org Infrastructure, to allow p2 updates and installs to use its worldwide system of mirrors, though other companies or networks can use in a similar way for their specific mirroring system.

Thanks to Stephan Herrmann for recognizing the need and getting this page started, updating it occasionally with new information, and trying to keep us all 'honest'. Such contributions make Eclipse better.

Enabling your repository to leverage mirrors

The p2.mirrorsURL should be added to your metadata so that p2 will see the list of available mirrors to choose during installation.

Who should add?

Everyone. Every build. At least, everyone at eclipse.org, that is. Other companies or networks may have their own mirroring system and scripts which are similar to this, but these instructions are specific for the Eclipse Foundation infrastructure.

It is true that not each and every build or repository is mirrored (e.g. integration builds usually are not) but having the property there, even if there are no mirrors, should not hurt anything ... p2 knows to simply fall back to the normal "originally requested" location if no mirrors are found (usually 'http://download.eclipse.org/...').

It is also true that those mirroring eclipse downloads and repositories have some choice in what they mirror (there are several options, such as "full", "releases only", "platform only" and similar). So, everyone should add the property, since you can never be sure if/when your repository might be mirrored. If it is not mirrored, by anyone, at some point in time, having the property there should not hurt anything ... p2 knows to simply fall back to the normal "originally requested" location if no mirrors are found (usually 'http://download.eclipse.org/...').

Where to add?

Adding this property to artifacts.xml is sufficient, since p2 uses a list of mirrors while retrieving artifacts, not the metadata itself. (Hence, no need to have one in content.jar/xml or compositeContent.jar/xml files). Don't forget, artifacts.jar must be updated or replaced after changing artifact.xml. (If both exist, artifacts.jar will always be fetched before artifacts.xml.)

Should composite repos be treated, too, or just their children (leaves)?

Just the leaves. No need to to specify the property in compositeArtifacts.xml/jar.

What to add?

The p2.mirrorsURL property has the following structure:

  <property name="p2.mirrorsURL" 
          value="http://www.eclipse.org/downloads/download.php?file={repository_path}&amp;format=xml"/>

• replace {repository_path} with the path where your artifacts.jar sits on download.eclipse.org,
  e.g., for http://download.eclipse.org/eclipse/updates/3.6/R-3.6.2-201102101200/artifacts.jar the {repository_path} would be /eclipse/updates/3.6/R-3.6.2-201102101200
• yes, & is 5 characters, in this context the escaped version is required.
• the last part, &protocol=http is important, since it restricts the list of mirrors to those that support http. Monitor bug 349679 for status/progress on p2 working with ftp sites. (In general, such as for zip files linked on download pages, and similar, the 'protocol' parameter is optional for the download.php script and when omitted both http and ftp mirrors are returned).
  bug 349679 seems to have been a red-herring (or some special case) so ftp and http should both work fine, so there is no need to specify the protocol attribute and the mirrors script (download.php) will return a list that contains both types ... and the more there are to try, the better.

This p2.mirrorsURL property will end up giving p2 a list of 0 to 100 or so URLs to try and fetch artifacts from. If your project is not mirrored (or, there is some other problem with mirrors) p2 will fall back automatically to using the originally requested site location (usually 'http://download.eclipse.org/...').

How to add?

via site.xml

Note: site.xml files are pretty "out dated" and normally should not exist on the repo site itself, even if you use it temporarily with some old tools to produce your metadata, its recommended you not leave on the repo site itself. The reason is, is that in some rare circumstances, p2 may misinterpret this and think it has found an old update manager site and try to handle it as such.]

If you use a site.xml the word goes that this is where you should specify this property. The Equinox/p2/Publisher should automatically transfer this information into your metadata.

If you don't use a site.xml you may consider manually editing artifacts.xml (which of course can be automated using, e.g., an xsl template, like this p2.xsl from the Mylyn project). When manually editing artifacts.xml add the property to the existing list of properties right at the top of the document. Don't forget to update the size attribute of the enclosing <properties> element.

copying properties from existing repositories during mirroring

Others have reported that the p2.mirror ant task can be used to insert properties like p2.mirrorsURL while mirroring the build output to the final destination. The following two snippets may serve for inspiration:

<!-- generate template with proper mirror urls -->
<copy
  file=".../template-artifacts.xml"
  tofile="${buildDirectory}/publishRepoTemplate/artifacts.xml">
  <filterset>
        ... specify string substitutions here ...
  </filterset>
</copy>

<!-- mirror to target (but only the bits that really changed) -->
<p2.mirror ...>
  <source location="..."/>
  <destination .../>
  <destination
    kind="artifact"
        compressed="true"
        name="${p2.repo.name.prefix} (Build ${buildId})"
        location="..."
        append="true"
        format="${templateRepoUrl}"/>
  ...
</p2.mirror>

These snippets were extracted from this build.xml from the gyrex project.

Custom Eclipse applications

Some projects have written Eclipse applications to set or change or remove the p2.mirrorsURL property, such as see WTP/Releng/Tools/addRepoProperties as one example. You might be able to re-use this code, as is, or feel free to copy (to your own namespace) and customize for your project.

How to avoid .blobstore for *.jar.pack.gz files

Technically, repositories that use .blobstore to store pack.gz files can be used just like any other repository with the same p2.mirrorsURL attribute as described in this page. (See bug 383145.) But, many people prefer to avoid .blobstore, since it goes into a "dot directory" (hidden by default for most systems and tools) it can be confusing to some people and tools.

In some scenarios and tools, apparently, the .blobstore form is never created, and the pack.gz files stored as siblings of the .jar files, without any action being needed.

But, if you find your scenarios and tools are creating one, and you wish to avoid it, you basically need to specify a property in your artifacts.jar/xml before creating or mirroring the pack.gz files:

<property name="publishPackFilesAsSiblings" value="true/">

One easy way to do this, if you are starting from scratch, is to have a "near empty" artifact.xml file, and then in your p2.mirror "destination" attribute, specify you'd like to use that near empty artifact.xml file, via the "format" attribute, to have it used as a template from which to get properties from ... publishPackFilesAsSiblings being one of the possible properties.

For example, the Platform builds use this sort of template when getting started:

<?xml version='1.0' encoding='UTF-8'?>
<?artifactRepository version='1.1.0'?>
<repository name='Build Template' type='org.eclipse.equinox.p2.artifact.repository.simpleRepository' version='1.0.0'>
  <properties size='2'>
    <property name='publishPackFilesAsSiblings' value='true'/>
    <property name='p2.mirrorsURL' value='http://www.eclipse.org/downloads/download.php?file=@p2RepoPath@/@BUILD_ID@&format=xml'/>
  </properties>
</repository>

And, don't worry, it sounds more complicated that it is ... once you get it set up the first time.

How to test?

If you change your artifacts.jar/xml file, you may have to wait a a while, for mirrors to get "in sync" before being able to test, such as a few hours, or at most overnight. Seriously, it might "appear" to be mirrored, but not, so give it at least half a day before starting to worry.

Avoid Mirror invalidation
In some cases, it is easy to temporarily "invalidate" your mirror list; that is, make it appear, temporarily, that there are no mirrors. This happens if you change the directory timestamps that contains the artifact.jar as you edit. It is the directory that is used in p2.mirrorsURL, so it will appear there are no mirrors if the timestamp changes to current time, even though all the mirrors still have your actual artifacts. But -- this is the tip -- the directory's timestamp only changes if files are added or removed, but not if simply the content of a file changes. So, you can avoid temporarily invalidating mirrors by copying the artifact.jar file to some 'temp' directory, unzip it it there, update it, and zip it back to artifacts.jar and then copy it back to the original location (all carefully of course). If you do somehow invalidate the mirrors list temporarily, it should not take too long to get some mirrors back (a few hours, or overnight for sure), since there's no real work to do ... just update the fact that they are in sync.

Confirm your artifacts site is being mirrored

The first test you do should be with a browser, to confirm the download.php script does indeed return some mirrors for your site. To do this, paste the full URL you specified as the value in the p2.mirrorsURL attribute into your browser (now collapse &amp; to &). The result should be an XML file in your browser with a list of mirror URLs. For an example, you can paste the following URL into your browser, to see who is mirroring the Juno SR1 artifacts:

http://www.eclipse.org/downloads/download.php?format=xml&file=/releases/juno/201209280900/

Pasting one of these resulting URLs into the browser should in turn take you to the directory holding your artifacts.jar and its peers. (You may have to add "artifacts.jar" to the end of the URL, if directory browsing is not allowed on a particular mirror).

Note that not all "builds" (or repositories) are mirrored. For example, nightlies, integration builds, staging sites, etc. are explicitly excluded. See IT Wiki page for details.

Confirm your p2.mirrorsURL works with p2 operations

You can not test from eclipse.org
For those with shell accounts, or those thinking it might be good to test as part of a Hudson job ... you can not. The "internal network" at the Eclipse Foundation and the "mirroring system" knows not to try and get something from a mirror, but instead to get it directly ... since it is right there, close by, just 3 Gbs away. Hence, to test if your mirrors are working as expected, you need to test from you own network. There might be similar issues or limitations on your company's network (hypothetically) if they have their own redirection/proxy/mirroring systems in place. If in doubt, you can ask.

To test whether install successfully uses mirrors (and which ones) you may use the following options file (stored, e.g., as /tmp/p2options or simply placed in .options file in 'eclipse' directory (the default location)):

 org.eclipse.equinox.p2.core/debug=true
 org.eclipse.equinox.p2.core/artifacts/mirrors=true

(The first line generally enables debugging p2, the second line selects specific tracing for mirror selection)

Then you may want to unpack a fresh Eclipse package and change into the eclipse directory.

Finally, use the normal Eclipse UI to do an "update" or for more focus, you can use the p2 Director application to install your feature using a command line similar to the following:

  ./eclipse -consoleLog -noSplash -debug /tmp/p2options -application org.eclipse.equinox.p2.director \
        -repository http://download.eclipse.org/myproject/myrepo \
        -installIU org.eclipse.myproject.myfeature.group

This should install your feature and print to the console where each artifact was downloaded from. If p2.mirrorsURL is not properly configured no such messages will be shown.

You may want to "capture" the console results in a separate file by adding something like the following to the end of your 'eclipse' command (at least on Linux):

2>&1 | tee fulllog.txt

Once "mirror selection tracing" is in such a log file, you can produce a quick "histogram" of frequencies of what mirrors are being selected with a command similar to the following (which was inspired by Markus Knauer in bug 372504#c9):

grep Selected fulllog.txt | cut -d ":" -f 5- | cut -d "(" -f 2 | cut -d "/" -f 1,2,3 |  sort | uniq -c | sort -r -n 

If things are working well, you should see few, if any, times that "download.eclipse.org" is selected. As one example, here are the results of one run, updating Indigo SR1 Java EE IDE to SR2 from the built-in repositories:

     92 http://mirror.cc.columbia.edu
     54 http://mirrors.med.harvard.edu
     50 http://mirror.cc.vt.edu
     43 http://mirrors.ibiblio.org
     43 http://carroll.aset.psu.edu
     26 http://download.eclipse.org
     24 ftp://eclipse.c3sl.ufpr.br
     20 http://ftp.jaist.ac.jp
     17 http://mirrors.xmission.com
     16 http://ftp.ussg.iu.edu
     13 http://www.gtlib.gatech.edu
     11 http://ftp.osuosl.org
      8 http://mirror.aarnet.edu.au
      8 http://ftp.yz.yamagata-u.ac.jp
      5 http://eclipse.mirrorcatalogs.com
      3 http://mirrors.ustc.edu.cn
      1 http://eclipse.stu.edu.tw

When not to use the p2.mirrorsURL property?

Only if you are developing a repository for your own "private" or company use and do not want it to go to other mirrors for any reason. This is certainly legitimate for non-Eclipse projects. But, also, occasionally companies mirror an Eclipse repository for their own internal use, and if they do not want their internal users to be redirected to other sites, they may need to "manually" remove the attribute from their mirrored version of the repository. Follow (or contribute to) bug 262005 for requested enhancements to the p2.mirror task to make this easier.

Can the p2.mirrorsURL property be harmful?

Yes. If it is incorrect. For example, if someone has released a "3.7" version, but their p2.mirrrorsURL value still points to "3.6 repositories" it will result in a lot of "wild goose chases" for artifacts it will never find and end up going back to eclipse.org anyway (and causing the bandwidth problems there, it was designed to help with).

So, please make sure your values are accurate and work as expected. Nicolas Bros, in bug 368826#c15 contributed this handy "one liner" for listing your p2.mirrorsURL values in your subdirectories of repositories (which could be easily re-written as a non-one-liner, if you desire to have in your own script file):

 find . -type f -name 'artifacts.jar' -or -name 'artifacts.xml' \
    | while read i; \
 do echo "$i"; \
    if [[ "$i" =~ .*/artifacts.jar ]]; \
    then echo "  "$(unzip -p "$i"|grep "p2.mirrorsURL"); \
    else echo "  "$(cat "$i"|grep "p2.mirrorsURL"); \
    fi; \
 done

Plus, there has, in the past, been discussions if the use of mirrors by p2 is a good thing, such as see the p2-dev list, but until the Eclipse Foundation gets its own world-wide server farm, it is the only workable solution known for the millions of installs and updates that Eclipse users do each year. (Note, it was primarily designed for installs and updates ... and it is "using it for builds" that appear to be what is mostly complained about). Not to mention, many improvements and bug fixes have been made over the years, and all users and committers are encouraged to open new bugs or feature requests (and/or contribute patches) if they find bugs or inadequate behavior.

Avoiding mirrors, even when using p2.mirrorsURL

You can disable the automatic mirror selection mechanism by setting the VM argument eclipse.p2.mirrors to false (remember that vm args have to be last on the command line preceded by the switch -vmargs). Normally this is not desired or recommended since it may be slower or, if many people did it, all the time, cause Eclipse Infrastructure to slow down for everyone and overuse bandwidth. But, it can be used occasionally for tests if you suspect a problem with mirroring (for example, if it appears you are getting old or stale content you might want to try by-passing mirrors temporarily, to test that.

Interesting bugs or pages with tools attached

Many of these have not been used/updated for a while, so they are listed here for information only. Its hoped you can find the tools useful (or help refine them) but use at your own risk ... and on non-production test systems, at least initially.

bug 269218 - Mirror Validator

bug 310132 - Publisher app + ant task should support statsURI option

WTP/Releng/Tools/addRepoProperties