GPW Lightning Retrospectives

This page collects notes for Lightning Retrospectives prepared for the Ganymede Provisioning Workshop. You won't need to make slides to present at the workshop. We'll provide access to these pages and a big font. Use the following template and digress as you see fit.

IBM Lotus Expeditor

• See www.ibm.com for a comprehensive discussion of Lotus Expeditor capabilities.
• Supports desktops, laptops, kiosks and mobile devices.
• Supports windows mobile devices, Linux, Windows.
• Supports remote mount of workspaces and installed sites. Every path is relative.
• Supports single user installs or multiuser installs.
• Support provisioning of multiple versions of features, including different JVMs through update.
• Remote management by Expeditor Server through Portal or Enterprise Management Agent.
• Local management by CD or Update manager.
• Network install, CD install, Web install.
• IBM products based on Expeditor include Sametime 7.5 and Notes 8.

What we have done: We have layered our provisioning code on top of the update core. Where we needed new functionality in the core we created patches and gave it back to eclipse. Most of the changes to the core were for the managed-only update policy. We may be the only ones using managed only policy since it was an unfinished idea when we started working with it.

What do we want: We have told our customers for 3 years that features are the way to manage provisioning. If features do not go forward then and equivalent capability needs to be defined.

Deployment issues associated with bundle artifacts needs to be addressed.

• OSGi restart capabilities are great, everyone wants that. However there are conditions that do require a restart. The framework needs to manage these requirements so restarts can be minimized. System artifacts that might require action are setting of system environmental variables, registry settings. Any large implementation will likely have native code that will require restart of eclipse platform, log off, or even restart the system. The simplest case is the eclipse launcher. How do we support replacement and restart?
• We currently use install handlers heavily to manage feature artifacts. System variables, Java properties, etc. may need to be managed by/for bundles. Each JVM bundle defines its startup properties. Any framework developed to manage these artifacts needs to include the launcher.

Multiuser install also has unique requirements.

• Configurations have to be explicitly managed. An example of implicit change is the configuration manager changing versions because of timestamps or because a plugin has been added to a site.
• Deployment has 2 phases for multiuser that can't be overlapped. An administrator or his proxy must be able to control the bundles that are deployed into the platform sites. Users or their proxy need to be able to selectively configure the bundles installed by the admin. The default policy used by update core violates this by overloading the install operation with configure operations.

Update UI

• The update UI needs to be abstracted in such a way that it is easy to replace or modify the UI.
• Some of functionality in UI should be relocated to the core -- for example update, and revert.

Language packs.

• There should be a standard way to install language packs.

See also: Lightning Retrospective Guide were similar notes were collected for the build workshop.

Equinox Provisioning Incubator

The evolution of Eclipse running in a wide range of runtime scenarios (e.g., tooling, RCP applications, cell phones and devices, and server) coupled with the increasingly dynamic nature of these computing environments is driving new requirements in the area of provisioning. Eclipse Update Manager has served well in relatively static, large-block, non-OSGi, tooling world but these additional requirements are exceeding the flexibility designed into Update Manager.

In response to this the Equinox team started an incubator to look at technologies and approaches to both meet the needs we see today and be flexible enough to meet needs we have yet to see.

The main goals of the incubator are to create an "extensible provisioning platform" that provides the base infrastructure and building blocks for provisioning systems that cover a wide range of usecases. Provisioning solutions lie along several intersecting axes.

• fully client managed to fully server managed
• coarse-grained "product" delivery to fine-grained "function" delivery
• static component/version lineups to dynamically computed lineups
• client-side component isolation to component pooling

The Equinox provisioning platform seeks to facilitate (though not necessarily directly provide) provisioning solutions at multiple points along each axis by separating the base provisioning mechanisms from the policies that drive those mechanisms.

The initial scope of this work is to create the base platform and enough concrete implementations to viably replace the current Eclipse Update Manager.

http://eclipse.org/equinox/incubator/provisioning http://wiki.eclipse.org/index.php/Requirements_for_a_new_update_manager

Maya Incubator Project

Maya is a new project to Eclipse being contributed from Cisco. While new to Eclipse, Maya is a technology developed within Cisco based on past provisioning experiences including working with technology used to manage thousands of Eclipse desktops. Maya is our ground-up rewrite to do managed provisioning the "right" way (disclaimer: Maya is our approximation of "right"; we look forward to further refining and evolving our understanding here at the workshop).

The Maya technology is being contributed to Eclipse as part of the new Maya project.

• Maya is currently being used in a limited rollout within Cisco.
• Maya was designed from day one with the intention of releasing to open source.
• Maya architectural design is robust; certain individual components still maturing.
• The technology is developed following Eclipse conventions for modularity and extensibility.

In approaching Maya, our experiences were based on past experiences using both Update as well as another internally developed provisioning platform used for a certain Eclipse-based tool stack.

• End-user empowerment is key -- but not the base requirement. Many users of Eclipse tooling couldn't care about how the software gets provisioned, instead, they are interested in accessing the software they need.
• Many users within an organization need to share a base stack of Eclipse tooling. In Maya we abstracted this notion into Profiles which can be shared by users. Users can then extend (add) additional software onto a Profile.
• Discovery of software sources in an enterprise has been problematic; in Maya, we took the burden central for managing the list of update sites to be scanned and made available through Maya.
• Users need to be able have access to multiple Eclipse Profiles ("stacks") at the same time.
• Update technology when looking to address delivery of complex RCP tools needs to be significantly easier than what we would expect a developer to go through.
• Provide a base framework that allows different vendors to offer value-add services to be deployed both server and client side (such as licensing, tool stack validation, etc.).

From a selfish point of view, we want to be able to pick up software from vendors and quickly deploy on our network without requiring changing out our provisioning or distribution infrastructure, or requiring users to manually install software on our system.

As a technology, Maya addresses some standard problems for Eclipse provisioning (in no particular order).

• Single entry-point into multiple Eclipse profiles including software updates.
• Automatic selection and failure/retry between download mirrors.
• Ability to override mirror lists for Enterprise-internal download sites.
• Plug-in and feature caching to be shared over multiple Eclipse instances.
• Automation of dependency resolution when defining software for an Eclipse instance (Profile).
• No need for end users to find or specify update site URLs as handled by administrator.

We are looking at the following for Ganymede:

• An extensible model for provisioning in Eclipse whether single end-user deployments or centralized managed deployments.
• A much simplified model for distribution better suited to do RCP applications in addition to the mode developer centric provisioning model of today.
• A common paradigm for products to be defined and deployed without requiring customized installers or custom client-side technology for each system.

In addition to the notes above, we recommend reading over the following links and potentially browse the Maya newsgroup for some additional architecture discussions.

• http://www.eclipse.org/proposals/maya/
• http://www.eclipsemaya.org/

IBM Installation Manager

In Q4 2006 IBM released the IBM Rational Software Delivery Platform 7.0 family of Eclipse-based products employing a new internally developed installation technology, the IBM Installation Manager. Some of the high level goals of the Installation Manager were

• A simplified, unified user experience for software provisioning
• Support for delivery of componentized software packages into shared environments
• Support for enterprise management and governance of the distribution of developer tools

The Installation Manager (IM) is an RCP application which is installed via bootstrap on client machines.

Architectural principles in the IM include the separation of decisions about what is be provisioned (Director function) from the steps taken to do the provisioning (Engine function) and the separation of the description of provisionable components (metadata) from the data to be provisioned (artifacts). We have implementations of metadata, transports, metadata and artifact repositories, extensibility paradigms, adapters for eclipse and native provisioning, and other parts of the IM which may be appropriate for contribution (either as code or concept) to Eclipse provisioning (particularly to the Equinox provisioning incubator).

Our unique experience comes from the larger end of the provisioning scale both in size and diversity of software packages (~7500 bundles, fragments, and features; lots of native artifacts) and in number of deployed managed clients (many 1000's). Some random areas related to our experience are build integration; consistency of install, update, rollback, fix, and uninstall scenarios; support of constrained, tested lineups; shell-sharing; qualification of install configuration (install into existing eclipse); silent install; media repositories; licensing and fulfillment system integration; and compatibility of provisioning with open source, partner, and third party components.

We want the provisioning in the Ganymede timeframe to provide an extensible, functional framework on which open source and commercial specialized provisioning systems can be built that will work reliably and consistently in a heterogeneous enterprise environment.

Smartup software update

Smartup is the provisioning solution for our non Eclipse based Smart Client technology. These technologies are the basis for large client/server sales representatives systems. Our largest system is within the financial/assurances area and has about 30.000 clients.

Features are:

• the end user is (and shall) in no way concerned by updating the software. This includes keeping track of installed versions and the versions to upgrade to as well as automatic detection of the communication properties such as internet settings, HTTP proxies, LAN, mobile,..
• ensure that the end user is always working with the latest version of the client software. Update cycles are very frequent (at least one per month).
• for identifying software ´artifacts´ (jars, dlls, documents, ..) versioning schemes and/or MD5 hashes are used.
• works well with low bandwidth connections (cell phone connections/ISDN). To minimize transport data the software update is capable of delivering updates as deltas between the outdated and the current version.
• updating of all components (application, software update itself, the os-specific native launcher and JRE)
• the software update server maintains all the applications and their different versions within a repository. Deliverables are packaged and handled similar to servlet war files. On server start and/or when requested by a client, deltas will be created between the current version on the client and the target version on the server.
• with profiles it is possible to divide an application into parts with limited capabilities. This allows e.g. focusing on specific tasks and obviously it minimizes the data needed for updating the software. Each of those parts (they can overlap) is in itself a working client.

We are currently working on extending our existing technology into the Eclipse/RCP ecosystem. Therefore we would like to share our experience and code if possible/feasible. We are also willing to support the eclipse project with man power.

Published articles concerning software update and Smart client technology in the German Java Magazin

same automatically translated

Eclipse Packaging Project (EPP)

The Eclipse Packaging Project is a new project at eclipse.org and will provide its first results with the Europa Release. Based on the assumption that the Eclipse SDK download package is the most popular download at eclipse.org, but that package is not what most users need for their work, the Packaging Project provides ready-to-install packages for different user profiles.

An additional feature of the Packaging Project is the installer: The project will provide not only raw zip or tar packages, but also native platform executables which contain an installer bundled with the install package.

Initial packages will be build for

• Java Developer
• Java EE Developer
• C/C++ Developer
• RCP Developer
• ...

Additional information:

• Project homepage
• Project proposal

The Packaging Project can provide package definitions and installers for the Ganymede Release. We have to decide where we can share our experience, which parts are of particular interest for other projects (Maya, Equinox Provisioning Project, ...), and where we need a common language (e.g. package definition).

Buckminster Project & Spaces Project (proposed)

Buckminster, an existing Technology project, provides a feature-set for dynamic configuration and provisioning an arbitrary Eclipse developer workspace. Buckminster is based on extensible frameworks that are intended to reuse existing build & assembly tools. For example, Buckminster can call PDE build mechanisms to build features, bundles and update sites. The Buckminster and Maven teams are now collaborating to use Buckminster as a mechanism for Eclipse users to discover and materialize components from Maven repositories.

Buckminster is intended to be complementary to Eclipse mechanisms for configuring and provisioning Eclipse runtimes. Our primary objectives in attending this conference is to contribute to a common understanding of how Eclipse runtime and workspace provisioning mechanisms relate and to ensure that Buckminster is able to leverage the next generation of run-time provisioning technology.

A specific topic we are interested in exploring is the feasibility of using a common component description to describe all types of components (features, bundles, binaries, source) across runtime or development. Buckminster's CSPEC, which describes dependencies and attributes (actions, artifacts, groupings of attributes) and their relationships, is our attempt to develop such a model in the context of workspace provisioning. We are interested in learning how our model can be improved and integrated with the new provisioning architecture.

We also represent the proposed Spaces project, which is creating an Eclipse feature-set that allows streamlined publishing of Eclipse-developed content to project infrastructure outside of Eclipse.org. (AOL is providing the first reference implementation of (and free community access to) of server-side project infrastructure.) We plan to provide a discovery service as part of this project and are hoping that the new update manager was able to communicate with that service. Buckminster currently includes code to with component discovery today, which we are planning to contribute.

Yoxos Install Manager (YIM)

The Yoxos Install Manager is a tool for managing features in an Eclipse installation. Users can search, choose and install features from a large library of installable features (>1000 features, >3500 plug-ins). The YIM comes with its own user interface (a perspective) and is build with Update Manager technology. A summary of our experiences in no particular order:

• update manager core works
• scalability is an issue (careful construction of update sites; cannot install too many features at once; number of features can be overwhelming)
• feature includes can cause problems (see paper)
• dependency info in features and bundles can be inconsistent (plug-in provider's fault, DRY, better tooling?)
• hard to recover from a broken installation
• better error handling and reporting in UM API (parsing stack traces to find out what went wrong, throwing exceptions that you can ignore)
• two different APIs (programmatic vs command-line)

Linux Distros

One of the project's goals is to "bring Eclipse technology to Linux distribution users." To achieve this, most of the major distributions (Fedora, Debian, Gentoo, Ubuntu, SUSE, RHEL, etc.) are now including the Eclipse SDK and starting to ship other features/plugins (ex. CDT, PyDev, RSSOwl, Azureus, Mylar, ...). Some points I'd like to touch on:

• distros can loosely be defined as collections of packages (libraries, apps, whatever)
• packages take the form of RPMs, .debs, .ebuilds (sort of), and others
• model: pristine source
• ability to express inter-dependencies, versioning, support for hooks
• lots of existing mechanisms and infrastructure
• experience different from other OSs
• don't want to disrupt existing management/verification tools
• community distros emphasize upstream cooperation - not really "products"
• integration could also help with delivery of ISV products
• issues: multi-user installations, difficulty in packaging
• selfish issues: JBoss, multi-OS