Equinox Summit 2007 Results

The Equinox Summit 2007 was arranged as a set of lightning talks and a series of breakout sessions. Each session had a number of parallel discussions with self-assigned and self-organizing participants. The groups then reported to a plenary session, summarizing their results and possibly adding more topics to be discussed in further breakouts.

Day One Results

First breakout meeting

Component Programming Models

• Service helper: ideal for one-time use.

• Spring
  • Injection model - no code to acquire the service
  • Client is handed a proxy to the service object
  • Framework manages binding of a real service to the proxy dynamically
  • Distinguish between mandatory and optional service requirements
    • Mandatory: your bundle isn't started until all the mandatory services are available
    • This only guarantees availability at a moment in time: at the moment of startup all mandatory services are available, but they could disappear after that moment.
    • If a mandatory service goes away, the services exported by the bundle needing the service is unregistered.
  • For stateful services, there is a listener model - bind an unbind events occur as service implementations arrive and depart
  • It would be interesting to be able to declare a default service implementation that would be used whenever no real service is available
• Spring and DS require eager activation, but only for service producers. Many bundles are just service consumers, and so they can be lazily started
• Requirements for Spring service support: Spring core, Spring AOP, a few other essential Spring core pieces
• Service Application Toolkit: IBM toolkit for making it easier to work with OSGi services
• How do we reconcile and combine the various component toolkits available: Spring, DS, SAT, extension registry
• Spring initializes the app context asynchronously. Typical Eclipse programming model assumes that all initialization has completed by the time any classes are loaded in that bundle.
• To be scalable, we need a model where service wiring is persisted across sessions, so we don't have the massive event storm of services being started and bound as the system comes up.
• Most bundles are POJO - how do they specify that they need Spring or some other component system to be available at runtime? Ideally 95% of bundles don't know or care about the active component system - but they need some component system to be around at runtime to inject the services they need.
• Can we come up with a common syntax where a bundle can specify its service and extension registry imports/exports, and be agnostic about the runtime that will parse it and provide the runtime capabilities.
• Scoping: extension registry / services are global - how can I publish extensions/services that are only available in a particular scope* only particular set of bundles have access to it. In J2EE world, sometimes only want to register/bind services within a particular web app. Scoped registries - local versus global extension registry.

p2 Design Deep Dive

Interesting topics:

• Profiles/ hierarchy of profiles

• Headless agent; managed installs

• Everything is possible

• Version range lineups,

• Security? JAAS

• Resolver - where does it go? when?

• Plugable download manager

• Plugable communications - within ECF. replacing ECF?

Tooling (builds)

• Enhancements to current export
  • pack/unpack/sign support
  • export to multiple targets (exporting platform dependent fragments)

• Current Export
  • leave behind generated files for debugging purposes

• Project Builder (can we get this into the pde.incubator)
  • a new builder project (like existing update site project)
  • Provides a mechanism to bootstrap from export to headless build
  • generates artifacts (features/build.properties/customTargets.xml/allElements.xml) that are input into pde.build
  • provides forms/editors (or wizards) to expose pde.build options
  • Consider different inputs : .product file, launch configuration
  • enable gradual discovery of customizability: customTargets.xml, plugin level build callbacks.
  • Synchronization between workspace and plugin level build properties.

• PDE/Build proper: selected refactorings to make project builder easier
  • remove need for allElements.xml

Server Side Web apps++

Pain points:

• PDE Tooling lacking - new server app -- need help from WTP
• initialization (listeners and lifecycle)
• filters (Spring authentication)
• Security -- Authentication and authorization
• Annotations Servlet 2.1 JEE - EJB 3.0. Can we support this.
• Concerns with existing 3rd party server side components - typically need some work - Buddy Class Loading

• How can we do ejb
• what about jms - unclear if we can do pure jms style apps. because of current servlet focus
• Spring use is common. We need watch this area.

• Cool ideas:
  • Embedding OSGi lower down in the app server. In particular below EJB layer. e.g. per EAR OSGi framework

• Why uses server-side OSGi ?
  • Better component model
  • p2 !!!

• RFC 66 -- unclear if this really replaces http service
• In most cases just putting off the bundlizing of your app

• Organizational - JEE JCP / OSGi EEG

Second breakout meeting

API Tooling

• Clarify definition of API (what is API, what is API compatibility)
  • Public API (as defined by Jeem's API white paper)
  • Internal code
  • Provisional API
  • SPI - Service provider interfaces
  • Eclipse-specific API such as extension point ids, other data in extension/extension-point markup
• Tool to annotate code with API rules (javadoc tags that specify if a class may be implemented, instantiated, etc)
• Build API rule file from javadoc tags
• Want both IDE integration, and headless build usage
• Scalability
  • Be able to run API tools on smaller components/bundles/pieces rather than the whole world
  • Store results in various forms: XML files, or perhaps database for large scale use
  • API data may be stored locally or remotely
• Discussion on whether API rule files could replace JDT access restrictions. It's really a super-set of access rules, can express things that acess rules cannot
• API usage reports: could have simple HTML pages, or Eclipse integration into views/editors, like the Emma Eclipse plugin does for coverage data
• Support for API tooling in other languages and .so/DLL files
• Use of reflection is problematic because it's hard to detect programmaticaly

Server Side Remote Management

Provisioning will handle and cover much of the use case around managing the provisioning of servers.

The notion of a "nano" entity for initial provisioning will be useful,

A entity that can be tickled to check, verify and rectify the state of an equinox runtime.

There is not much available for managing totally non-running systems. Systems will need to have something running to talk to.

There are several frameworks to add on top of running osgi systems for managing the installs JMX, MX4J, Remote Console, DMTAdmin.

Things to look into: -DMT Admin: There is no spec available? -OSGi Monitoring Spec.

Day Two Results

Third breakout meeting

ECF/Transports/Repositories

We started with a discussion of requirements for the p2 download manager and transport layer:

• Robustness of download manager
  • Mirroring
  • Different repositories have different characteristics - some repositories can support many connections
  • Adaptive re-prioritization - if you are downloading 1GB of data, you may need to switch to a different mirror dynamically
  • Weighting of repositories based on latency, availability, etc.
  • For downloading many small pieces, may be more efficient to combine them into a single large artifact for download purposes (minimizing round-trips)
• BitTorrent:
  • Not good for many small files
  • Great for Eclipse release periods where there are large download peaks.
  • Could work well in a corporate environment where a large group of people are provisioning the same software around the same time.
• Where do we embed all this intelligence - in a single uber-smart download manager, or should some of that smarts be hidden away in repository implementations? Could have a single smart repositories that acts as a facade to a group of backing repos.

ECF:

• Scott: ECF is just intended to be a slim asynchronous transfer API - while some of these details could be handled at the transport level, some of this complexity should be built in a layer above ECF
• Requirement for ECF: be able to introspect download stats in order to make adaptive decisions, alter repo weightings, etc. Want to be able to ask a transport up front for its anticipated throughput/latency, and get progress details that allow to update with actual throughput/latency as transfer progresses
• Also need new API from ECF to start artifact downloads at a particular offset in an artifact. This is needed both for resume of downloads, and for multi-threaded download of a single file (each thread downloads a chunk, potentially from different mirrors, and then chunks are reassembled upon completion of download). See bug #205011.

Getting more concrete on the design and how the pieces fit together. First cut: Intelligence and design making largely in download manager. However, repository implementations need to be able to answer various questions in order for the download manager to be able to make its decisions:

• Memory and disk space requirements
• CPU time requirements
• Latency/throughput of transport
• Download manager also needs to account for local machine characteristics - how many concurrent downloads can it run, etc

Aiming for a relatively simple download manager with pluggable download strategies. Can start with a simple d/l strategy much like our current implementation. Then be able to plug in a much fancier d/l strategy that does mult-threading, adaptive behaviour, etc.

Work items:

• Modify current d/l manager to have pluggable strategies
• API for asynchronous interaction with repositories
• API for querying progress of repos
• Repo API for starting artifact downloads at some offset in the artifact

p2 Tooling

"Everything is possible with enough contributions"

self-hosting has been difficult historically

• provision from workspace
• imagine target platform + your workspace are one big repository
• push from workspace
• "Export -> metadata repository"

build tooling

• ability to batch IU generation
• are we generating a repository as an archive?
• all plug-ins are pushed and published in a repository
• archiving of a repository is a build operation
• installing agent and installing and downloading a runnable piece should give the same thing

deployment tooling

• when do we integrate the generator during build/exporting?

authoring tooling

• long-term we want people to author IUs (editors)
• probably not for 3.4
• for 3.4, we will probably have some sort of feature.xml -> IU translation

development tooling

• hints, IUs, feature.xml -> generator => IUs

repository tooling

• manage content (browsing, expiration)
• simple APIs for repositories (addIU, removeIU)

other concerns

• remote generation of metadata and remote management of clients
• what is the "runtime"? is provisioning included in "runtime"?
• how do we manage remote runtimes? DMT admin?
• workspace would be a "management console" using a specific API
• No cost for people doing plain plugin development
  • pull model: running instance pulls from workspace implicitly-generated metadata
• Are IUs self-contained?
  • we have no support for translation right now
  • currently we're replicating common information across IUs
• How do we do translation of IUs?
  • tooling support
• Dynamic tooling is a separate topic
  • are my bundles really dynamic?
  • will removing my bundles remove them completely?
• feature transitioning (branding, icons, etc.)

User Authentication

We started by going over the current state of JAAS and trusted/non-trusted bundles experiment

• Trusted bundles: seems like everybody liked the concept, this is the good level of "trust" granularity, having bundle signed is good enough to establish initial "trust" state

• Role-based access control: this is what everybody asked about. It needs to be done on multiple levels:
  • Framework (bundles "appear" and "disappear" depending on who's looking)
  • UI (perspective, views, menus)

It should not become too fine grained like Java permission models (finer grain -> administration headache)

• Need an "administration":
  • "policy" on what user can install (especially new extension fragments)
  • associate names <-> roles <-> permissions

• Login timing: different users have different needs
  • Server side: "late" login (app is well and running)
  • Shared install: login before workspace selection (so that it can drive workspace and preferences based on the user)

• Server-side: JAAS approach itself is not sufficient for authorization. Role-based model of implemented by framework and UI might help

• Cool new idea: login determining user preferences

• Real keystore and secure data storage would be nice to have
  • Question: what keystore format should we use

• OS-based single login can work with headless runs

Framework Deep Dive

Why is ContextClassLoader hard?

---

• in an enterprise situation you always know the context (this war, this ear).
  • all lower level libraries can get the context
• OSGi has no clear entry point, you don't know who your client is
• the specification is mute on the subject of CCL. It defaults to the system class loader
• Spring OSGi does some manipulation of the CCL. Ensures that CCL can see bundle classes during activation. Can also manage CCL on service invocation (configurable to service client bundle, or service provider bundle)

ContextFinder

---

Equinox has the ContextFinder that tries to find the context of who is calling. CCL is set to the ContextFinder. When forName is called, the *caller's* bundle is asked to load the class.

Note: you can use the context finder on other OSGi implementations.

BuddyPolicy

---

The Equinox buddy policy allows you to attach as a buddy of another bundle. e.g. to say that you are a buddy of Hibernate. Buddy policy does not required classes to be exported in order for them to be visible.

The "dependent" policy says anyone who depends on me directly or indirectly is a buddy. The "registered" policy forces bundles to say they want to be a buddy of X.

There is no way for a bundle A to declare that it wants access to private classes in bundle B.

What if bundle A doesn't know that bundle B will need to access it? For example, a framework creates this requirement without A's knowledge. Peter Kriens has a blog entry describing some possible solutions to this.

One proposal is to trap "ClassNotFoundException" and then go hunting for matching classes. Has a weakness that would force classes to be loaded this way to be exported. Can be problems if two bundles both have a private type with the same qualified name.

Some problems with the Buddy mechanism:

• If two buddies both have an internal class C, when the buddy bundle asks for C, which should it get?
• Allows static references, not just dynamic ones. We noted that the Eclipse tooling will prevent static references if you develop using eclipse.
• Class.forName caching

Class.forName

---

Class.forName model is broken. Always call loadClass instead. Class.forName caches the resulting class (in the VM) in both the loading and calling class loader caches.

Other ContextClassLoader issues:

---

Equinox sets the CCL to the ContextFinder. Inherited by spawned threads. If any bundle ever sets the CCL to something different and forgets to set it back, this thread will never recover. In some app servers it might recover on next request (but not all, e.g. OC4J).

System Bundle Fragments

---

There are 2 types:

1. To supply extra classes to the framework implementation

 Added for conditional permission admin, types that are sensitive from a security perspective and should not be loaded from a bundle.

2. Boot class path extensions. (Adding things to VM boot classpath). For example, if you need to add types in the "java" namespace. This is not implemented in equinox. Requires vm restart.

An "Extension fragment" is an extension to the fragment classloader.

Other:

---

The EclipseStarter code is not pretty.

Fourth breakout meeting

p2 Engine

• Install IUs:

1. 1. collect (download)
   2. install
   3. configure

• IUs are installed by a particular touchpoint. Have touchpoint data
• Engine talks to touchpoint

• How to contribute "actions"
  • actions are things like unzip, copy, change file permissions.

• Each touch point registers actions that other touchpoints can use
  • do actions have preconditions?
  • are they context free
  • how to contribute them
  • dependencies between actions -> are they transactional

• Touch point data contains javaScript (Ant?, perl?, contributable?)
• Actions are implemented in Java
  • each language used in touch point data "connector" that can expose these java actions

• Reaction to cancel? (failure) -> cancel itself is long running
  • rollback/undo (hard -> backup old artifacts that are overwritten etc...)
  • How long to keep undo data, undo a whole install?

p2 User Interface

• Currently update sites are global, but should that information be profile-specific?
• Could just have a browser-based UI. Use a Browser widget within Eclipse, and hook link handling to initiate installs/updates from UI
• Update: no UI needed. Just a dialog asking if you want to restart
• Better support for browsing/filtering the list of available things to install. Organize by topic, filter like preference dialog
• Would be nice to have an aggregator for update sites, so there is some central registry to browse for available software (such as EPIC)
  • However, we can't really resolve such a large set of data because the resolution algorithm likely won't scale up that much

Springing into Equinox

Two topics were discussed: Springs and Service Activator Toolkit (SAT)

Springs

An overview of Springs was provided:

• Service provides, service consumers, proxies, dependency injection, all good things :-)
• Springs has "smoothing" effect hiding services coming and going in a highly dynamic system (this works for proxies themselves, but not necessary for objects obtained from those proxies)
• Supports "scopes", in particular "bundle" scope added for OSGi

Distribution:

• Distribution includes Core and Additional modules. Current version comes in a form of bundles that can be immediately used in Eclipse / OSGi.
• The OSGi-specific pieces need to be downloaded separately <please insert a link here>
• Proposed Springs implementation supports OSGi R4, today needs Java 1.4 (uses beans)
• Size of the Springs: in the range of 3Mb without dependencies
• External dependency: Apache commons logging

What's happening with OSGi standardization of Springs

• Current version is very close to be final. The specification will be finalized in December 2007.

Next steps for Springs:

• look at popular WebApps and make life easier for them

Considerations:

• Needs to support smaller execution environment (Foundation 1.1?). This might be easier to do for "core" part
• Could we have PDE tools for it?

Service Activator Toolkit (SAT) – automates startup sequence where OSGi services involved

• "in order for you to run you need all your required services".
• keeps OSGi specific things mostly out of the code
• has default OSGi logging service
• if service goes away bundle that requires it gets deactivated, no re-wiring (?)
• describes why bundles don't start
• includes tools to avoid circular dependencies
• it is an Eclipse project, small size (170K), used of Open health Care Framework (OHF)

Overall:

It feels like there are some common concepts in different component models. Extension registry, OSGi services, DS, Springs all work by wiring together "producers" and "consumers". All have they interesting points.

Is there a "kernel" in the framework that could be used by OSGi services, DS, Springs, extension registry?

On another point, could we combine "core" of Springs and SAT to get a small "OSGi service helper"?

Logging

• Eclipse Logging:
  • Launcher Log
  • FW log / Plugin log
  • DEBUG trace

• Problems:
  • JUL and Named Loggers
  • Commons Logging and discovery

• LogService lacks:
  • conditional Logging (isDebug, isLoggable)
  • Named Loggers -- useful for optimization, per bundle
  • Arbitraty Objecy, Context --- CBE, Status, Mapped Diagnostic Context

• Progress:

1. Implementation in the Incubator - SLF4J provides bridges for ACL, Log4J, potentially JUL too.
2. Inc patches and contribs from Paul and Stefan
3. Seems important enough to get further contributions
4. Move to Equinox NS -- currently extends LogService -- backwards compatible
5. Review impl -- caching Named Loggers, Event bus is simple.
6. Experiment as a bundle and FW fragment.

• If its good and small ... we can begin thinking about FW integration

• Eclipse integration -- Need help for other serviceability folk.

Fifth breakout meeting

Remote management planning

• command registration mechanism
• Other consoles: KF, Felix

Simple commands:

• stop the framework
• open file
• open url to install bundles (p2)

More Complex commands/management:

• • install/uninstall/start/stop bundles
  • in-process communication: from pojo to osgi

• input file of commands to run

How to discover / select running instance

• get list of running processes?
• OS level registration, file associations

lightweight command service, Both console and communication mechanism would be based on it.

• • exposes core.commands?
• communication mechanism is not the console itself

• client side needs to be very lightweight (implementable in c for the native launcher)
• actual communication mechanism: socket?
• how does this line up with OS messages: eg drag/drop

Related bugs:

• Bug 201154: Add support in SWT for event-based document opening
• Bug 178927: launcher: way to pass arguments from launcher to a running application instance
• Bug 4922: EditorMgmt Need ability to open a file in eclipse from the command line

Dynamic programming planning

Need a concrete programming model for service registration and use. Adoption of OSGi services in the Eclipse world is seriously hampered by lack of a simple idiom for service lookup and use

• Want to end up in a world where we're back to POJOs, without needing to know what service delivery mechanism is used
• Currently in p2 we are using a simplistic approach that isn't reliable in a dynamic world, but has a simple use model
• We should investigate using SAT (Service Activator Toolkit) for wiring/initialization of services. It is easy to use, relatively lightweight (about 150KB of code). It cuts down a lot of duplicated code in bundle activators that is traditionally needed for service wiring

What can we do to help make use of the extension registry dynamic enabled? We currently have ExtensionTracker as a way for clients to add hooks for handling extensions being added/removed

Dynamic markup: allow bundles to declare their level of dynamic enablement/awareness. That way when a bundle is added/removed/updated, we can determine if a restart is required.

• This is a good thing, but there isn't agreement on whether this should be a manifest header, or a separate piece of metadata (for example in the corresponding IU)
• What is the default assumption for a bundle that is missing dynamic markup? Should we assume the bundle is dynamic or not?

Action plan:

• Investigate use of SAT
• Look at Spring core when Spring M3 comes out (mid-October)
• Settle on dynamic markup syntax and start rolling it out
• Review extension registry usage

p2 planning

Backwards compatibility:

• Compatibility with existing update.core APIs
• Data compatibility with feature.xml, platform.xml, site.xml, etc
  • Generally data files such as platform.xml are considered implementation details of the platform and are subject to change
• Compatibility support for people implementing install handlers

Co-existence model: Update.core APIs are still there, but if you invoke them, they will forward to corresponding function in p2 API. Main body of update.core is essentially thrown out in favour of p2 implementation

Compatibility model: update.core does its own (old) thing, and then run some generator or other tool on the resulting platform.xml files, etc, to reconcile with p2 world

Community will continue to craft feature.xml files in 3.4 timeframe, and we will use Generator to build p2 metadata

Source bundles: Want a better story for packaging source. 1-1 correspondence between source bundles and binary bundles is most flexible for packaging/install purposes

A rough outline of the current plan:

M3:

• Rename packages and bundles
• Be able to install to any platform/os from a single update site
• Shared install
• Refine end-user UI
• Establish concrete set of functionality to be available in 3.4 final
• Pluggable download manager strategy

M4:

• Graduate p2 bundles from incubator into Equinox proper
• Firm up APIs - javadoc, etc

M5:

• p2 goes live in SDK in first M5 build (first I-build after M4)

M5a:

• Fix critical p2 bugs found the day after M5 release that caused community panic

Summary

Day one summary: Everything is possible

Day two summary: Everything is possible with sufficient contribution. Contribution is not just code.