OTDT/JDTCore

As a central component of the OTDT we are maintaining a branch of the org.eclipse.jdt.core plugin.

This page summarizes background, design rationale and consequences of this branch. It is created as input for bug 330534 where the inclusion of Object Teams in the Indigo release train is discussed.

Extending Java

A large number of research projects over the last decade or so has used Java as a platform for performing research regarding the design of object oriented programming languages. Despite the large number of (prototypical) tools this field is hardly represented at Eclipse.org, which means no standard technology and process exists for this kind of project.

Characterizing the Relation between Java and OT/J

Extending Java can mean a lot of different things. Without going too deep into the description of OT/J this specific relation to Java can be characterized as follows:

• The syntax is extended at several levels (see also OTJLD §A OT/J Syntax):
  • New modifiers like team, callin seamlessly integrate into the existing grammar
  • The new block construct within() {} is syntactilly similar to, e.g., while. It can appear in any location within an OT/J program.
  • A new syntax for type parameters is added (T<@anInstance>), that may occur in any location using type references.
  • Method signatures can, in specific contexts, declare a parameter with two types using the "ARole as ABase param" syntax.
  • Method bindings (callin/callout) introduce new syntax (e.g., "roleMethod -> baseMethod") representing a new kind of class-members.

So essentially the language design by intention creates a new cohesive language rather than two separated sub-languages.

• Typesystem
  • The introduction of a new kind of type parameter has impact on all aspects of type checking. These parameters can be freely combined with any existing type constructors like arrays and regular type parameters.
• Inheritance
  • Even without any new syntax a new kind of inheritance relationship is introduced (role classes as inner classes of a team are virtual classes). A role class may override an inherited role class and it implicitly inherits all members of the class it overrides.

Technical implications of the Relation between Java and OT/J

• Syntax integration:
  • The syntactic integration implies that the resulting language cannot easily be subdivided into two sub-languages that would be parsed by separate parsers with delegation between them two.
  • To achieve greatest possible compatibility, the scanner enables a subset of keywords only after the keyword team has been consumed.
  • Specifically, within and the new type parameters are enabled in all contexts.
  • The scanner has a master switch for disabling even within and special type parameters.
• Type checking:
  • As type checking in the JDT compiler is essentially performed in resolveType methods of all AST classes, changes in the type system affect most of these plus also the more centralized parts in ClassScope and MethodVerifier, e.g..
• Inheritance:
  • Since OT/J supports a mild form of multiple inheritance which cannot be accepted by the JVM, prior to code generation this structure has to be flattened somehow. In OT/J this is solved in two steps:
    • role classes are split into an interface part (truthfully modeling the multiple inheritance) and a class part (containing the implementation).
    • implicit inheritance is implemented by copying (and adjusting) bytecodes from a super-role to the sub-role.

  The hidden duality of class and interface parts of the same source-code class is pervasive because (a) type checking requires to operate on the interface part (b) field access requires to switch over to the class part (c) role instantiation requires a hidden redirection to a synthetic creation method instantiating the class part.

• Compilation order:
  • Due to the invasive nature of some translations the general rule of translating one class at a time had to be abandoned and an explicit state model exists that manages the order of individual translation steps for individual classes (even within the same compilation unit).

Extensible Java IDE

Ideally, generic tooling would be used for adding new language features to Java and extending all of the IDE to reflect the language enhancement. Tools that strive at some sub-goals of this vision include:

• JastAdd - extensible compiler, not at Eclipse.org
• IMP - meta IDE
• Xtext - IDEs for DSLs

Ideally, all these projects would join forces with all of JDT to produce an environment that has all the extensibility for creating JDT-quality IDEs for any extensions of Java.

Since this perfect platform hasn't arrived at the train (ups?) for now and the years to come, a different architecture has to be chosen.

OTDT Architecture

The basic idea is to just swap the core component for Java with the extended OT/J variant in order to leverage all tooling that the Eclipse community provides for Java development also for development with OT/J.

Core

This is where the branched variant of the org.eclipse.jdt.core resides. It is based on the original sources of the JDT/Core, with many changes within existing classes and some org.eclipse.objectteams packages for added classes.

• As of version 3.7M3 / 0.8M3 the diff between the original and the OT branch measures 2.9Mbyte (zipped 550Kbyte) - counting only changes in existing files, not added files which live in the objectteams namespace.
• As of version 4.4 M2 / OTDT 2.3 M2 the differences affect more than 2300 file locations.

Upper layers

A few new plugins exist that add a few new views to the IDE and for implementing basic support for binding the OT runtime environment (for compile time and run time) etc.

Developing OT/J code happens through the JDT/UI. This is possible because the JDT/UI talks to the JDT/Core, which in this configuration is the variant with OT-support built in. In a few places the JDT/UI expects to know about all possible AST types, which is no longer true with OT/J. These locations are adapted non-invasively using OT/Equinox (internally realized by loadtime bytecode weaving). Additionally, many functions of the JDT/UI (code assist, refactoring ...) have been extended using OT/Equinox in order to provide enhanced support for OT/J. E.g., preconditions of refactorings need to check additional information, quick fixes may offer additional proposals, etc.

The same pattern is applied to plugins like JDT/Debug, JDT/DebugUI, PDE/UI etc.

OTDT Development Process

Since the JDT/Core and OT/J reside in different source code repositories, the branch is not directly handled using builtin support. Yet, over the last 7 years we have developed a routine of staying up-to-date with the latest in the JDT/Core as such:

• All modifications to original jdt.core classes are systematically marked with begin-end comments, the original code remains in place (commented out) for comparison during merging.
• Merging: At regular intervals a diff containing all changes in the original CVS is created and applied to the OT/J branch. After skipping 3.1 the migration from 3.0 to 3.2 kept one student busy for almost 6 months. We had several years were all changes of a whole year were merged in one big effort, but meanwhile we're down to synchronizing branches at least once per 6-week milestone. This has become routine.
• By also actively following the bugzilla inbox of JDT/Core I'm well informed about pending changes and in some cases (like bug 330304) observing JDT/Core lets me detect in a very timely fashion where their patches require additional action for OT/J.
• Other than that we follow the normal rules of development at Eclipse.

OTDT Testing

We regularly run a number of test suites:

• original JDT/Core tests.

Actually, we maintain a branches of org.eclipse.jdt.core.tests.compiler and org.eclipse.jdt.core.tests.model. These branches contain a few unavoidable changes, such that the modifications actually document the actual behavioral difference between the original JDT/Core and our branch. More on that later

org.eclipse.jdt.core.tests.builder runs without modifications.

• Selected test suites from JDT/UI. These tests run from the original sources, with a very small number of adaptations applied using OT/Equinox
• A test suite for all language features of OT/J
• Further OT specific test suites complementing all of the original jdt suites.

One test run currently includes over 86.000 test cases. All tests pass and no build is ever published for which this is not true. Exception as of Luna: JDT/UI's LeakTestSuite should some failures. We need to investigate if this indicates a context-sensitive bug in JDT/UI or a bug caused by OTDT.

Releng

We have a PDE based build process, the most significant detail being the two phase process:

• build the OT/J compiler
• use the OT/J compiler for a full build and test run

For some of our very early releases we shipped full binaries because we had to manipulate various details. As p2 matured it allowed us to move towards a deployment and provisioning story that uses only documented features:

• We are shipping three features:
  • "Object Teams Development Tooling":

    master feature

  • "Object Teams Patch for JDT/Core":

    patch feature which replaces org.eclipse.jdt.core with our variant, this feature is included from the above

  • "Object Teams Equinox Integration":

    runtime modules for OT/J and OT/Equinox, this feature can be used alone, some of its bundles are required by bundles of the master feature

• We use p2 touchpoint instructions for the following:
  • Installing OT/Equinox weaving as an Equinox Adaptor Hook (using setProgramProperty) (here bug 329784 is relevant for compatibility with others).
  • Configuring Equinox and the JVM to work around issues regarding cyclic classloader dependencies (using setProgramProperty and addJvmArg).
  • Set the special backwards dependency (patched plugin -> patch feature) that avoids unintentional installation of the OT variant of jdt.core (see below)

• Version numbers:
  • All Object Teams artifacts currently have independent version number 2.7.0.qualifier, the milestones are synchronized with the release train
  • The modified org.eclipse.jdt.core has the same version (major.minor.micro) as the respective original. However, the qualifier is encoded as OTDT_r270_qualifier
    • the "OTDT" segment makes this version smaller than the original "v20180528-0519" in order to avoid accidental selection of the OTDT variant, where in fact the original is expected (bug 535509). This also signals the origin of the artifact.
    • the "r270" segment documents the corresponding OT version (here 2.7.0)
    • the final "qualifier" is used as normal

Coexistance with other JDT/Core patches

• Due to the construction of the version qualifier within each major.minor.micro stream the OT variant will always be greater than the original.

• The reference from a patch feature to the original feature being patched must use an exact version number.
  • A xsl transformation is used to cut of the generated extended qualifier, so instead of depending on 3.7.0.v20100824-0800-7z8dFb7FMTfEQ4wDz0DJlHYd9H15 we depend on [3.7.0.v20100824-0800,3.7.0.v20100824-0801).
• The reference from a feature to its included plugins uses an exact version number.

• It is no problem to update a patch feature with newer version of that patch feature

• It is not possible to install a second independent patch feature affecting the same original feature.

Patches from JDT/Core: should the JDT/Core team need to ship a patch feature for urgent fixes this has the following implications for users having the OTDT installed:

• They cannot directly install the patch due to conflict with the OT patch feature
• They can choose to uninstall the OTDT and install the JDT/Core patch instead, sacrificing OT functionality for a JDT/Core bugfix.
• They can wait until a corresponding OT patch is published containing the same JDT/Core patch merged into the OT-branch
  • The OT team will eagerly provide such patches

Other projects: should other projects chose the same strategy as the OTDT, these two projects will not be compatible, due to the limitation of one patch feature.

Unintended installation of the OT variant

As long as a user installs the JDT explicitly via the JDT feature, the exact version reference will ensure that the original jdt.core is installed.

If the org.eclipse.jdt.core is installed only due to indirect requirements of a bundle to be installed, p2 would select the highest available bundle version, which would result in fetching the OT variant without explicitly requesting so (iff all live in the same repo or both repos are enabled).

Solution: As proposed in bug 330534#c8 the OT variant of the jdt.core plugin now declares a non-optional non-greedy dependency on the patch feature. The effect is that this plugin can only be installed with the OT feature but the OT feature will never be pulled in implicitly. Thus, the user has to explicitly select and install "Object Teams Patch for JDT/Core" to get the OT variant. All implicit installs of jdt.core due to bundle dependencies will pick the (highest) original version. Additionally, since bug 535509 the OT version is constructed to be less than the corresponding JDT version, leading installation to prefer the JDT version over the OT variant.

With the above solution it was still possible that p2 would suggest to install the Object Teams patch feature during check for updates.

Resolved: This is a issue in p2 has been resolved via bug 350133

User Install Experience

When installing the OTDT several non-standard things happen that should be worth reporting to the user.

• bug 316702 this is the master issue proposing "More feedback on what's installing",

see also this proposal for "install capabilities"

• bug 330338 specifically addresses the visualization of patch features.