Triquetrum/Kepler Tree Shaking

This page documents an experiment in adding a Kepler actor to Triquetrum by using tree shaking, where the model is run and the classes that are loaded are added to Kepler.

A better solution is to add the actor and director in question and keep adding files until it compiles. See Triquetrum/Kepler.

What Kepler classes are necessary?

The next step is to determine what Kepler classes are necessary.

The demo primarily uses org.kepler.ddp.actor.ExecutionChoice, which is a composite actor that contains submodels that implement the different ways the actor can be executed.

The -verbose option of the java command will report what classes are loaded.

Unfortunately, it seems that adding -verbose to the run configuration in Eclipse does not report the Kepler classes that are loaded, so we use the command line

To run the model without the UI from the command line

cd kepler/build_area
ant run -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml

Edit kepler/build-area/src/org/kepler/build/Run.java and insert the following into the runSuite() method:

 java.createJvmarg().setLine("-verbose");

Compile the build system:

 ant -f kepler-tasks.xml

Run the model and save the output to a file:

 ant run -v -Dworkflow=../ddp/workflows/demos/wordcount-execution-choice.xml >& /tmp/run.txt

To see how many org.kepler classes are loaded (the sed command converts any inner classes that have $ in their name to the parent class, so org.kepler.util.sql.Table$IndexType becomes org.kepler.util.sql.Table)

grep 'Loaded' /tmp/run.txt | grep org.kepler | awk '{print $3}' | sed 's/\$.*\//' | sort | uniq |  sed 's@\.@/@g' | awk '{print $1 ".java"}' > /tmp/files

FIXME: Kepler classes should not be in org.kepler because the Kepler project does not have the kepler.org domain name.

The above returns 188 files. Not all of these classes will be necessary at runtime when running the model, but the important thing here is to make a guess at what classes are necessary.

The script below, when run in the kepler directory, will create a tar file with all the .java files:

#!/bin/sh                                                                          

if [ ! -d files ]; then
   mkdir files
fi

files=`cat /tmp/files`
for file in $files
do
    srcs=`ls -1d */src`
    for src in $srcs
    do
      	if [ -f $src/$file ]; then
            (cd $src; tar -cf - $file) | (cd files; tar -xvf -)
            break;
	fi
    done
done

(cd files; tar -cf ../files.tar .)
ls -l files.tar

Place the above in a file called kepler/findKeplerFiles.

Run the script with:

 cd kepler
 sh ./findKeplerFiles

Create copies of the files in the org.kepler.triquetrum.ddpDemo module (your pathname will vary)

 cat files.tar | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src; tar -xf -)

FIXME This is really wrong because we are creating one bundle for all the Kepler files used by one demo. A better way would create multiple bundles for the various Kepler modules.

Tracking Down Dependencies

Right click on the org.kepler.triquetrum.ddpDemo project and select refresh. In the Problems tab, there will be many compilation problems. Below are the problems we saw and the solutions:

_controller cannot be resolved in KeplerGraphFrame

The error is:

 Description	Resource	Path	Location	Type
 _controller cannot be resolved to a variable	KeplerGraphFrame.java	/org.kepler.triquetrum.ddpDemo/src/org/kepler/gui	line 245	Java Problem

KeplerGraphFrame extends ptolemy.vergil.actor.ActorGraphFrame. Triquetrum does not use ActorGraphFrame

The solution is to remove org.kepler.triquetrum.ddpDemo/src/org/kepler/gui/KeplerGraphFrame.java

_debugging cannot be resolved in ExecutionChoiceDirector

The error is:

 Description	Resource	Path	Location	Type
 _debugging cannot be resolved to a variable	ExecutionChoiceDirector.java	/org.kepler.triquetrum.ddpDemo/src/org/kepler/ddp/actor	line 101	Java Problem

The problem is that ExecutionChoiceDirector extends CaseDirector, which is not found.

Looking at the errors in the imports, we see we need Case and Refinement as well. Case extends MultiInstanceComposite, so we need that.

The solution is to copy over these files:

(cd $PTII; \
 tar -cf - ptolemy/actor/lib/hoc{Case,CaseDirector,MultiCompositeActor,MultiCompositePort,Refinement,RefinementPort}.java ptolemy/actor/util/Time.java ptolemy/math/ExtendedMath.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src/; tar -xvf -)

DDFDirector is needed by ExecutionChoiceDirector

Copy over DDFDirector and ActorEnablingStatus, which is needed by DDFDirector. Also copy over DFUtilities, NotSchedulableException, Time and ExtendedMath:

 (cd $PTII; \
 tar -cf - ptolemy/actor/lib/hoc/{Case,CaseDirector,MultiCompositeActor,MultiCompositePort,Refinement,RefinementPort}.java ptolemy/actor/util/Time.java ptolemy/math/ExtendedMath.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.\

triquetrum.ddpDemo/src/; tar -xvf -)

ParameterPort and PortParameter are needed by ExecutionChoiceDirector

The error is:

 Description	Resource	Path	Location	Type
Access restriction: The type 'ParameterPort' is not API (restriction on required library 
'/Users/cxh/src/workspaceTriq04Mar/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool /plugins/ptolemy.core_11.0.0.201605041253.jar')	ExecutionChoiceDirector.java

/org.kepler.triquetrum.ddpDemo/src/org/kepler/ddp/actor line 44 Java Problem

FIXME: The issue here is that PortParameter and ParameterPort are in the ptolemy.core module, but that module is not available

The workaround is to copy them:

(cd $PTII; tar -cf - ptolemy/actor/parameters/{ParameterPort,PortParameter}.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src/; tar -xvf -)

It turns out that we need a number of classes from ptolemy.core. Unfortunately, adding

   ptolemy.core;version="11.0.0",

to the MANIFEST.MF file results in an error message: "No available bundle exports ptolemy.core"

However, the org.kepler.triquetrum.ddpDemo project lists the following as a Plug-in Dependency

 /Users/cxh/src/workspaceTriq04Mar/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool/plugins/ptolemy.core_11.0.0.201605041253.jar

That jar file has the following MANIFEST.MF:

Manifest-Version: 1.0^M
Bundle-SymbolicName: ptolemy.core^M
Export-Package: org.ptolemy.classloading;version="11.0.0",org.ptolemy.^M
 classloading.osgi;version="11.0.0",ptolemy.actor;version="11.0.0",pto^M
 lemy.actor.parameters;version="11.0.0",ptolemy.actor.sched;version="1^M           
 1.0.0",ptolemy.actor.util;version="11.0.0",ptolemy.data;version="11.0^M           
 .0",ptolemy.data.expr;version="11.0.0",ptolemy.data.type;version="11.^M           
 0.0",ptolemy.data.unit;version="11.0.0",ptolemy.graph;version="11.0.0^M           
 ",ptolemy.graph.analysis;version="11.0.0",ptolemy.graph.analysis.anal^M
 yzer;version="11.0.0",ptolemy.graph.analysis.strategy;version="11.0.0^M           
 ",ptolemy.graph.mapping;version="11.0.0",ptolemy.kernel;version="11.0^M           
 .0",ptolemy.kernel.attributes;version="11.0.0",ptolemy.kernel.undo;ve^M
 rsion="11.0.0",ptolemy.kernel.util;version="11.0.0",ptolemy.math;vers^M
 ion="11.0.0",ptolemy.util;version="11.0.0"^M
Bundle-Name: Ptolemy Core^M
Bundle-Version: 11.0.0.201605041253^M
Bundle-ManifestVersion: 2^M
Bundle-RequiredExecutionEnvironment: JavaSE-1.7^M
Bundle-Activator: ptolemy.kernel.activator.Activator^M
Eclipse-BuddyPolicy: registered^M
Bundle-Vendor: CHESS^M
Import-Package: org.osgi.framework;version="1.3.0",org.osgi.util.track^M
 er;version="1.5.1",org.ptolemy.commons;version="11.0.0"^M

However, if I right click on the org.kepler.triquetrum.ddpDemo project, select Build Path -> Configure Build Path, then under the Libraries tab, expand Plug-in Dependencies, scroll to ptolemy.core..., expand it and expand the Access rules tab, I can see that ptolemy/actor/parameters/* is not present

ptolemy.core Access rules

The workaround is to add ptolemy/actor/parameters/* and move it above the forbidden rule.

For ptolemy.core, add:

• ptolemy/actors/parameters/*
• ptolemy/graph/*

For ptolemy.moml, add:

• ptolemy/moml/filters/*

For ptolemy.gui, add

• ptolemy/actor/gui/style/*

However, it seems like the above changes are not persistent?

FIXME: Where are these Access rules set?

Solution: Erwin wrote:

OSGi supports 2 modes to define dependencies between bundles :

• Import-Packages : then you should list each individual java package name that you need (in this case that would be ptolemy.actor.parameters and many others)
• Require-Bundle : there you can just put the bundle name (ptolemy.core in this case) and that registers all that bundle's packages

If MANIFEST.MF is updated:

Require-Bundle: org.eclipse.ui,
 org.eclipse.core.runtime,
 ptolemy.gui,
 ptolemy.core,
 ptolemy.moml

Then a number of compilation errors are solved.

org.apache.commons.io.FilenameUtils is missing from ExecutionChoice

The error:

 import org.apache.commons.io.FilenameUtils;

Solution: Add the following to org.kepler.triquetrum.ddpDemo MANIFEST.MF:

  org.apache.commons.io;version="2.2.0",

ExecutionChoice uses lots of GUI code

The imports below are missing from ExecutionChoice:

 import org.kepler.gui.KeplerGraphFrame;
 import org.kepler.gui.ModelToFrameManager;
 import org.kepler.gui.frame.TabbedKeplerGraphFrame;

 import ptolemy.actor.gui.EditorFactory;

 import ptolemy.gui.ComponentDialog;

 import ptolemy.gui.ExtensionFilenameFilter;
 import ptolemy.gui.JFileChooserBugFix;
 import ptolemy.gui.PtFileChooser;
 import ptolemy.gui.PtGUIUtilities;
 import ptolemy.gui.Query;

FIXME: Not sure what to do about these. It seems like ExecutionChoice has a very tight connection to GUI classes like KeplerGraphFrame that are not present in Triquetrum

Hadoop

The error is:

 Description	Resource	Path	Location	Type
Configuration cannot be resolved to a type	DDPDataSink.java	/org.kepler.triquetrum.ddpDemo/src/org/kepler/ddp/actor/pattern	line 258	Java Problem

The issue is that these imports in DDPDataSink are not present

 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.FileUtil;
 import org.apache.hadoop.fs.Path;

One solution would be to import the jars from ~/KeplerData/workflows/module/hadoop/tools/share/hadoop/

A better solution would be to find a hadoop Eclipse bundle upon which we could depend.

Kepler provides a hadoop project that could be helpful here!

Solution: Right click on the org.kepler.triquetrum.ddpDemo project -> Build Path -> Configure Build Path and add the following package dependencies

• configuration-manager (For apache logging) FIXME: Might be able to get this elsewhere
• ddp-common
• hadoop

ExecutionChoice Revisited

After looking at the number of changes above, a different approach of just adding the ExecutionChoice actor might be in order.

The first step was to modify the Plug-in dependencies for ptolemy.core, ptolemy.actor and ptolemy.gui as above.

ExecutionChoice depends on KeplerDocumentationAttribute, which depends on DocAttribute, so we copy those files:

 (cd $PTII; tar -cf - ptolemy/vergil/basic/{DocAttribute,KeplerDocumentationAttribute}.java) | (cd ~/src/workspaceTriq04Mar/org.kepler.triquetrum.ddpDemo/src/; tar -xvf -)

Providing the ExecutionChoice actor to the run time

Based on Providing the actor class to the runtime

Create a package called org.kepler.ddp.actor.activator and then create Activator.java in that package:

package org.kepler.ddp.actor.activator;


import org.kepler.ddp.actor.ExecutionChoice;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceRegistration;
import org.osgi.framework.Version;
import org.ptolemy.classloading.ModelElementClassProvider;
import org.ptolemy.classloading.osgi.DefaultModelElementClassProvider;
import org.ptolemy.commons.ThreeDigitVersionSpecification;
import org.ptolemy.commons.VersionSpecification;

public class Activator implements BundleActivator {
	public void start(BundleContext context) throws Exception {

	    // FIXME figure out a more compact way to create a version-aware provider,
	    // that uses the bundle version but is not too dependent on OSGi APIs itself.
	    Version bundleVersion = context.getBundle().getVersion();
	    VersionSpecification providerVersion = new ThreeDigitVersionSpecification(
	        bundleVersion.getMajor(),
	        bundleVersion.getMinor(),
	        bundleVersion.getMicro(),
	        bundleVersion.getQualifier());

	    _apSvcReg = context.registerService(ModelElementClassProvider.class.getName(),
	        new DefaultModelElementClassProvider(providerVersion, ExecutionChoice.class),
	        null);
	  }

	  public void stop(BundleContext context) throws Exception {
	    _apSvcReg.unregister();
	  }

	  // private stuff
	  /** The svc registration for the actor provider */
	  private ServiceRegistration<?> _apSvcReg;

}

Packaging the plugin(s) for your RCP editor

See Packaging the plugin(s) for your RCP editor

Create a simple (non-Java plugin): File -> New Plug-in Project.

• Name: org.kepler.triquetrum.ddpDemo.palette
• Uncheck Create a Java Project
• Click Next, then Finish

Right click on the newly created project, select File -> New -> File and create plugin.xml.

plugin.xml should look like:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
	<extension 
	     name="Triquetrum palette ExecutionChoice extension"
	     point="org.eclipse.triquetrum.workflow.editor.paletteContribution">
	  <group displayName="Examples">
	     <entry 
	            class="org.kepler.ddp.actor.ExecutionChoice"
	            displayName="ExecutionChoice"
	            type="Actor">
	    </entry>
	  </group>
	</extension>
</plugin>

MANIFEST.MF should look like:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Palette entry for ExecutionChoice
Bundle-SymbolicName: org.kepler.triquetrum.ddpDemo.palette;singleton:=true
Bundle-Version: 1.0.0.qualifier
Require-Bundle: org.eclipse.triquetrum.workflow.editor;bundle-version="1.0.0"

Implementing an actor feature

See Implementing an actor feature

1. File -> New -> Feature Project
   1. Name: org.kepler.triquetrum.ddpDemo.feature
   2. Finish

feature.xml should contain:

<?xml version="1.0" encoding="UTF-8"?>
<feature
      id="org.kepler.triquetrum.ddpDemo.feature"
      label="Feature"
      version="1.0.0.qualifier">

   <plugin
         id="org.kepler.triquetrum.ddpDemo"
         download-size="0"
         install-size="0"
         version="0.0.0"
         unpack="false"/>

   <plugin
         id="org.kepler.triquetrum.ddpDemo.palette"
         download-size="0"
         install-size="0"
         version="0.0.0"/>

</feature>

Right now, there is a problem because ptolemy.core is required. However I decided to push on and create the feature and try to load it.

Loading the ExecutionChoice feature

1. Start up Triquetrum RCP Editor. See Building From Sources -> Running
2. In the Triquetrum RCP Editor, do Help -> Install New Software

Cannot complete the request

The following dialog pops up:

"Cannot complete the request. This installation has not been configured properly for Software Updates. See the error log for details"

The log is:

 !ENTRY org.eclipse.equinox.p2.ui.sdk 2 0 2016-06-02 11:50:03.790
 !MESSAGE Could not locate the running profile instance. The eclipse.p2.data.area and eclipse.p2.profile properties may not be set correctly in this application's config.ini file.

Erwin suggested that running an update from a Triquetrum launched from another Eclipse is complex and that it would be better to Download a Triquetrum and run it from the command line.