Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

DLTK IDE Guide:Step 2. Towards an Editor

Summary

This tutorial describes the set of steps to create an Integrated Development Environment (IDE) for a dynamically-typed language using Eclipse Dynamic Languages Toolkit (DLTK). In this tutorial we will make a simple editor for Python language.

Requirements

  • Eclipse 3.5, 3.6, 3.7
  • Latest DLTK 3.0
  • Plugins code from cvs
  • ANTLR runtime bundle from orbit project: [[1]]

Python source parser.

Some words about source parsers. DLTK allows contribution of source code parsers via org.eclipse.dltk.core.sourceParsers extension point. Source parser builds source module Abstract Syntax Tree. AST is used in SourceElementParsers to building structure model.

Python source parser is quite complicated and cannot be created during this short tutorial. We provide parser implementation in *org.eclipse.dltk.examples.python.part2 plugin*.

Source parser and source element parser contribution from plugin.xml: 

<extension point="org.eclipse.dltk.core.sourceParsers">
      <parserContribution
            natureId="org.eclipse.dltk.examples.python.nature">
         <parser
               class="org.eclipse.dltk.python.internal.core.parser.PythonSourceParserFactory"
               description="Python Source parser"
               id="org.eclipse.dltk.example.python.sourceParser"
               name="sourceParser.name"
               priority="0" />
      </parserContribution>
</extension>
 
<extension point="org.eclipse.dltk.core.sourceElementParsers">
      <parser
            class="org.eclipse.dltk.python.internal.core.parser.PythonSourceElementParser"
            nature="org.eclipse.dltk.examples.python.nature"
            priority="1">
      </parser>
</extension>

DLTK allows priority overriding of source parser and source element parser. Parsers from part2 plug-in will override previous contribution from org.eclipse.dltk.examples.python plug-in.

DLTK provides generic AST implementation for top level elements (module, types, methods, fields, etc). Some of these classes are used in DLTK APIs (for example in search).

But it is not required to have DLTK based AST. Any kind of AST can be used. For example we use AST produced by Rhino engine for JavaScript. DLTK provides several features that have been already implemented (e.g. easy source element parser, search, etc) if a language developer chooses to use DLTK AST based hierarchy.

We use ANTLR v3 based parser for Python IDE. This parser uses DLTK AST classes. Python ANTLR grammar file contains java code to create AST nodes.

Editor

DLTK has implementation of a generic code editor with many useful features. All these features can be easily configured by a developer.

Simple editor. The first thing that's required to create is a separate plug-in for the editor. Let's name it org.eclipse.python.examples.part3. We need to add the following dependencies to our new plug-in:

  • org.eclipse.ui
  • org.eclipse.core.runtime
  • org.eclipse.jface.text
  • org.eclipse.ui.editors
  • org.eclipse.ui.ide
  • org.eclipse.dltk.examples.python
  • org.eclipse.dltk.examples.python.part2
  • org.eclipse.dltk.ui
  • org.eclipse.dltk.core

For creating an eclipse editor we need to create extension of "org.eclipse.ui.editors" extension point with following content: 

<extension point="org.eclipse.ui.editors">
      <editor class="org.eclipse.dltk.examples.python.internal.ui.editor.ExamplePythonEditor"
            default="false"
            icon="icons/obj16/python_obj.gif"
            id="org.eclipse.dltk.examples.python.part3.ui.editor"
            name="Example Python Editor">
         <contentTypeBinding contentTypeId="org.eclipse.dltk.examples.python.content-type"/>
      </editor>
   </extension>

Editor definition contains a link to the content type defined earlier. Pay attention that icon attribute is not required by default but the editor won't work without it. Plug-in activator class should be named: "ExamplePythonUI". When the definition is done we can add the implementation of ExamplePythonEditor class. The initial implementation only contains some identifiers required by the base ScriptEditor class.

ExamplePythonEditor.java 

public class ExamplePythonEditor extends ScriptEditor {
 
	public static final String EDITOR_ID = "org.eclipse.dltk.examples.python.part3.ui.editor";
 
	public static final String EDITOR_CONTEXT = "#PythonEditorContext";
 
	protected void initializeEditor() {
		super.initializeEditor();
		setEditorContextMenuId(EDITOR_CONTEXT);
	}
 
	public String getEditorId() {
		return EDITOR_ID;
	}
 
	protected IPreferenceStore getScriptPreferenceStore() {
		return PythonUI.getDefault().getPreferenceStore();
	}
 
	public IDLTKLanguageToolkit getLanguageToolkit() {
		return ExamplePythonLanguageToolkit.getDefault();
	}
}

When the class was created we can start to test it. The reference to the editor can be found in "Open with" context menu. This editor will be used for python files by default: Dltk 020 editor 001.png

In the opened editor we can see a python source module. The Outline view contains appropriate data which was created using a structured model.

Dltk 020 editor 002.png

Source configuration

Each editor contains a viewer that controls highlighting and basic code editing. We need to implement several classes for the viewer. Some of them will be used later for syntax highlighting. Let's create IPythonPartitions interface with required partition constants (partitioning is used for syntax highlighting): IExamplePythonPartitions.java 

public interface IExamplePythonPartitions {
 
	public final static String PYTHON_PARTITIONING = "__python_partitioning";
 
	public final static String PYTHON_COMMENT = "__python_comment";
	public final static String PYTHON_STRING = "__python_string";
 
	public final static String[] PYTHON_PARITION_TYPES = new String[] {
			IPythonPartitions.PYTHON_STRING, IPythonPartitions.PYTHON_COMMENT,
			IDocument.DEFAULT_CONTENT_TYPE };
}

ExamplePythonTextTools class creates SourceViewerConfiguration based class for viewer configuration. ExamplePythonTextTools.java 

public class ExamplePythonTextTools extends ScriptTextTools {
 
	private final static String[] LEGAL_CONTENT_TYPES = new String[] {
			IExamplePythonPartitions.PYTHON_STRING, IExamplePythonPartitions.PYTHON_COMMENT };
 
	public ExamplePythonTextTools(boolean autoDisposeOnDisplayDispose) {
		super(IExamplePythonPartitions.PYTHON_PARTITIONING, LEGAL_CONTENT_TYPES,
				autoDisposeOnDisplayDispose);
	}
 
	public ScriptSourceViewerConfiguration createSourceViewerConfiguraton(
			IPreferenceStore preferenceStore, ITextEditor editor,
			String partitioning) {
		return new ExamplePythonSourceViewerConfiguration(getColorManager(),
				preferenceStore, editor, partitioning);
	}
}

We also need to add the following code to the ExamplePythonUI class (plug-in activator) to hold the text tools instance. ExmplePythonUI.java 

private ExamplePythonTextTools fPythonTextTools;
 
public synchronized ExamplePythonTextTools getTextTools() {
	if (fPythonTextTools == null)
		fPythonTextTools= new ExamplePythonTextTools(true);
        return fPythonTextTools;
}

ExamplePythonSourceViewerConfiguration.java 

public class ExamplePythonSourceViewerConfiguration extends
		ScriptSourceViewerConfiguration {
 
	public ExamplePythonSourceViewerConfiguration(IColorManager colorManager,
			IPreferenceStore preferenceStore, ITextEditor editor,
			String partitioning) {
		super(colorManager, preferenceStore, editor, partitioning);
	}
 
	public IAutoEditStrategy[] getAutoEditStrategies(
			ISourceViewer sourceViewer, String contentType) {
		return new IAutoEditStrategy[] { new DefaultIndentLineAutoEditStrategy() };
	}
 
	public String[] getIndentPrefixes(ISourceViewer sourceViewer,
			String contentType) {
		return new String[] { "\t", "        " };
	}
 
	protected ContentAssistPreference getContentAssistPreference() {
		return ExamplePythonContentAssistPreference.getDefault();
	}
}

ExamplePythonContentAssistPreference.java 

public class ExamplePythonContentAssistPreference extends
		ContentAssistPreference {
 
	private static ExamplePythonContentAssistPreference instance;
 
	public static ContentAssistPreference getDefault() {
		if (instance == null) {
			instance = new ExamplePythonContentAssistPreference();
		}
		return instance;
	}
 
	protected ScriptTextTools getTextTools() {
		return ExamplePythonUI.getDefault().getTextTools();
	}
}

Add getTextTools() method to the ExamplePythonEditor class: 

public ScriptTextTools getTextTools() {
	return ExamplePythonUI.getDefault().getTextTools();
}

Partitioning.

Partition is a range of source code with a specific kind. For example: code partition (for source code), commentary partition (for source code comments), string partition (for strings in source code).

Partitions are used for highlighting. We can associate different scanners with different partitions. Content assistance is also based on partitions: we can use different completions for different partitions.

Add partitions scanner for python partitions support. We will use a rule based partition scanner (provided by Eclipse Platform) with a set of custom rules. ExamplePythonPartitionScanner.java 

public class ExamplePythonPartitionScanner extends RuleBasedPartitionScanner {
	public ExamplePythonPartitionScanner() {
		IToken string = new Token(IExamplePythonPartitions.PYTHON_STRING);
		IToken comment = new Token(IExamplePythonPartitions.PYTHON_COMMENT);
 
		List rules = new ArrayList();
 
		rules.add(new EndOfLineRule("#", comment));
		rules.add(new MultiLineRule("\"\"\"", "\"\"\"", string, '\\'));
		rules.add(new MultiLineRule("\'\'\'", "\'\'\'", string, '\\'));
		rules.add(new MultiLineRule("\'", "\'", string, '\\'));
		rules.add(new MultiLineRule("\"", "\"", string, '\\'));
 
		IPredicateRule[] result = new IPredicateRule[[rules.size()]];
		rules.toArray(result);
		setPredicateRules(result);
	}
}

Modify ExamplePythonTextTools class and return partition scanner from getPartitionScanner() method: ExamplePythonTextTools.java 

public class ExamplePythonTextTools extends ScriptTextTools {
 
	private final static String[] LEGAL_CONTENT_TYPES = new String[] {
			IExamplePythonPartitions.PYTHON_STRING,
			IExamplePythonPartitions.PYTHON_COMMENT };
 
	private IPartitionTokenScanner fPartitionScanner;
 
	public ExamplePythonTextTools(boolean autoDisposeOnDisplayDispose) {
		super(IExamplePythonPartitions.PYTHON_PARTITIONING,
				LEGAL_CONTENT_TYPES, autoDisposeOnDisplayDispose);
		fPartitionScanner = new ExamplePythonPartitionScanner();
	}
 
	public ScriptSourceViewerConfiguration createSourceViewerConfiguraton(
			IPreferenceStore preferenceStore, ITextEditor editor,
			String partitioning) {
		return new ExamplePythonSourceViewerConfiguration(getColorManager(),
				preferenceStore, editor, partitioning);
	}
	public IPartitionTokenScanner getPartitionScanner() {
		return fPartitionScanner;
	}
}

We will use this participant class from ExamplePythonEditor connectPartitioningToElement() method: 

protected void connectPartitioningToElement(IEditorInput input,
			IDocument document) {
		if (document instanceof IDocumentExtension3) {
			IDocumentExtension3 extension = (IDocumentExtension3) document;
			if (extension.getDocumentPartitioner(IExamplePythonPartitions.PYTHON_PARTITIONING) == null) {
				ExamplePythonTextTools tools = ExamplePythonUI.getDefault().getTextTools();
		                tools.setupDocumentPartitioner(document, IExamplePythonPartitions.PYTHON_PARTITIONING);
			}
		}
	}

Color constants.

We will use three highlighting types in our editor: keyword, string, comment. All another types can be easily added. DLTK provides a set of base classes for easy highlighting configuration.

Some steps are required:

  • Specify color constants
  • Provide org.eclipse.core.runtime.preferences extension point implementation and set color values for constants.
  • Use constants in highlighting rules.
  • Add color configuration pages using DLTK base classes.

So let's go thought these steps:

  • Color constants:*

IExamplePythonColorConstants contains constants for editor IExamplePythonColorConstants.java 

public interface IExamplePythonColorConstants {
	public static final String PYTHON_STRING = DLTKColorConstants.DLTK_STRING; //$NON-NLS-1$
	public static final String PYTHON_COMMENT = DLTKColorConstants.DLTK_SINGLE_LINE_COMMENT; //$NON-NLS-1$
	public static final String PYTHON_KEYWORD = DLTKColorConstants.DLTK_KEYWORD; //$NON-NLS-1$
	public static final String PYTHON_DEFAULT = DLTKColorConstants.DLTK_DEFAULT; //$NON-NLS-1$
}
  • Preference initialization code:*

Let's extend org.eclipse.core.runtime.preferences extension point and create class ExamplePythonUIPreferenceInitializer: 

<extension
         point="org.eclipse.core.runtime.preferences">
      <initializer
            class="org.eclipse.dltk.examples.python.internal.ExamplePythonUIPreferenceInitializer">
      </initializer>
   </extension>

ExamplePythonUIPreferenceInitializer.java 

public class ExamplePythonUIPreferenceInitializer extends
		AbstractPreferenceInitializer {
 
	public void initializeDefaultPreferences() {
		IPreferenceStore store = ExamplePythonUI.getDefault()
				.getPreferenceStore();
 
		EditorsUI.useAnnotationsPreferencePage(store);
		EditorsUI.useQuickDiffPreferencePage(store);
 
		// Initialize DLTK default values
		PreferenceConstants.initializeDefaultValues(store);
 
		// Initialize python constants
		PreferenceConverter.setDefault(store, IExamplePythonColorConstants.PYTHON_COMMENT, new RGB(63, 127, 95));
		PreferenceConverter.setDefault(store, IExamplePythonColorConstants.PYTHON_KEYWORD, new RGB(127, 0, 85));
		PreferenceConverter.setDefault(store, IExamplePythonColorConstants.PYTHON_STRING,  new RGB(42, 0, 255));
 
		store.setDefault(IExamplePythonColorConstants.PYTHON_COMMENT + PreferenceConstants.EDITOR_BOLD_SUFFIX, false);
		store.setDefault(IExamplePythonColorConstants.PYTHON_COMMENT + PreferenceConstants.EDITOR_ITALIC_SUFFIX, false);
 
		store.setDefault(IExamplePythonColorConstants.PYTHON_KEYWORD + PreferenceConstants.EDITOR_BOLD_SUFFIX, true);
		store.setDefault(IExamplePythonColorConstants.PYTHON_KEYWORD + PreferenceConstants.EDITOR_ITALIC_SUFFIX, false);
 
		store.setDefault(PreferenceConstants.EDITOR_TAB_WIDTH, 8);
		store.setDefault(PreferenceConstants.EDITOR_SYNC_OUTLINE_ON_CURSOR_MOVE, true);
 
		store.setDefault(CodeFormatterConstants.FORMATTER_TAB_CHAR, CodeFormatterConstants.TAB);
		store.setDefault(CodeFormatterConstants.FORMATTER_TAB_SIZE, "8");
		store.setDefault(CodeFormatterConstants.FORMATTER_INDENTATION_SIZE,"8");
	}
}

ExamplePythonUIPreferenceInitializer class initializes some default DLTK preferences.

Partitions: We should provide scanner for each partition type already specified. Scanners will scan partitions and return color regions: 

private AbstractScriptScanner fCodeScanner;
	private AbstractScriptScanner fStringScanner;
	private AbstractScriptScanner fCommentScanner;
        //This method called from base class.
        protected void initializeScanners() {
                //This is our code scanner
		this.fCodeScanner = new ExamplePythonCodeScanner(
				this.getColorManager(), this.fPreferenceStore);
                // This is default scanners for partitions with same color.
		this.fStringScanner = new SingleTokenScriptScanner(this
				.getColorManager(), this.fPreferenceStore,
				IExamplePythonColorConstants.PYTHON_STRING);
		this.fCommentScanner = new SingleTokenScriptScanner(this
				.getColorManager(), this.fPreferenceStore,
				IExamplePythonColorConstants.PYTHON_COMMENT);
	}
        public IPresentationReconciler getPresentationReconciler(
			ISourceViewer sourceViewer) {
		PresentationReconciler reconciler = new ScriptPresentationReconciler();
		reconciler.setDocumentPartitioning(this
				.getConfiguredDocumentPartitioning(sourceViewer));
 
		DefaultDamagerRepairer dr = new DefaultDamagerRepairer(
				this.fCodeScanner);
		reconciler.setDamager(dr, IDocument.DEFAULT_CONTENT_TYPE);
		reconciler.setRepairer(dr, IDocument.DEFAULT_CONTENT_TYPE);
 
		dr = new DefaultDamagerRepairer(this.fStringScanner);
		reconciler.setDamager(dr, IExamplePythonPartitions.PYTHON_STRING);
		reconciler.setRepairer(dr, IExamplePythonPartitions.PYTHON_STRING);
 
		dr = new DefaultDamagerRepairer(this.fCommentScanner);
		reconciler.setDamager(dr, IExamplePythonPartitions.PYTHON_COMMENT);
		reconciler.setRepairer(dr, IExamplePythonPartitions.PYTHON_COMMENT);
 
		return reconciler;
	}
        public void handlePropertyChangeEvent(PropertyChangeEvent event) {
		if (this.fCodeScanner.affectsBehavior(event)) {
			this.fCodeScanner.adaptToPreferenceChange(event);
		}
		if (this.fStringScanner.affectsBehavior(event)) {
			this.fStringScanner.adaptToPreferenceChange(event);
		}
	}
 
	public boolean affectsTextPresentation(PropertyChangeEvent event) {
		return this.fCodeScanner.affectsBehavior(event)
				|| this.fStringScanner.affectsBehavior(event);
	}

Code scanner for python language. Support of keyword, string and commentary highlighting. ExamplePythonCodeScanner.java 

public class ExamplePythonCodeScanner extends AbstractScriptScanner {
	private static String[] fgKeywords = { "and", "del", "for", "is", "raise",
			"assert", "elif", "from", "lambda", "break", "else", "global",
			"not", "try", "class", "except", "if", "or", "while", "continue",
			"exec", "import", "pass", "yield", "def", "finally", "in", "print",
			"self", "return" };
	private static String fgTokenProperties[] = new String[] {
			IExamplePythonColorConstants.PYTHON_COMMENT,
			IExamplePythonColorConstants.PYTHON_DEFAULT,
			IExamplePythonColorConstants.PYTHON_KEYWORD };
 
	public ExamplePythonCodeScanner(IColorManager manager, IPreferenceStore store) {
		super(manager, store);
		this.initialize();
	}
 
	protected String[] getTokenProperties() {
		return fgTokenProperties;
	}
 
	protected List createRules() {
		List rules = new ArrayList();
		IToken keyword = this.getToken(IExamplePythonColorConstants.PYTHON_KEYWORD);
		IToken comment = this.getToken(IExamplePythonColorConstants.PYTHON_COMMENT);
		IToken other = this.getToken(IExamplePythonColorConstants.PYTHON_DEFAULT);
		// Add rule for single line comments.
		rules.add(new EndOfLineRule("#", comment));
		// Add generic whitespace rule.
		rules.add(new WhitespaceRule(new ExamplePythonWhitespaceDetector()));
		// Add word rule for keywords.
		WordRule wordRule = new WordRule(new ExamplePythonWordDetector(), other);
		for (int i = 0; i < fgKeywords.length; i++) {
			wordRule.addWord(fgKeywords[[i]], keyword);
		}
		rules.add(wordRule);
		this.setDefaultReturnToken(other);
		return rules;
	}
	public class ExamplePythonWhitespaceDetector implements IWhitespaceDetector {
		public boolean isWhitespace(char character) {
			return Character.isWhitespace(character);
		}
	}
	public class ExamplePythonWordDetector implements IWordDetector {
		public boolean isWordPart(char character) {
			return Character.isJavaIdentifierPart(character);
		}
		public boolean isWordStart(char character) {
			return Character.isJavaIdentifierStart(character);
		}
	}
}

Running: Dltk 23 script editor.png

Getting the code.

The tutorial source code is available from Eclipse Git:

End of the second part.

Retrieved from "https://wiki.eclipse.org/index.php?title=DLTK_IDE_Guide:Step_2._Towards_an_Editor&oldid=318318"

This page was last modified 15:26, 13 October 2012 by Eclipsepedia anonymous user Unnamed Poltroon.

Back to the top