MemoryAnalyzer/Extending Memory Analyzer

Introduction

The Memory Analyzer tool offers several possibilities to extend it. This page contains an overview of the different extension points and what can be achieved with them.

Within the extensions one will usually extract certain pieces of information from the objects in the heap dump. See MemoryAnalyzer/Reading Data from Heap Dumps for more details on how to read data from a heap dump.

Setting up a development environment for writing extensions

It is not necessary to download the source of Memory Analyzer to be able to write extensions. A recent binary version is sufficient.

1. Have a copy of an Eclipse Java Development environment installed
2. Download a recent copy of of Memory Analyzer [from October 2010 or later]
   1. Download Memory Analyzer 1.12.0 Download site
   2. Update using update site (if you haven't got the latest version) Update site
3. Create MAT as a target platform:
   1. Windows->Preferences->Plug-in Development->Target Platform
   2. Add->Nothing->Next
   3. Name: MAT
   4. Locations->Add->Installation
   5. Location: path_to_MAT/mat
   6. Finish
   7. Select MAT as active target platform
4. Create a new plug-in project:
   1. File->New->Other->Plug-in project
   2. Name: MAT Extension
   3. ->Next
   4. Execution Environment: JavaSE-1.8 (that's all MAT currently requires)
   5. No activator (unless you are doing something complicated)
   6. No UI contribution
   7. No API analysis
   8. No template
   9. ->Finish
   10. Dependencies
       1. add org.eclipse.mat.api
       2. Save (cntl-S)
   11. Extensions
       1. select org.eclipse.mat.api.nameResolver
       2. ->Finish
       3. click on impl
       4. Adjust package name and class name to suit
       5. ->Finish
   12. Add for example

       @Subject("java.lang.Runtime")

       before the class definition

   13. Organize imports (cntl-shift-O)
   14. Edit the code to perform the required function. For example

       In

       public String resolve(IObject object)

       Change

       return null;

       to

       return "The Java runtime of size"+object.getUsedHeapSize();

       Note the hover javadoc help for IObject, IClassSpecificNameResolver. Note the method list for object.

   15. Save
5. To test:
   1. Select Plug-in, Run As->Eclipse Application
6. To package
   1. File->Export->Plug-in Development->Deployable plug-ins and fragments
   2. ->next
   3. select plug-in
   4. Destination: Directory: path_to_MAT/mat
   5. ->Finish

The Name Resolver Extension

The name resolver extension point provides a mechanism to give a readable description of an object, similar to what a toString() method will do. Some extensions which MAT provides are for example for to show the content of objects representing String, to show the bundle symbolic name for Equinox classloaders, to show the name for Thread objects, etc…

The extension should implement the IClassSpecificNameResolver interface which defines a single method.

public String resolve(IObject object) throws SnapshotException;

The method takes an IObject as an argument and should return a string representation.

To specify the class for which the resolver should be used one can use the @Subject( annotation.

The method getClassSpecificName of IObject will look for extensions which match the class of the object and execute the resolve() method to return the proper String. Thus it is relatively easy to return a description based on one or more String fields, as strings are already resolved.

Here is a sample implementation that will return the name of an Eclipse Job:

@Subject("org.eclipse.core.runtime.jobs.Job")
public class JobNameResolver implements IClassSpecificNameResolver
{
 
	@Override
	public String resolve(IObject object) throws SnapshotException
	{
		IObject name = (IObject) object.resolveValue("name");
		if (name != null) return name.getClassSpecificName();
		return null;
	}
 
}

Queries in Memory Analyzer

Introduction to Queries

Most of the functionality in Memory Analyzer which is exposed to the user of the tool is provided via queries (implementing the IQuery interface, for example "Histogram", "Retained Set", etc... Queries extract and process data from the heap dump using the MAT's API, and provide the result to the user in the form of a table, a tree, free text, etc... Queries show up in the "Queries" menu of the tool, and often in the context menus on objects. An important feature of the queries is that they can "collaborate", i.e. the user can use (part of) the result of one query and pass it as input parameters to another query. Here is an example of such "cooperation" - you select "Histogram" to show a class histogram of all objects, then choose say java.util.HashMap and from the context menu call "Retained Set". You can then select a line in this retained set and pass the corresponding objects to yet another (possibly created by you) query.

The IQuery Interface

To implement a query one needs to implement the IQuery interface. The implementation should provide a (default) constructor without parameters.

The IQuery interface defines just one method:

    public IResult execute(IProgressListener listener) throws Exception {

As a parameter one gets only a progress listener (IProgressListener) to report progress. All other input that a query needs is deaclared with annotations on the fields of the query and injected from Memory Analyzer at runtime. The fields used for arguments injection should be declared public. For more details on getting input, see #Passing Arguments to a Query. The return type of the execute method is IResult, which is just a marker interface. The different result types are described in the section #Query Results.

Scope

Queries are stateless. Every time the user executes a query a new instance of the IQuery implementation is created. The required input is injected into the fields of the instance and the execute method is called.

Describing the Query with Annotations

When you write a query it will appear in the context menus, and Memory Analyzer will open an Arguments Wizard for specifying the prequired arguments, and this wizard will also show some help for the query and its arguments. The metadata - how a query will be named, under which category (sub-menu) it will appear, the help text, etc... are provided by annotating the query.

The following meta-data related annotations are available:

• @CommandName – used for command line and query browser
• @Name – the visible name on the menu, nn| to set order
• @Category – the menu section (sub-menu), / to cascade, nn| to set order
• @Help – explanation of query
• @HelpUrl – link into the help system
• @Icon – icon for the query (shown in the menu)
• @Usage – example usage – defaults to command name + args

The values can also be externalized. To do so, put them into an annotations.properties file in the package directory.

Sample code snippet:

@Category("Sample Queries")
@Name("List Jobs Query")
@Help("This is a sample query, which lists all jobs with a given name")
public class SampleQuery implements IQuery {
 
...

To externalize these values, a annotations.properties will look something like:

SampleQuery.category = Sample Queries
SampleQuery.name = List Jobs Query
SampleQuery.help = This is a sample query, which lists all jobs with a given name

Passing Arguments to a Query

A nice property of queries is that they can interact with each other. In other words, parts of the result from one query (say one line in a histogram) can be passed to a different query using the context menus. Therefore, queries should just declare what kind of arguments they require and delegate to the Memory Analyzer to collect this information and inject it into the queries before executing them. Memory Analyzer does this by opening the Arguments Wizard.

To declare an input parameter a query has to define a public field and annotate it with the @Argument annotation. To provide a help message specific on the concrete argument, add also the @Help annotation to the public field.

The following types are currently supported as arguments:

ISnapshot 

the snapshot corresponding to the currently open editor

IHeapObjectArgument 

good way of getting objects

String, Pattern, int, boolean, float, double 

these get supplied via query wizard or via command line

IContextObject 

row with one object

IContextObjectSet 

row with multiple objects or OQL query to return those objects

IQueryContext 

a more general way of extracting information about the snapshot which is not tied to the snapshot API

arrays or lists of the above 

use them for multiple items

enums 

can be used to provide a fixed choice list

File 

for input or output files

Comparison Queries

Comparison queries are run from the Compare Basket but are invoked in a similar way. Each row of the Compare Basket is a whole result of a previous query; either a tree or table. Queries with arguments suitable for a comparison operation are only offered in the Compare Menu and not from the editor pane. Comparison arguments are as follows:

List or array of IResultTable 

for comparison queries only operating on tables

List or array of IResultTree 

for comparison queries only operating on trees

List or array of IStructuredResult 

for comparison queries operating on tables and trees

List or array of RefinedTable 

for comparison queries only operating on tables, uses the filtered and sorted version of the previous result with any derived columns like retained size

List or array of RefinedTree 

for comparison queries only operating on tables, uses the filtered and sorted version of the previous result with any derived columns like retained size

List or array of RefinedStructuredResult 

for comparison queries operating on tables and trees, uses the filtered and sorted version of the previous result with any derived columns like retained size

List or array of ISnapshot 

the snapshots corresponding to the tables / trees, in the same order

Consider using RefinedStructuredResult for your comparison queries as the query may then be more flexible for the end user.

Standard arguments available to comparison queries

String, Pattern, int, boolean, float, double 

these get supplied via query wizard or via command line

enums 

can be used to provide a fixed choice list

File 

for input or output files

See CompareTablesQuery.java for an example.

Qualifications on Query Arguments

The following parameters on the @Argument annotation can be used to specify some further restrictions / hints:

isMandatory 

a boolean parameter to tell MAT if it can execute the query without the argument

flag 

a String used instead of the field name to identify the argument in the command line and in the query browser

Advice 

qualifies the way data is inserted into the field

• HEAP_OBJECT 

  the int or Integer is an object id, not a number

  SECONDARY_SNAPSHOT 

  the snapshot is another snapshot, which should be prompted for, not the current one

  CLASS_NAME_PATTERN 

  the pattern will be used to match class names

  DIRECTORY 

  the file parameter is meant to be a directory

  SAVE 

  the file parameter is meant to be used to save data

Reading data from supplied arguments

See MemoryAnalyzer/Reading Data from Heap Dumps for how to extract data from supplied arguments, including ISnapshot, IObject and object IDs.

Calling One Query from Another

Supplied queries are not a Memory Analyzer API, so user written queries should not link to them directly. It is possible to call them by name, though the query names and arguments can vary from release to release.

String query = "SELECT s, toString(s) from java.lang.String s";
IResult ir = SnapshotQuery.lookup("oql", snapshot).setArgument("queryString", query).execute(listener);

or

SnapshotQuery query = SnapshotQuery.parse("dominator_tree -groupby BY_CLASSLOADER", snapshot);
IResultTree t = (IResultTree)query.execute(new VoidProgressListener());

which shows how an enum argument can be set. Setting them directly via setArgument often doesn't work as the enum type is inaccessible.

Query Results

TextResult 

A simple result that renders its input as text.

IStructuredResult 

A way of display data about lots of objects

• IResultTable 

  A table of objects

  • Histogram 

    A table of objects where each row is all the objects of one class

    ListResult 

    A way of displaying a Java List of things, where the fields from each thing are also given

    PropertyResult 

    A way of displaying details about one object based on a list of attributes

  IResultTree 

  A tree of objects

IResultPie 

A pie chart

QuerySpec 

A good way of displaying the results of executing another query

TODO improve description, add remaining results

Reports in Memory Analyzer

Introduction

Several queries can be combined into a report which could then be run from the Run Report... menu option or in batch mode.

Report definition

The report definition is written in XML and can be validated using the schema held in org.eclipse.mat.report/schema/report.xsd. The result of running a report will be an HTML page or a CSV data file. Queries can be run using the query element, using the command element to specify which command should be run. Other reports can be run using the template element. The section element can be used to combine multiple query, template and section elements and is displayed in a report as a collapsible part or a separate file.

Two examples show how the report definition is written. overview.xml has a section and suspects.xml also has a template element referencing another report to be run and included.

The param element allows output to be controlled. The params can be used to control generation of tables - as HTML or CSV files, to limit or increase the number of lines displayed, and to omit, sort or filter columns. Some params just control the current section - some also control any inner sections unless overridden. A param can also be used elsewhere in the report as ${param_name} - in the command name and other param values. Values are documented in param. keys, param.html. keys and param.rendering. keys, more conveniently seen at Constant field values. Since Memory Analyzer 1.9, parameters can be passed to a report definition using ParseHeapDump as options thus: ParseHeapDump myheapdump.hprof -myparam=myparam_value myreport.xml. A param value given on the command line will override a value given in a report definition.

The report definition can be tested as Run Expert System Test > Run Report.

Report definition extension point

If a report definition is incorporated into a plug-in then the report definition extension point should be used. Then when the new plug-in is installed into MAT the report will be available to run from the Memory Analyzer GUI.

Values in the report definition can then be externalized using "%myval Default value" where myval is also defined in the plugin.properties file.

Executing a Report in Unattended Mode

Once you create a Report extension you can find it next to other reports like the "Leak Suspects" in the Memory Analyzer GUI. Besides this, one can execute reports in an unnatended mode by running the "org.eclipse.mat.api.parse" application.

Here is an example how to setup a Run Configuration to execute the report "my_repory" located in the plugin "my_report_plugin":

• Run As -> Run Configurations
• Set "Run An Application" to "org.eclipse.mat.api.parse"
• Set the Program arguments to

${file_prompt} my_report_plugin:my_repory

When you run this you will get a popup to select a heap dump file. It will be then parsed and the report will be executed for it. The result however is not open in the IDE, it is saved in the file system next to the dump.

Parameters

You can also define options on the command line. This can be as

${file_prompt} "-my_parm=my special value" my_report_plugin:my_repory

and then access the variable inside the report XML as

<section name="My report" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://www.eclipse.org/mat/report.xsd"
        xsi:schemaLocation="http://www.eclipse.org/mat/report.xsd platform:/resource/org.eclipse.mat.report/schema/report.xsd">
        <param key="my_parm" value="default value" />
        <query name="OQL Query command">
                <command>my_query -myopt "${my_parm}"</command>
        </query>
</section>

Request Resolvers

Introduction

A request resolver is a piece of coding which is capable of extracting detais about what a thread was doing, using the information from the thread object and its java local objects. The information provided by a request resolver is included in the Leak Suspects report.

When is this useful? There are often OutOfMemoryErrors which are not caused by a memory leak, but rather by some "greedy" operation - an attempt to load a huge file fully into memory, an attempt to build scan a whole DB table and keep the results in memory, etc... In such cases the Leak Suspect report will often point to the thread as the suspect object, because its local objects (objects on the thread's stack) are eating too much memory. In such cases it is very helpful to know what the thread was doing and to get some insights on this activity.

Examples:

• tell that a thread was processing an HTTP request and extract the concrete request and some parameters
• show an SQL statement that has been processed when an OOM error occured
• display the name of the Eclipse Job which has been processed
• etc...

The IRequestDetailsResolver Interface

To implement a request resolver one needs to implement the IRequestDetailsResolver interface. To specify for which type of local objects this request resolver can provide information, use the the @Subject annotation on the implementation class.

The interface defines just one method:

    void complement(ISnapshot snapshot, IThreadInfo thread, int[] javaLocals, int thisJavaLocal,
                    IProgressListener listener) throws SnapshotException;

The complement method will be called by Memory Analyzer whenever it collects information for a thread and the thread has a local object (somewhere on the stack) which matches the type specified by @Subject. As parameters one will receive all necessary context information to extract the needed information:

• snapshot - the whole dump
• thread - a IThreadInfo object representing the thread being analyzed
• javaLocals - all the local variables, as object IDs
• thisJavaLocal - the object ID of the local object matching the @Subject

Within the complement() method one should extract helpful information about the activity of the thread and add it to the IThreadInfo object using the addRequest() method. The addRequest() method takes two parameters - a String with short description appearing on the first page of the Leak Suspects report, and an IResult with more details about the request (could be a table with all properties, etc...).

Here is a code sample for a request resolver:

/* Specify that I can extract information from ProgressManager$JobMonitor objects */
@Subject("org.eclipse.ui.internal.progress.ProgressManager$JobMonitor")
public class JobRequestResolver implements IRequestDetailsResolver {
 
    @Override
    public void complement(ISnapshot snapshot, IThreadInfo thread,
	    int[] javaLocals, int thisJavaLocal, IProgressListener listener)
	    throws SnapshotException {
 
	IObject monitor = snapshot.getObject(thisJavaLocal); // get the IOjbect for the JobMonitor
	IObject job = (IObject) monitor.resolveValue("job"); // get the value of the job field
	String jobName = job.getClassSpecificName(); // get the symbolic represenation of the job
 
	String summary = "This thread executes the job [" + jobName + "]";
	IResult sampleDetails = new TextResult("Job object is = [" + job.getDisplayName() + "]");
 
	thread.addRequest(summary, sampleDetails); // add the request information
 
	thread.addKeyword(jobName); // add the job name to the keywords
    }
}

This sample request resolver will add to the leak suspect report a line like:

This thread executes the job [Sample greedy job]

if the thread was processing an Eclipse job. It will add as details the object instance of the Job implementation.

Adding a New Heap Dump Format

Memory Analyzer can also be extended to support more heap dump formats. A detailed description how to do this can be found here: MemoryAnalyzer/Adding a new heap dump format to the Memory Analyzer

Contributing back to the project

If your extension to Memory Analyzer would be useful to other people, please consider contributing it back to the project.