Skip to main content

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.

Jump to: navigation, search

MemoryAnalyzer/OQL

Object Query Language

Object Query Language is an SQL like language used by Memory Analyzer for exploring a heap dump. There is documentation in the help but this wiki allows newer features to be explained and discussed before the documentation is updated.

Simple 

SELECT * FROM java.lang.String

Displays all String objects as a tree. 

SELECT s as String,s.value as "characters" FROM java.lang.String s

Displays all String objects as a table. 

SELECT s as String,s.value as "characters", inbounds(s),inbounds(s).@length FROM java.lang.String s
String                          |characters                                 |inbounds(s)| inbounds(s).@length
--------------------------------------------------------------------------------------------------------------
java.lang.String [id=0x22e58820]|char[] [id=0x22e60f50;length=16;size=48]   |[I@620f7a39|                   1
java.lang.String [id=0x22e59150]|char[] [id=0x22e62ff0;length=6;size=24]    |[I@1f7b8d59|                   1
java.lang.String [id=0x22e5b560]|char[] [id=0x22e6b730;length=537;size=1088]|[I@28551755|                   1
--------------------------------------------------------------------------------------------------------------

There are two sorts of objects encountered with OQL, IObject which represent Java objects in the snapshot and regular Java objects generated by OQL processing.

java.lang.String [id=0x22e58820] is a IInstance representing a String from the snapshot.

char[] [id=0x22e60f50;length=16;size=48] is an IPrimitiveArray representing a character array from the snapshot.

[I@620f7a39 is a regular Java integer array holding a several of ints which are the object IDs Memory Analyzer uses to represent IObjects in the snapshot.

OQL (Memory Analyzer) versus SQL (MAT/Calcite)

As well as the built-in OQL, there is an extension plug-in for MAT called MAT Calcite which adds SQL processing

Topic OQL SQL
General syntax 
SELECT s FROM java.lang.String s
 
SELECT s.this FROM "java.lang.String" s
Built-in functions 
SELECT toString(s), classof(s), 
s.@objectAddress, s.@usedHeapSize, s.@retainedHeapSize
FROM java.lang.String s
 
SELECT toString(s.this),getType(s.this),
getAddress(s.this),shallowSize(s.this),retainedSize(s.this) 
FROM "java.lang.String" s
More functions 
SELECT h, h[0:-1].size(), h.table, 
h.table.@length, h.modCount, h.getField("modCount") 
FROM java.util.HashMap h
 
SELECT h.this,getSize(h.this),h.this['table'],
LENGTH(h.this['table']), h.this['modCount'], getField(h.this,'modCount') 
FROM "java.util.HashMap" h
Comments /* multi-line comment */ /* multi-line comment */
Single line comment // comment -- comment
JOIN Simulated by JOIN Operations Supported
LIMIT and OFFSET Simulated by LIMIT and OFFSET Supported
ORDER BY Click on column headers to sort Supported
GROUP BY Can be simulated by GROUP BY.

Also 'Java Basics > Group by Value' query might help. Also, if the row is backed by an object (the from clause returned a list of objects) then the 'Group by' menu bar option allows 'Group by classloader' and 'Group by package'.

Supported
COUNT Can be simulated by COUNT. Supported
MAX,MIN Not directly supported. Could be

simulated by clicking on a column name and taking the top or bottom value.

Supported
AVG,SUM Not supported Supported

OQL/Calcite provides advanced SQL functions such as JOIN (INNER JOIN,LEFT JOIN,RIGHT JOIN,FULL JOIN), CROSS JOIN, GROUP BY, ORDER BY. Some can be simulated by OQL with a bit of work.

Enhancements in November 2019

Various updates and enhancements have been made to OQL under 552879: OQL enhancements for sub-selects, maps, context providers, DISTINCT. These are available from snapshot builds for testing and are subject to change. Please comment on the bug, forum or development mailing list

Some of the aims of these changes were to permit more complex queries such as:


SELECT DISTINCT

`DISTINCT` used to just operate on the results of a query if it returned objects rather than general select items. 

SELECT DISTINCT OBJECTS classof(s) FROM "java.lang\.S.*" s
SELECT * FROM OBJECTS "java.lang\.S.*"

`DISTINCT` now also operates on SELECTs with select items. It uses the whole row considered as a list as the item to be considered as distinct. It also uses the optimization that the input FROM items are also considered as being distinct (either as ints/IObjects or more general `FROM` items). 

SELECT DISTINCT classof(s) FROM "java.lang\.S.*" s

Sub SELECT with select items

sub-selects were permitted where the sub-select returned an object list. 

SELECT v, v.@length FROM OBJECTS ( SELECT OBJECTS s.value FROM java.lang.String s  ) v

sub-selects are now also permitted which have select items. 

SELECT v,v.s,v.val FROM OBJECTS ( SELECT s,s.value as val FROM java.lang.String s  ) v
Row                                                                                |Object                          |Array
--------------------------------------------------------------------------------------------------------------------------------------------------------------
{s=java.lang.String [id=0x26ba8a30], val=char[] [id=0x26ba8a48;length=14;size=40]} |java.lang.String [id=0x26ba8a30]|char[] [id=0x26ba8a48;length=14;size=40]
{s=java.lang.String [id=0x26ba8998], val=char[] [id=0x26ba89b0;length=56;size=128]}|java.lang.String [id=0x26ba8998]|char[] [id=0x26ba89b0;length=56;size=128]
{s=java.lang.String [id=0x26ba8160], val=char[] [id=0x26ba8178;length=56;size=128]}|java.lang.String [id=0x26ba8160]|char[] [id=0x26ba8178;length=56;size=128]
{s=java.lang.String [id=0x26b9d390], val=char[] [id=0x26b9d3a8;length=15;size=48]} |java.lang.String [id=0x26b9d390]|char[] [id=0x26b9d3a8;length=15;size=48]
{s=java.lang.String [id=0x26b9d358], val=char[] [id=0x26b9d370;length=8;size=32]}  |java.lang.String [id=0x26b9d358]|char[] [id=0x26b9d370;length=8;size=32]
--------------------------------------------------------------------------------------------------------------------------------------------------------------

The outer select processes the result of the sub-select row by row, with a single RowMap `Map` object representing the row. The key/value pairs are the sub-select items with the sub-select column names as the keys. If the keys are standard identifiers, i.e. generally alpha-numeric then attribute processing can be used as `v.s` rather than having to to `v.get("s2")` which can still be used, perhaps for column names with spaces.

The whole sub-select continues to return a `CustomTableResultSet` which is an `IResultTable` but this has been enhanced to also be a `List` of `RowMap` items. It is quite hard to operate in OQL on the whole result as if it is supplied to an outer select then the `Iterable` nature means it will be processed row by row. 

SELECT * FROM OBJECTS ( SELECT s, s.value AS val FROM java.lang.String s ) v
[{s=java.lang.String [id=0x26ba8a30], val=char[] [id=0x26ba8a48;length=14;size=40]}, {s=java.lang.String [id=0x26ba8998], val=char[] [id=0x26ba89b0;length=56;size=128]}, {s=java.lang.String [id=0x26ba8160], val=char[] [id=0x26ba8178;length=56;size=128]}, {s=java.lang.String [id=0x26b9d390], val=char[] [id=0x26b9d3a8;length=15;size=48]}, {s=java.lang.String [id=0x26b9d358], val=char[] [id=0x26b9d370;length=8;size=32]}, {s=java.lang.String [id=0x26b9d318], val=char[] [id=0x26b9d330;length=11;size=40]}, {s=java.lang.String [id=0x26b9d2e8], val=char[] [id=0x26b9d300;length=4;size=24]}, {s=java.lang.String [id=0x26b9c758], val=char[] [id=0x26b9c770;length=21;size=56]}, {s=java.lang.String [id=0x26b9c6c8], val=char[] [id=0x26b9c6e0;length=13;size=40]}, {s=java.lang.String [id=0x26b9c690], val=char[] [id=0x26b9c6a8;length=8;size=32]}, ...

Shows the whole table as a list

LIMIT and OFFSET

SQL has LIMIT and OFFSET to choose only some of the items from the FROM clauses. This can be simulated in OQL. 

SELECT eval((SELECT * FROM OBJECTS ( SELECT s, s.value AS val FROM java.lang.String s ) v))[3] FROM OBJECTS 0
eval((SELECT * FROM OBJECTS ( SELECT s, s.value AS val FROM java.lang.String s  ) v ))[3]
-----------------------------------------------------------------------------------------
{s=java.lang.String [id=0x26b9d390], val=char[] [id=0x26b9d3a8;length=15;size=48]}
-----------------------------------------------------------------------------------------

Processes the whole table as a select item.

This could be used to simulate SQL LIMIT and OFFSET clauses. 

SELECT z.s FROM OBJECTS ( eval((SELECT s FROM "java.lang.String" s ))[10:29] ) z

This extracts 20 entries, skipping the first 10. Note the array slice processing, with the start and end offsets as 0-based but inclusive.

Compare with MAT Calcite (SQL) 

SELECT s.this FROM "java.lang.String" s LIMIT 10 offset 20

Context Menu for object columns

If a column appears to hold heap objects, or lists or arrays of heap objects, then the context menu now offers a choice to process that column's item of the selected rows. 

SELECT s AS String, s.value AS "Char array", inbounds(s) AS Inbounds FROM java.lang.String s
String                          |Char array                               |Inbounds
--------------------------------------------------------------------------------------
java.lang.String [id=0x26ba8a30]|char[] [id=0x26ba8a48;length=14;size=40] |[I@6ad112de
java.lang.String [id=0x26ba8998]|char[] [id=0x26ba89b0;length=56;size=128]|[I@18a0721b
java.lang.String [id=0x26ba8160]|char[] [id=0x26ba8178;length=56;size=128]|[I@2ae2fa13
--------------------------------------------------------------------------------------

Context Menu:

SELECT ... s
The entire row - based on the underlying object s. Copy OQL: 
SELECT s AS String, s.value AS "Char array", inbounds(s) AS Inbounds FROM OBJECTS 20798,20796,20793 s
String
Just the String item in column 'String'. Copy OQL: 
SELECT s AS String FROM OBJECTS 20798,20796,20793 s
Char array
Just the char array. Copy OQL: 
SELECT s.value AS "Char array" FROM OBJECTS 20798,20796,20793 s
Inbounds
All the inbounds as heap objects. Copy OQL: 
SELECT inbounds(s) AS Inbounds FROM OBJECTS 20798,20796,20793 s


The context menu has a `Copy > OQL Query` option which returns an OQL query representing the selected rows and appropriate column.

The report plug-in which converts result tables to HTML now uses the context menu name to match with the table column to put HTML linksin the correct place across the columns in the table rather than always in the first column. This also applies for other queries, so the system properties query when used in a report has in-place links for keys and values.

Question - should the context menu appear for all columns, in case a column has a heap object in rows other than the first. The context menu would then appear for non-object columns holding strings or numeric values. Is it confusing to offer a context menu for those, when no queries (apart from Copy Selection) can do anything.

Map processing

Map heap objects in the heap dump can be now accessed using array notation, returning Map.Entry items. Previously array access returned any Map.Entry heap objects in the heap dump for the map, which could then be used to find the key and value via the `key` and `value` fields. Not all maps have entry objects, so the new system means that `getKey()` and `getValue()` can be used to access the keys and values.

Question: when array access returns map entry objects, should those objects have a fake object ID of the actual map heap object, or of the Map.Entry heap object if one was available? This affects all collection extraction of maps, not just OQL. See [1] 

SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0)
map                              |kv
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
java.util.HashMap [id=0x22f49970]|[{key=java.lang.String [id=0x22f44658], value=java.lang.String [id=0x22f44670]}, {key=java.lang.String [id=0x22f44688], value=java.lang.String [id=0x22f446a0]}]
java.util.HashMap [id=0x22f49948]|[{key=java.lang.String [id=0x22f44628], value=java.lang.String [id=0x22f44640]}]
java.util.HashMap [id=0x22f49920]|[{key=java.lang.String [id=0x22f445c8], value=java.lang.String [id=0x22f445e0]}, {key=java.lang.String [id=0x22f445f8], value=java.lang.String [id=0x22f44610]}]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Extracts the map and a list of key / values pairs.

Question: Should MAT use the select clause or the AS column name as the key for the RowMap and subsequent access, or should it just use a simple name if given as part of the select, and a generated term such as `Expr1000` or `EXPR$0` (Apache Calcite) if more complex? Is an autogenerated term required in the case of duplicated select items or column names?

Flattening

The second column is a list of key/value pairs. It would be nice to process these further, and auto-flattening is one way to achieve this. If a sub-select returns a RowMap containing values which are lists or arrays, then auto-flattening splits that RowMap into multiple RowMaps, one for each entry of the list or array. Other objects are just repeated in the RowMap. If there are multiple lists or arrays of different lengths then items beyond the end of the list or array are replaced by null.

Question: This is very experimental and is intended to be a basic alternative to SQL JOIN. Is auto-flattening of sub-selects the way to go, or should it operated on FROM method calls as well or an alternative. Should there be a `flatten((SELECT ...))` function, or a `flatten()` method on the result table?

Question: A SELECT as a SELECT item can with OQL return multiple rows or columns. SQL expects at most one row and one column. Should a SELECT with one column be dequalified? E.g. 
SELECT z, z.st.v FROM OBJECTS ( SELECT (SELECT s.value AS v FROM java.lang.String s ) AS st, t FROM java.lang.Long t  ) z
the z.st.v needs several levels of qualification. As there is just one column, should this be just 'z.st'?

The aim is to achieve a result such as 

Map                              |Key                             |Value
----------------------------------------------------------------------------------------------------------------------------------------------------------------
java.util.HashMap [id=0xe35e96a8]|java.lang.String [id=0xe35ef478]|com.sun.management.internal.PlatformMBeanProviderImpl$4 [id=0xe35cbf68]
java.util.HashMap [id=0xe35e96a8]|java.lang.String [id=0xe35e96d8]|jdk.management.jfr.internal.FlightRecorderMXBeanProvider$SingleMBeanComponent [id=0xe35e98b0]
java.util.HashMap [id=0xe35ce190]|java.lang.String [id=0xe35c1950]|java.lang.Object [id=0xe0c145f0]
java.util.HashMap [id=0xe35ce190]|java.lang.String [id=0xe35c1900]|java.lang.Object [id=0xe0c145f0]
java.util.HashMap [id=0xe35cbde0]|java.lang.String [id=0xe35cbc70]|java.lang.Object [id=0xe0c145f0]
----------------------------------------------------------------------------------------------------------------------------------------------------------------



  1. SELECT z.map as Map, z.kv.key as Key, z.kv.value as Value FROM OBJECTS ( SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0) ) z
  2. SELECT z.map AS Map, z.kv.key AS Key, z.kv.value AS Value FROM OBJECTS (eval(( SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0))) ) z
  3. SELECT z.map AS Map, z.kv.key AS Key, z.kv.value AS Value FROM OBJECTS (flatten((SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0))) ) z
  4. SELECT z.map AS Map, z.kv.key AS Key, z.kv.value AS Value FROM OBJECTS (eval(( SELECT h AS map, (SELECT e.getKey() AS key, e.getValue() AS value FROM OBJECTS ${h}[0:-1] e ) AS kv FROM java.util.HashMap h WHERE (h[0:-1].size() > 0)).flatten()) ) z

Question: How should flattening handle lists or arrays with no items? Should the row be omitted, or should a row with no entries for the array be generated as a null for the item, or should the empty array be left unchanged? It is hard to add back a row later, but it is possible to filter rows with null or an empty list. It is hard to tell an empty list/array from one which contains one entry which is the same as the default value OQL chose (null or an empty list/array).


Here is another example of flattening - a query to see if any child of a parent does not have a back reference to the parent. 

SELECT group AS Group, thread AS Thread 
FROM OBJECTS ( SELECT t AS group, t.threads[0:-1] AS thread FROM java.lang.ThreadGroup t  ) 
WHERE ((thread != null) and (thread.group != group))

This selects all the java.lang.ThreadGroup objects and then generates rows with two columns, the group and a list of the child threads. This is then flattened to a rows of the group and a single child thread, where the select then checks for a non-null child Thread and a child which does not point back to the ThreadGroup. Note here the omission of the alias name before the 'WHERE' as it is not necessary - 'group' and 'thread' in the outer select do not need to be qualified with an alias name.

GROUP BY

SQL GROUP BY can be simulated in the following fashion: 

SELECT s.sz AS Size, 
(SELECT OBJECTS m FROM java.util.HashMap m WHERE (m[0:-1].size() = s.sz)) AS Maps 
FROM OBJECTS ( SELECT DISTINCT h[0:-1].size() AS sz FROM java.util.HashMap h  ) s
  1. This first obtains a list of things to group by, which here is a list of sizes.
  2. The sizes are then returned to the next phase as a sub-select.
  3. Then the select items clause only chooses objects which match the current GROUP BY value.
  4. Then the results are converted to an object list, which appears in the columns as an int[] array, which can then be used as a context menu.

Another example grouping by number of inbounds: 

SELECT s.sz AS Size, 
(SELECT OBJECTS m FROM INSTANCEOF java.lang.Object m WHERE (inbounds(m).@length = s.sz)) AS Objects 
FROM OBJECTS ( SELECT DISTINCT inbounds(h).@length AS sz FROM INSTANCEOF java.lang.Object h  ) s

COUNT

SQL COUNT can be simulated in the following fashion using the @length attribute on arrays, or on a list or by converting an array to a list and then using size(). 

SELECT z.size AS Size, 
z.maps AS Maps, 
z.maps.@length AS "Count", 
z.maps[0:-1].size() AS "Count (another way)" 
FROM OBJECTS ( eval((
SELECT 
s.sz AS size, 
(SELECT OBJECTS m FROM java.util.HashMap m WHERE (m[0:-1].size() = s.sz)) AS maps 
FROM OBJECTS ( SELECT DISTINCT h[0:-1].size() AS sz FROM java.util.HashMap h  ) s
)) ) z
  1. This first obtains a list of things to group by, which here is a list of sizes.
  2. The sizes are then returned to the next phase as a sub-select.
  3. Then the select items clause only chooses objects which match the current GROUP BY value.
  4. Then the results are converted to an object list, which appears in the columns as an int[] array
  5. The select is then wrapped by an eval() so that the outer select does not flatten it
  6. The outer select then generates the result, with the size, the maps, and two ways of counting the elements in the map array, once using @length and one using size().

Another example: 

SELECT z.size AS Size, 
z.objects AS Objects,
z.objects.@length AS "Count", 
z.objects[0:-1].size() AS "Count (another way)" 
FROM OBJECTS ( eval((
SELECT s.sz AS size, 
(SELECT OBJECTS m FROM INSTANCEOF java.lang.Object m WHERE (inbounds(m).@length = s.sz)) AS objects 
FROM OBJECTS ( SELECT DISTINCT inbounds(h).@length AS sz FROM INSTANCEOF java.lang.Object h  ) s
)) ) z

JOIN operations

OQL does not have SQL-style JOIN operations apart from UNION. With flattening it is possible to simulate some of these operations, but the statements required are more complex.

Consider a JOIN on `java.lang.Integer` and `java.lang.Long` based on the value fields of both.

CROSS JOIN

This operation generates every combination of the left and right tables (sets of objects), so can generate a huge sized result table. 

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value" 
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l ) AS lv FROM java.lang.Integer i  ) z

This selects all the `java.lang.Integer` heap objects, then for each Integer heap object then generates a row with the object and a list of all the java.lang.Long objects, then flattens the rows. Each flattened row has one Integer heap object (accessed via 'z.i' or 'i') and one Long heap object (accessed via 'z.lv.l' or 'lv.l'). The objects in one row do not necessarily match in value.

Compare to MAT Calcite (SQL): 

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i CROSS JOIN "java.lang.Long" l

LEFT JOIN / LEFT OUTER JOIN

This operation generates every row from the left table (set of objects) and includes in that row any corresponding row from the right table. The result table is the same size as the left. 

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value"  
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l WHERE (l.value = i.value)) AS lv FROM java.lang.Integer i  ) z

This selects all the `java.lang.Integer` objects, then for each Integer then generates a row with the object and a list of all the java.lang.Long objects with the same value, then flattens the rows. Each flattened row has one Integer heap object (accessed via 'z.i' or 'i') and possibly one matching Long heap object or null (accessed via 'z.lv.l' or 'lv.l').

Compare to MAT Calcite (SQL): 

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i LEFT JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

INNER JOIN

This operation generates every row from the left table (set of objects) which has a matching row in the right table (set of objects). The result table is no bigger than the smaller of the left and right tables. 

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value" 
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l WHERE (l.value = i.value)) AS lv FROM java.lang.Integer i  ) z 
WHERE (z.lv != null)

This selects all the `java.lang.Integer` objects, then for each Integer then generates a row with the object and a list of all the java.lang.Long objects with the same value, then flattens the rows and excludes any row without a java.lang.Long value. 

SELECT z.iv.i AS Integer, z.iv.i.value AS "Integer value", z.l AS Long, z.l.value as "Long value" 
FROM OBJECTS ( SELECT (SELECT i FROM java.lang.Integer i WHERE (i.value = l.value)) AS iv, l FROM java.lang.Long l  ) z 
WHERE (z.iv != null)

This selects all the `java.lang.Long` objects, then for each Long then generates a row with the object and a list of all the java.lang.Integer objects with the same value, then flattens the rows and excludes any row without a java.lang.Integer value. Each flattened row has one Integer heap object (accessed via 'z.i' or 'i') and one Long heap object (accessed via 'z.lv.l' or 'lv.l') which matches by the 'WHERE (i.value = l.value)' clause.

Compare to MAT Calcite (SQL): 

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i INNER JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

RIGHT JOIN / RIGHT OUTER JOIN

This operation generates every row from the right table (set of objects) and includes in that row any corresponding row from the left table which matches. The result table is the same size as the right. 

SELECT z.iv.i AS Integer, z.iv.i.value AS "Integer value", z.l AS Long, z.l.value as "Long value" 
FROM OBJECTS ( SELECT (SELECT i FROM java.lang.Integer i WHERE (i.value = l.value)) AS iv, l FROM java.lang.Long l  ) z

This selects all the `java.lang.Long` objects, then for each Long then generates a row with the object and a list of all the java.lang.Integer objects with the same value, then flattens the rows. Each flattened row has one Long heap object (accessed via 'z.l' or 'l') and possibly one matching Integer heap object or null (accessed via 'z.iv.i' or 'iv.i').

Compare to MAT Calcite (SQL): 

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i RIGHT JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

FULL OUTER JOIN

This operation generates has a row for every row from the left table (set of objects) and every row from the right table (set of objects), but when the left table has a row with the same value as a row from right table they will be included in the same output row. The result table is at least as big as the bigger of the left table and right table. 

SELECT z.i AS Integer, z.i.value AS "Integer value", z.lv.l AS Long, z.lv.l.value as "Long value" 
FROM OBJECTS ( SELECT i, (SELECT l FROM java.lang.Long l WHERE (l.value = i.value)) AS lv FROM java.lang.Integer i  ) z 
UNION (
SELECT z.iv.i AS Integer, z.iv.i.value AS "Integer value", z.l AS Long, z.l.value as "Long value" 
FROM OBJECTS ( SELECT (SELECT i FROM java.lang.Integer i WHERE (i.value = l.value)) AS iv, l FROM java.lang.Long l  ) z 
WHERE (z.iv = null)
)

This does a LEFT JOIN / LEFT OUTER JOIN then combines the rows with a list of all java.lang.Long objects which do not have a corresponding java.lang.Integer object. Each row contains either an Integer heap object or a Long heap object or both.

Compare to MAT Calcite (SQL): 

SELECT i.this,i.this['value'] AS "Integer value", l.this,l.this['value'] AS "Long value" 
FROM "java.lang.Integer" i FULL OUTER JOIN "java.lang.Long" l ON i.this['value']+0 = l.this['value']+0

Bug fixes

  • When there is a union queries with a select which returns no items, that select was then omitted from the command window.
SELECT s FROM java.lang.String s UNION (SELECT s FROM java.lang.Missing s)
this was then redisplayed as 
SELECT s FROM java.lang.String s
  • Context dependency fix. OQL processing optimizes some queries by detecting that some parts are not context dependent and will evaluate the same each time, so can just be evaluated once. This processing was not correct for sub-select. Now a sub-select in a select item will be correctly re-evaluated if required.
  • Progress monitoring has been improved so the progress monitor bar graph better shows how much more work needs to be done to complete a query. Also, cancelling a long running OQL query works more swiftly.

Writing Queries

Writing queries can be a bit of an art, with some trial and error required. For example, consider this problem:

"I would like to find all the unreachable objects of a particular type."

Normally, Memory Analyzer discards unreachable objects and the only thing visible are some totals in the unreachable objects histogram. This is not useful for OQL, so we need the -keep_unreachable_objects option. The unreachable objects will not then be discarded, but will be retained by some artificially inserted UNREACHABLE_OBJECT garbage collection roots.

Let us try some examples:

The snapshot has some interesting methods, including getGCRoots(). 

SELECT * FROM OBJECTS ${snapshot}.getGCRoots() r

As it is a no-argument method starting with get we can access it using a bean attribute. 

SELECT * FROM OBJECTS ${snapshot}.@GCRoots r
[21800, 21801, 21802, 21803, 21804, 21805,

This returns an array of integers - the MAT object IDs.

Treat them as values: 

SELECT r FROM OBJECTS ${snapshot}.@GCRoots r
      r
--------
 21,800
 21,801
 21,802
--------

Treat them as objects: 

SELECT OBJECTS r FROM OBJECTS ${snapshot}.@GCRoots r
Class Name                                                        | Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------
class java.lang.IllegalArgumentException @ 0x2b75f188 System Class|          104 |           104
class java.lang.NumberFormatException @ 0x2b75f1e8 System Class   |          104 |           104
class java.text.CharacterIterator @ 0x2b760648 System Class       |          104 |           104
-------------------------------------------------------------------------------------------------

This gives all the GC roots, but we only want the unreachable roots.

The snapshot getGCRootInfo(int id) method might help. Let use try it out using an object ID above that is a GC root. 

SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(21800) t
[org.eclipse.mat.parser.model.XGCRootInfo@2b963bbc]

so it returns an array of XGCRootInfo.

Let OQL look at each element: 

SELECT t FROM OBJECTS ${snapshot}.getGCRootInfo(21800) t
r
-------------------------------------------------
org.eclipse.mat.parser.model.XGCRootInfo@2b963bbc
-------------------------------------------------

Using the MAT API reference we can see that this type has a getType() method, so access it using as a bean. 

SELECT t,t.@type FROM OBJECTS ${snapshot}.getGCRootInfo(21800) t
t                                                | t.@type
-----------------------------------------------------------
org.eclipse.mat.parser.model.XGCRootInfo@2b963bbc|       2
-----------------------------------------------------------

Using the API reference we can see that GCRootInfo.Type.UNREACHABLE has a value of <2048> so we can select just the GCRootInfo objects with that value as: 

SELECT t,t.@type FROM OBJECTS ${snapshot}.getGCRootInfo(21800) t WHERE t.@type = 2048

This does not return anything as object ID 21800 is a SYSTEM root. However, we can use it to select from all the GC roots, relying on this sub SELECT clause being null if it does not find an UNREACHABLE root. We can simplify the select item to * as it is not important. 

SELECT OBJECTS r FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null
Class Name                                          | Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------
int[11] @ 0x22ef4c58 Unreachable                    |           56 |            56
int[9] @ 0x22ed2ad8 Unreachable                     |           48 |            48
int[7] @ 0x22ed2b28 Unreachable                     |           40 |            40
int[17] @ 0x22ed3150 Unreachable                    |           80 |            80
java.lang.ref.SoftReference @ 0x22f52cd8 Unreachable|           32 |           400
-----------------------------------------------------------------------------------

This is looking promising - we have a list of objects, all of which are UNREACHABLE GC roots. We now need the retained set to find everything that is normally discarded, but is now only retained via these artificial roots. 

SELECT AS RETAINED SET r FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null
r
--------------------------------------------------------------------

java.lang.Class [id=0x26970198;name=java.lang.Throwable[]]
java.lang.Class [id=0x26970370;name=java.lang.Error[]]
java.lang.Class [id=0x26970558;name=java.lang.VirtualMachineError[]]
--------------------------------------------------------------------

Some of the output is above. Alternatively to see them as a tree, not a table. 

SELECT AS RETAINED SET * FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null
Class Name                                                        | Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------
int[33] @ 0x22e5be20 Unreachable                                  |          144 |           144
int[27] @ 0x22e5bed0 Unreachable                                  |          120 |           120
int[17] @ 0x22e5bf60 Unreachable                                  |           80 |            80
sun.reflect.NativeConstructorAccessorImpl @ 0x22e5c5d8 Unreachable|           24 |           184
sun.reflect.NativeConstructorAccessorImpl @ 0x22e5c5f0 Unreachable|           24 |           184
-------------------------------------------------------------------------------------------------

If we are quickly interested in the types then the 'Show as Histogram' button in the top right of the toolbar will show the totals by class. If we are interested in the subclasses then the 'Group by superclass' option does that.

However, we might want to do it entirely by OQL - for example as part of another query or in batch mode. We now need to choose just the objects of the type we are interested in. There are several ways.

  1. We could list all the objects of the type we are possibly interested in, and keep those in the retained set.
  2. We could look at each of the retained objects and see if each one is in the list of objects we are interested in.
  3. We could look at each of the retained objects, find its class, then its class name and see if the class name matches the one we are interested in.
SELECT * FROM java.util.ArrayList o
SELECT * FROM java.util.ArrayList o WHERE o in
(SELECT AS RETAINED SET * FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null)
Class Name                      | Shallow Heap | Retained Heap
---------------------------------------------------------------
java.util.ArrayList @ 0x22efe620|           24 |           320
java.util.ArrayList @ 0x22efe608|           24 |           320
java.util.ArrayList @ 0x22efe5a8|           24 |           320
java.util.ArrayList @ 0x22efcfd8|           24 |           384
java.util.ArrayList @ 0x22efb998|           24 |           320
java.util.ArrayList @ 0x22efb928|           24 |           320
---------------------------------------------------------------

or 

SELECT * FROM OBJECTS (SELECT AS RETAINED SET * FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null) u
 WHERE u in (SELECT * FROM java.util.ArrayList o)
Class Name                      | Shallow Heap | Retained Heap
---------------------------------------------------------------
java.util.ArrayList @ 0x22efb928|           24 |           320
java.util.ArrayList @ 0x22efb998|           24 |           320
java.util.ArrayList @ 0x22efcfd8|           24 |           384
java.util.ArrayList @ 0x22efe5a8|           24 |           320
java.util.ArrayList @ 0x22efe608|           24 |           320
java.util.ArrayList @ 0x22efe620|           24 |           320
---------------------------------------------------------------

or 

SELECT * FROM OBJECTS (SELECT AS RETAINED SET * FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null) u
 WHERE u.@clazz.@name = "java.util.ArrayList"
Class Name                      | Shallow Heap | Retained Heap
---------------------------------------------------------------
java.util.ArrayList @ 0x22efb928|           24 |           320
java.util.ArrayList @ 0x22efb998|           24 |           320
java.util.ArrayList @ 0x22efcfd8|           24 |           384
java.util.ArrayList @ 0x22efe5a8|           24 |           320
java.util.ArrayList @ 0x22efe608|           24 |           320
java.util.ArrayList @ 0x22efe620|           24 |           320
---------------------------------------------------------------

If we wanted to know if about objects which extended java.util.AbstractCollection then use: 

SELECT * FROM INSTANCEOF java.util.AbstractCollection o WHERE o in
(SELECT AS RETAINED SET * FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null)
Class Name                      | Shallow Heap | Retained Heap
---------------------------------------------------------------
java.util.ArrayList @ 0x22efb928|           24 |           320
java.util.ArrayList @ 0x22efb998|           24 |           320
java.util.ArrayList @ 0x22efcfd8|           24 |           384
java.util.ArrayList @ 0x22efe5a8|           24 |           320
java.util.ArrayList @ 0x22efe608|           24 |           320
java.util.ArrayList @ 0x22efe620|           24 |           320
java.util.HashSet @ 0x22f2b600  |           16 |           136
java.util.Vector @ 0x22f2b5e8   |           24 |            80
java.util.Vector @ 0x22f2b638   |           24 |            80
---------------------------------------------------------------

or 

SELECT * FROM OBJECTS (SELECT AS RETAINED SET * FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null) u
 WHERE u in (SELECT * FROM INSTANCEOF java.util.AbstractCollection o)
Class Name                      | Shallow Heap | Retained Heap
---------------------------------------------------------------
java.util.ArrayList @ 0x22efb928|           24 |           320
java.util.ArrayList @ 0x22efb998|           24 |           320
java.util.ArrayList @ 0x22efcfd8|           24 |           384
java.util.ArrayList @ 0x22efe5a8|           24 |           320
java.util.ArrayList @ 0x22efe608|           24 |           320
java.util.ArrayList @ 0x22efe620|           24 |           320
java.util.Vector @ 0x22f2b5e8   |           24 |            80
java.util.HashSet @ 0x22f2b600  |           16 |           136
java.util.Vector @ 0x22f2b638   |           24 |            80
---------------------------------------------------------------

or 

SELECT * FROM OBJECTS (SELECT AS RETAINED SET * FROM OBJECTS ${snapshot}.getGCRoots() r
 WHERE (SELECT * FROM OBJECTS ${snapshot}.getGCRootInfo(r) t WHERE t.@type = 2048) != null) u
 WHERE u.@clazz.doesExtend("java.util.AbstractCollection")
Class Name                      | Shallow Heap | Retained Heap
---------------------------------------------------------------
java.util.ArrayList @ 0x22efb928|           24 |           320
java.util.ArrayList @ 0x22efb998|           24 |           320
java.util.ArrayList @ 0x22efcfd8|           24 |           384
java.util.ArrayList @ 0x22efe5a8|           24 |           320
java.util.ArrayList @ 0x22efe608|           24 |           320
java.util.ArrayList @ 0x22efe620|           24 |           320
java.util.Vector @ 0x22f2b5e8   |           24 |            80
java.util.HashSet @ 0x22f2b600  |           16 |           136
java.util.Vector @ 0x22f2b638   |           24 |            80
---------------------------------------------------------------

Extracting Thread information

With a recent level of MAT this should work: 

SELECT u.Thread AS Thread, u.Frame.@text AS Frame 
  FROM OBJECTS ( 
    SELECT t AS Thread, ${snapshot}.getThreadStack(t.@objectId).@stackFrames AS Frame 
      FROM java.lang.Thread t  ) u

The inner select 

SELECT t AS Thread, ${snapshot}.getThreadStack(t.@objectId).@stackFrames AS Frame 
  FROM java.lang.Thread t

extracts each thread and an array of stack frames. The outer select then flattens that array with the same thread reference for each of its stack frames. 

Thread                           |Frame
--------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread [id=0x7b6a1e7f0]|at java.lang.Object.wait(J)V (Native Method)
java.lang.Thread [id=0x7b6a1e7f0]|at java.lang.Object.wait(JI)V (Unknown Source)
java.lang.Thread [id=0x7b6a1e7f0]|at com.squareup.okhttp.ConnectionPool.performCleanup()Z (ConnectionPool.java:305)
java.lang.Thread [id=0x7b6a1e7f0]|at com.squareup.okhttp.ConnectionPool.runCleanupUntilPoolIsEmpty()V (ConnectionPool.java:242)
java.lang.Thread [id=0x7b6a1e7f0]|at com.squareup.okhttp.ConnectionPool.access$000(Lcom/squareup/okhttp/ConnectionPool;)V (ConnectionPool.java:54)
java.lang.Thread [id=0x7b6a1e7f0]|at com.squareup.okhttp.ConnectionPool$1.run()V (ConnectionPool.java:97)
--------------------------------------------------------------------------------------------------------------------------------------------------


You can even extract each local from each frame using another select. 

    SELECT v.Thread as Thread, toString(v.Thread) AS Name, v.Frame AS Frame, ${snapshot}.getObject(v.Objs) AS Local 
      FROM OBJECTS ( 
        SELECT u.Thread AS Thread, u.Frame.@text AS Frame, u.Frame.@localObjectsIds AS Objs 
          FROM OBJECTS ( 
            SELECT t AS Thread, ${snapshot}.getThreadStack(t.@objectId).@stackFrames AS Frame 
              FROM java.lang.Thread t
          ) u
      ) v 
      WHERE (v.Objs != null)
v.Thread                         |Name                 |Frame                                                                                                           |Local
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread [id=0x7b6a1e7f0]|OkHttp ConnectionPool|at com.squareup.okhttp.ConnectionPool.performCleanup()Z (ConnectionPool.java:305)                               |java.util.ArrayList [id=0x7bb402eb8]
java.lang.Thread [id=0x7b6a1e7f0]|OkHttp ConnectionPool|at com.squareup.okhttp.ConnectionPool.performCleanup()Z (ConnectionPool.java:305)                               |com.squareup.okhttp.ConnectionPool [id=0x6c556ccb0]
java.lang.Thread [id=0x7b6a1e7f0]|OkHttp ConnectionPool|at com.squareup.okhttp.ConnectionPool.runCleanupUntilPoolIsEmpty()V (ConnectionPool.java:242)                   |com.squareup.okhttp.ConnectionPool [id=0x6c556ccb0]
java.lang.Thread [id=0x7b6a1e7f0]|OkHttp ConnectionPool|at com.squareup.okhttp.ConnectionPool.access$000(Lcom/squareup/okhttp/ConnectionPool;)V (ConnectionPool.java:54)|com.squareup.okhttp.ConnectionPool [id=0x6c556ccb0]
java.lang.Thread [id=0x7b6a1e7f0]|OkHttp ConnectionPool|at com.squareup.okhttp.ConnectionPool$1.run()V (ConnectionPool.java:97)                                         |com.squareup.okhttp.ConnectionPool$1 [id=0x6c556ccd8]
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Displaying all the fields of objects

SELECT t.s AS "Object", toHex(t.s.@objectAddress) AS "Object address", t.f.@name AS "Field name", t.f.@value AS "Field value" 
    FROM OBJECTS ( SELECT s, s.@fields AS f FROM "java.util.*" s 
        WHERE (s implements org.eclipse.mat.snapshot.model.IInstance) ) t

This extracts all the plain Java objects (not arrays) of the java.util package. It then extracts the fields, and uses flattening to process those fields one by one. 

Object                          |Object address|Field name     |Field value
--------------------------------|--------------|---------------|-----------
java.util.TreeMap [id=0x3acd178]|0x3acd178     |comparator     |
java.util.TreeMap [id=0x3acd178]|0x3acd178     |root           |0x3c32f60
java.util.TreeMap [id=0x3acd178]|0x3acd178     |size           |2
java.util.TreeMap [id=0x3acd178]|0x3acd178     |modCount       |2
java.util.TreeMap [id=0x3acd178]|0x3acd178     |entrySet       |
java.util.TreeMap [id=0x3acd178]|0x3acd178     |navigableKeySet|
java.util.TreeMap [id=0x3acd178]|0x3acd178     |descendingMap  |
java.util.TreeMap [id=0x3acd178]|0x3acd178     |keySet         |
java.util.TreeMap [id=0x3acd178]|0x3acd178     |values         |
java.util.TreeMap [id=0x3acd1a8]|0x3acd1a8     |comparator     |
java.util.TreeMap [id=0x3acd1a8]|0x3acd1a8     |root           |0x3c32f80
--------------------------------|--------------|---------------|-----------
Retrieved from "https://wiki.eclipse.org/index.php?title=MemoryAnalyzer/OQL&oldid=444754"

This page was last modified 09:39, 1 February 2022 by Eclipsepedia anonymous user Unnamed Poltroon.

Back to the top