COSMOSQueryInterfaceInitialDesign

Design Requirements

Remoteable The Query service must be accessible to remote consumers on both java and non-java platforms.

Introspectable The Query service's capabilities must be discoverable at runtime. Mechanisms must be provided to introspect the service in local and remote environments. The Query service must allow consumers to determine the types of queries that may be specified for the service implementation, the return formats of the service supports, and the valid combinations of query and return formats. Additionally, the Query service must provide a mechanism for determining the scope of the data being queried (for example, the set of machines for which a particular query service exposes log data).

Decoupled from Implementation The Query service API must not mandate a particular underyling data store type or query language. The API must be flexible enough to allow adoptors to easily integrate existing data stores, but must be suitable to encourage de-facto standards and/or standard implementations to emerge.

Optimizable The Query service API must not preclude optimizations for special cases. For example, an application that understands a particular database schema may wish to use SQL queries if possible.

High Level Structure

The Query service is implemented as a QueryComponent type, which acts as a root node in a DataCollection Assembly. QueryComponent implementations implement a QueryService interface, which is converted into a WSDM capability on the QueryComponent at runtime. The QueryComponent also supports the MetadataExchange capability, allowing a QueryComponent instance to provide both a WSDL and SML description of itself.

The QueryService interface by itself is very generic. It is envisioned that additional 'de-facto' standard interfaces (eg a LogDataService) will emerge and either layer on top of sit adjacent to the QueryService interface.

QueryComponents may interact with Transforms and Filters to modify the collections to be returned by specific queries. In order to control the path through a series of transformations and/or filters, all sub-trees wired into a Query Assembly must terminate with a special type of component (a QueryResponse component) that allows the framework to determine the types of query dialect/response pairs are supported by component.

Query Capability The Query capability is implemented using the following interface:

@ManagedResourceCapability(namespace="http://cosmos.eclipse.org/capabilities/query")
public interface IDataQueryService extends IWireSource, IWireTerminal {


	@ManagedOperation(namespace=NAMESPACE_URI)
	String[] getSupportedDialects();
	
	@ManagedOperation(namespace=NAMESPACE_URI)
	String[] getSupportedResponses();
	
	@ManagedOperation(namespace=NAMESPACE_URI)
	boolean supportedQuery(String dialect, String response);	
	
	@ManagedOperation(namespace=NAMESPACE_URI)
	IDataQueryResult query(String dialect, String response, String queryString) throws Exception;
	
	@ManagedOperation(namespace=NAMESPACE_URI)
	IDataQueryResult pageQuery(String dialect, String response, String queryString, int max, int start) throws Exception;

}

This interface is annotated to expose itself as a WSDM manageability capability with a URI of http://cosmos.eclipse.org/capabilities/query. The getSupportedDialects and getSupportedResponses methods provide an initial introspection capability, where the supportedQuery method allows the consumer to determine support for a particular combination of dialect and response types.

The query method supports a generic query with no restrictions on the return value.

The pageQuery method supports page-oriented query semantics, useful for large data sets.

The IDataQueryResult is a simple wrapper object that may contain either a single object or a single collection.

Query Component Types

The QueryComponent type may only exist as the direct child of an outbound DataCollection Assembly. The QueryComponent may support a default (preferred) response type, but may also be wired to Transformers or Filters to convert or remove results for the query response collection. For any component chains wired to a Query, those chains MUST be terminated with a QueryResponse Component.

<?xml version="1.0" encoding="UTF-8"?>
<!--
 *  Test configuration for outbound pipeline
 -->
<context xmlns="http://www.eclipse.org/xmlns/cosmos/1.0"
	xmlns:cosmos="http://www.eclipse.org/xmlns/cosmos/1.0"
	xmlns:test="http://cosmos.eclipse.org/tests/runtime/1.0"
	cosmos:name="TestQuery" cosmos:direction="out">
	<query cosmos:factory="TestQueryFactory" cosmos:optimizable="true">
		<test:binding/>
		<transform cosmos:factory="TestTransformFactory" cosmos:optimizable="false">
			<test:binding/>
			<response cosmos:factory="TestResponseFactory" cosmos:optimizable="true">
				<test:binding/>
			</response>
		</transform>
	</query>    
</context>

Query Response Type

The QueryResponse component type acts as a collection point for objects that are dispatched through an outbound assembly wire. For inbound assemblies, all objects dispatched through a DataCollection wire are either ignored or persisted - there is no requirement to return objects from the wire. For outbound assemblies, the whole reason for being is to return objects from the outbound DataCollection wire - hence the requirement for a Query Response component. For cases where the desired return type requires a wire to be active, the ResponseObject performs the role of populating the collection object in a IDataQueryResult instance provided by the root QueryComponent.

Data Visualization Design Considerations

As outlined in the Data Visualization design document[1] the data reporting component requires a set of APIs to visualize the data stored in the data collection repository.

The following rudimentry APIs were created to visualize logging and statistical data for the Eclipse Con 07 demo.

• getDetailEvents - provides a list of objects containing CBE properties
• getEventCountBySeverity - provides a count of the number of severe, statistical and informational events for a particular log file. Groups events based on severe value and source component id.
• getObservations - provides a list of objects containing statistical associated by time

Given the above APIs it was possible to construct the following reports:

• Top 10 sub component - this report provides the user with the ability to pin point sub components such as applications that have raised a large proportion of severe events. As a result users are able to isolate problems.
• Statitical reports - this reports graphs statistical data versus time. Trends in the data can be visually shown to analyze the data.

The Eclipse Con 07 demo utilized a data layer provided by TPTP that understood the data format such as CBEs and the Statistical schema. The data layer provided the following:

• a connection api that consumes connection information such as hostname, port number, jdbc driver, username and password to the database repository
• an output format api that allows the client code to construct meta data to tell the data layer how it should format the results of the query
• a query API that consumes a connection object and an output format object. This api returns a list of objects representing the result.

In order to meet the deadlines, a tactical decision was made to perform most of the filtering, grouping and sorting made in the BIRT report engine. Inevitably, this slowed down the rendering of the report. These operations will be executed at a lower level (in the data collection layer) to make use of the datasource optimization (especially when using databases).

In order to support the Top 20 sub component report and statitical reports in the June release the Report Component requires the following:

• an API to connect to a datasource
• a set of Query APIs that has the ability to filter, group and sort a set of data and returns a result set containing a list of data.
• The result set should support paging so that the user interface only works off a subset of data.

Specifically some of the query APIs should be:

• getEventCountGroupedBy(String groupCriteria, String sortCriteria, String filterCriteria)
• getDetailsEvents(String sortCriteria, String filterCriteria)
• getObservationCountGroupedBy(String groupCriteria, String sortCriteria, String filterCriteria)
• getObservations(String sortCriteria, String filterCriteria)

These apis should be a set of services provided by the data collection layer to maximize their reuse.

Demo Code

The following snippet of code shows the actual code used to constuct the getDetailsEvents API.

	public List getDetailEvents(IDataSource dataSource,
			IDataOutputFormat dataOutputFormat) {

		//1. create the datasource based on jdbc jar location, host , port , user and password
		if (dataSource == null)
			dataSource = createDataSource(JDBCDriverLocation,HOST,PORT,USER,PASSWORD); // default data source

		//2. form an output format that descibes the list of properties that should be extracted from the repository
		if (dataOutputFormat == null) {
			dataOutputFormat = (ILogRecord) DataAccessFactory
					.getLogDataService().createDataOutputFormat(
							ILogRecord.class);
			IDataProperty dataProperty = DataAccessFactory.getLogDataService()
					.createDataProperty("@msg", "string", null);
			dataOutputFormat.getDynamicProperties().add(dataProperty);
			dataProperty = DataAccessFactory.getLogDataService()
					.createDataProperty("@creationTime", "string", null);
			dataOutputFormat.getDynamicProperties().add(dataProperty);
			dataProperty = DataAccessFactory.getLogDataService()
					.createDataProperty("@extensionName", "string", null);
			dataOutputFormat.getDynamicProperties().add(dataProperty);
			dataProperty = DataAccessFactory.getLogDataService()
					.createDataProperty("@severity", "string", null);
			dataOutputFormat.getDynamicProperties().add(dataProperty);
			dataProperty = DataAccessFactory.getLogDataService()
					.createDataProperty("@location", "string", null);
			dataOutputFormat.getDynamicProperties().add(dataProperty);
			dataProperty = DataAccessFactory.getLogDataService()
					.createDataProperty("@subComponent", "string", null);
			dataOutputFormat.getDynamicProperties().add(dataProperty);

			if (dataOutputFormat.getDataContext() == null) {
				dataOutputFormat
						.setDataContext(DataAccessFactory.getLogDataService()
								.createDataContext(LOG_ID));
			}
		}

		//3. call the data access layer 
		return (List) (DataAccessFactory.getDataAccessController()
				.executeCommand(
						null,
						DataAccessFactory.getLogDataService().getLogRecords(
								dataSource, dataOutputFormat, null, null),
						true, null).getResult());
	}

This API returns a list of IDataProperty objects:

public class DataProperty implements IDataProperty{
	String name;
	String type;
	String value;
	
	public DataProperty(String name, String type, String value) {
		this.name = name;
		this.type = type;
		this.value = value;
	}

	public String getName() {
		return name;
	}

	public String setName(String name) {
		String prev = this.name;
		this.name = name;
		return prev;
	}

	public String getType() {
		return type;
	}

	public String setType(String type) {
		String prev = this.type;
		this.type = type;
		return prev;	

	}

	public String getValue() {
		return value;
	}

	public String setValue(String value) {
		String prev = this.value;
		this.value = value;
		return prev;			
	}
}

Demo Code - Demonstrating Grouping Logic

The following snippet of code shows how grouping was done. This grouping was necessary to construct the stack bar chart. Notice that code was necessary to group the list of objects returned from the getDetailEvents API.

	public List getSeverityList(IDataSource dataSource,
			IDataOutputFormat dataOutputFormat) {
		List res=new ArrayList();
		
		//reuse getDetailEvents API
		List logRecords = getDetailEvents(dataSource, dataOutputFormat);
		
		//////////////////////////////////////////////////////
		//The following is logic to group the events based on 
		//a grouping criteria (i.e. subComponent and Severity
		//////////////////////////////////////////////////////
		Map orderByComponentId = new HashMap();
		for (int i = 0; i < logRecords.size(); i++) {
			ILogRecord logRecord = (ILogRecord)logRecords.get(i);
			List properties = logRecord.getDynamicProperties();

			OrderCBEDataStructure dtStructure = null;
			Object id = null;
			for (int x = 0; x < properties.size(); x++){
				IDataProperty property = (IDataProperty)properties.get(x);
				
				
				
				if (property.getName().equals("@subComponent")){
					id = property.getValue();
					OrderCBEDataStructure dtStructureTemp = (OrderCBEDataStructure)orderByComponentId.get(id);
					if (dtStructureTemp == null){							
						dtStructureTemp = new OrderCBEDataStructure();
						orderByComponentId.put(id, dtStructureTemp);
					}
					if (dtStructure != null){
						dtStructureTemp.severe += dtStructure.severe;
						dtStructureTemp.warning += dtStructure.warning;
						dtStructureTemp.info += dtStructure.info;
						dtStructure = dtStructureTemp;
					}
				}
				else if (property.getName().equals("@severity")){
					//default value of severity is 10
					short severity = 10;
					if (property.getValue() != null)
						severity = Short.parseShort(property.getValue());
			
					if (dtStructure == null)
						 dtStructure = new OrderCBEDataStructure();
					if (severity >= 50) {
						dtStructure.severe++;
	                 	} else if ((severity >= 30) && (severity <= 49)) {
	                 		dtStructure.warning++;
	                 	} else {
	                 		dtStructure.info++;
	                 	}
				}
			}
		         
		}
		Iterator iter = orderByComponentId.keySet().iterator();
		while (iter.hasNext()){
			Map properties = new HashMap();
			Object subComponent = iter.next();
			OrderCBEDataStructure dtStructure = (OrderCBEDataStructure)orderByComponentId.get(subComponent);
			properties.put("subComponent", "" + subComponent);
			properties.put("severe", "" + dtStructure.severe);
			properties.put("warning", "" + dtStructure.warning);
			properties.put("info","" + dtStructure.info);
			res.add(properties);
		}
		return res;
	}