Data Collection and Normalization

Scope and Mission

The Data Collection and Normalization component will implement a data collection framework as described on pages 20 and 21 of the COSMOS project creation review pdf here.

The initial scope of the Data Collection and Normalization component supports the Use Cases that the COSMOS project is defining to demonstrate the value of SML throughout the application lifecycle. It will accept managent data from the instrumentation produced by the COSMOS Build to manage component and provide data normalization and persistance and a query API to support its consumption by the COSMOS Data Reporting component.

Tactical Plan

Much of the initial work of the data collection component will be done in cooperation with the TPTP project. To facilitate the timely creation of a demonstrable implementation, the data collection component will leverage key TPTP technologies such as the Common Base Event (CBE) format, TPTP agents such as JMX and the Generic Log Adapter (GLA), and the Agent Controller.

See also the prototype proposal in the Resources section below.

Architectural Vision

Persistence API

The framework will provide an extension point for data persistance. Each supported data type will be consumed by a persistor that supports persisting a particular data format into its own database table or other data store.

Initially, persistors will be provided for the EMF models that TPTP currently supports.

Data Collection Control API

A WSDM interface will be provided for the purpose of configuring the data collection agents.

In the longer term, this interface will be usable to manage the monitored infrastructure components.

Adapters for Data Collection agents into Persistence API

The framework will provide an extension point for data collection agent adapters. One possible approach is to provide a service that connects agent adaptors to an appropriate data persistor based on the data type supported by the adaptor such as CBE or WEF. A provision to specify these connections declaratively is required, but they could also be constructed dynamically via a WSDM interface.

The connection service should specifically support the ability to inject interceptors between the agent adapters and the data persistors.

Initial adapters will include log, statistical, and perhaps trace adapters but the framework will be generalized so it can be extended to support any additional models as required.

Query API

The query API will provide a web service interface to the data store(s).

Its binding will be constructed in a manner analogous to the Data Collection adapters where extensions can be created to implement any desired query mechanism without requiring dependence on the type or location of the underlying data store.

Multiple web services will be provided to allow the consumer to select appropriate query semantics.

The initial targets are SQL and / or XPath.

COSMOSTPTPQuereryIntrfaceNotes

COSMOSQueryInterfaceInitialDesign

Database Schema

Each data collection adapter will define the schema for its datastore. A reasonable minimum expectation is that each database will be keyed by timestamp and the unique ID of the managed resource.

The initial target is Derby, with support for MySQL and proprietary databases later. An attempt will be made to avoid proprietary SQL to permit a wide choice of database engines.