COSMOS Design 193420

Design Discussion for 193420: Create simple stand alone example of the data collection framework

The purpose of this is a starting discussion point, NOT a solution.

Comments

Introduction

This is intended to be the most basic example of using the COSMOS data collection framework.

It will have no prerequisites other than COSMOS itself. Is this true??

This overview will walk the developer through an introduction to the use of the two key constructs in the data collection component, query assemblies and collection assemblies. <brief description of query assemblies> <brief description of collection assemblies>

Details/Ideas

1. List the mechanisms for generating the key family file (eg. Dynamic or hard coded)

2. For each mechanism take the raw data and generate a key family file.

   a.  In the document *** show what the data looks like, so for an rdbms do a select * from the table and 
       show the output
   b.  For a text file, literally paste the text file in the document ***

3. For each mechanism manaually create the key family file (we don’t want to get into how we will eventually generate them)

4. Copy the key family files into the document ***

5. List the possible place where these files could reside eg. In Memory, in a database, in a file on a share, within an application server, ftp server etc.

6. Define ways that the client can get to this key family data, does it get it? Does it get pushed it? Does it request it and then it is pushed? Does it get the whole file? Does it just get the reference to where it is ?

7. What are the formats that this key family file are in (eg. Straight XML, SDMX compliant, pojo)

8. For each of the different formats that it can be in, what mechanism will be used to read them on the client side ? Do we use a Sax parser? A Dom Parser? Some SDMX based parser ? This will end up being the helper APIs for the client side.

9. Does the client have the ability to know what format(s) the key family file is in? If so, how does it get that information ? How does it then request to the Data Manager that it wants it in a specific format?

10. Once the client has retrieved the key family file and parsed it, how will it decide on what it wants ? Do we always convert the key family file into one format on the client side so that when it comes to look at what is available, it just looks at that one format? Eg. We set up a multi array string that contains the data and the client always does its processing against this array.

11. The formulation of the query to request the data manager:-

   a.  How does the client now what query mechanism the Data Manager uses ?  Is it XML, SDMX, XPATH, SQL, SML ?
       How is the data manager going to relay that to the client?
   b.  When the client knows what query mechanisms are available, it has to choose which one it will use 
       (this then becomes a method in the class, eg   public void chooseQueryType()
   c.  List 3 queries for retrieving data from the rdbms, and 3 queries for the flat file (list them in 
       Plain Old English POE) put in the document ***
   d.  Convert each of these queries into the known query types, and put in the document ***
   e.  Write the pseudo code that describes the process of selection from the string array, and then converting 
       that into the query
   f.  What is the mechanism for getting this query to the DM ? execution of a standard method in the DM’s web 
       service, and passing the query as a string variable  ?

12. Once the DM has the query from the client,

   a.  how does it know what the query language is?
   b.  how does it know if it is a valid query?   (clientQuery.validate(inQuery))
   c.  How does it know what format the client wants the data to be returned as ?

13. Formatting the data before its returned to the client

   a.  When the data is retrieved, how do we generate the data to be returned
   b.  List the methods that could be executed for each generation type
   c.  For each of the 3 example queries fort both the rdbms and flat file, write in the document *** 
       exactly how the file will look before it gets sent back to the client.

14. Returning the data to the Client

   a.  How does that data get back to the client?  SOAP?

15. Client processes the data

   a.  Write the pseudo code of how the client takes the file of data and parses it.

---