COSMOS DC Design Discussions

Presentation

• Binding service: The binding service is used for instantiating and initializing components. The current design requires exploiters of the framework to provide a binding service for user defined components. However, the information required for the binding service can be extracted from the assembly XML and can be abstracted into the framework level. If you look at the SampleBindingService and GLABindingService, they are almost in the same pattern.
• The coupling between components
  • The components should be loosely coupled.
  • The current framework design uses an event model to pass events/data from component to component in the assembly.
  • after each event is processed, the object is dispatched by calling the wire method of the next "component" in the assembly. This is potentially inefficient and make optimization difficult. For example, when I wrote the CBE data sink, I had to do an insert for each CBE. If I have access to the whole list of CBE objects, I can do a batch data insert operation.
  • Question: Do we need to support the use case where the same data source need to be dispatched to multiple data transformers or data sinks concurrently? The current assembly XML schema design uses a containment structure for components, and allows a data source to pipe data into multiple filters, and each filter in turn pipe data into multiple data sinks. (The code example has not exploited this scenario yet.)

<context>

Invalid language.

You need to specify a language like this: <source lang="html4strict">...</source>

Supported languages for syntax highlighting:

4cs, 6502acme, 6502kickass, 6502tasm, 68000devpac, abap, actionscript, actionscript3, ada, algol68, apache, applescript, apt_sources, arm, asm, asp, asymptote, autoconf, autohotkey, autoit, avisynth, awk, bascomavr, bash, basic4gl, bf, bibtex, blitzbasic, bnf, boo, c, c_loadrunner, c_mac, caddcl, cadlisp, cfdg, cfm, chaiscript, cil, clojure, cmake, cobol, coffeescript, cpp, cpp-qt, csharp, css, cuesheet, d, dcl, dcpu16, dcs, delphi, diff, div, dos, dot, e, ecmascript, eiffel, email, epc, erlang, euphoria, f1, falcon, fo, fortran, freebasic, freeswitch, fsharp, gambas, gdb, genero, genie, gettext, glsl, gml, gnuplot, go, groovy, gwbasic, haskell, haxe, hicest, hq9plus, html4strict, html5, icon, idl, ini, inno, intercal, io, j, java, java5, javascript, jquery, kixtart, klonec, klonecpp, latex, lb, ldif, lisp, llvm, locobasic, logtalk, lolcode, lotusformulas, lotusscript, lscript, lsl2, lua, m68k, magiksf, make, mapbasic, matlab, mirc, mmix, modula2, modula3, mpasm, mxml, mysql, nagios, netrexx, newlisp, nsis, oberon2, objc, objeck, ocaml, ocaml-brief, octave, oobas, oorexx, oracle11, oracle8, otj, oxygene, oz, parasail, parigp, pascal, pcre, per, perl, perl6, pf, php, php-brief, pic16, pike, pixelbender, pli, plsql, postgresql, povray, powerbuilder, powershell, proftpd, progress, prolog, properties, providex, purebasic, pycon, pys60, python, q, qbasic, rails, rebol, reg, rexx, robots, rpmspec, rsplus, ruby, sas, scala, scheme, scilab, sdlbasic, smalltalk, smarty, spark, sparql, sql, stonescript, systemverilog, tcl, teraterm, text, thinbasic, tsql, typoscript, unicon, upc, urbi, uscript, vala, vb, vbnet, vedit, verilog, vhdl, vim, visualfoxpro, visualprolog, whitespace, whois, winbatch, xbasic, xml, xorg_conf, xpp, yaml, z80, zxbasic

     <filter>
       <sink>
       </sink>
     </fliter>

</context>

This is a more generic design but brings with it complexity. This document structure will also make it difficult to deine an XML scheme to describe the structure. We can adapt a serial design which is more straight forward:

<context>

Invalid language.

You need to specify a language like this: <source lang="html4strict">...</source>

Supported languages for syntax highlighting:

4cs, 6502acme, 6502kickass, 6502tasm, 68000devpac, abap, actionscript, actionscript3, ada, algol68, apache, applescript, apt_sources, arm, asm, asp, asymptote, autoconf, autohotkey, autoit, avisynth, awk, bascomavr, bash, basic4gl, bf, bibtex, blitzbasic, bnf, boo, c, c_loadrunner, c_mac, caddcl, cadlisp, cfdg, cfm, chaiscript, cil, clojure, cmake, cobol, coffeescript, cpp, cpp-qt, csharp, css, cuesheet, d, dcl, dcpu16, dcs, delphi, diff, div, dos, dot, e, ecmascript, eiffel, email, epc, erlang, euphoria, f1, falcon, fo, fortran, freebasic, freeswitch, fsharp, gambas, gdb, genero, genie, gettext, glsl, gml, gnuplot, go, groovy, gwbasic, haskell, haxe, hicest, hq9plus, html4strict, html5, icon, idl, ini, inno, intercal, io, j, java, java5, javascript, jquery, kixtart, klonec, klonecpp, latex, lb, ldif, lisp, llvm, locobasic, logtalk, lolcode, lotusformulas, lotusscript, lscript, lsl2, lua, m68k, magiksf, make, mapbasic, matlab, mirc, mmix, modula2, modula3, mpasm, mxml, mysql, nagios, netrexx, newlisp, nsis, oberon2, objc, objeck, ocaml, ocaml-brief, octave, oobas, oorexx, oracle11, oracle8, otj, oxygene, oz, parasail, parigp, pascal, pcre, per, perl, perl6, pf, php, php-brief, pic16, pike, pixelbender, pli, plsql, postgresql, povray, powerbuilder, powershell, proftpd, progress, prolog, properties, providex, purebasic, pycon, pys60, python, q, qbasic, rails, rebol, reg, rexx, robots, rpmspec, rsplus, ruby, sas, scala, scheme, scilab, sdlbasic, smalltalk, smarty, spark, sparql, sql, stonescript, systemverilog, tcl, teraterm, text, thinbasic, tsql, typoscript, unicon, upc, urbi, uscript, vala, vb, vbnet, vedit, verilog, vhdl, vim, visualfoxpro, visualprolog, whitespace, whois, winbatch, xbasic, xml, xorg_conf, xpp, yaml, z80, zxbasic

...

  <filter>...</fliter>
  <sink>...</sink>
</context>

• wire method: The method name is not in the interface. Reason is probably because we need a different name for each component, and the parameter object type is also different. Assumption of the method name and introspection on the available methods and comparison with a list of acceptable object types is done. I find this process quite complicated, and potentially difficult to debug. An alternative is to use a generic method name such as process, that pass in an array of Objects. It's up to the implementation of the component to decide how to handle the object.
• Separation of concerns: There are three crosscutting concerns in the data collection code:
  1. assembly definition: the definition of the assembly XML files, and writing code for a assembly component (e.g. a transformer).
  2. metadata definition: Mapping of legacy data models to the generic metadata specification (SDMX-like model)
  3. WSDM management features
• Currently, the above three concerns are too intertwined, making the framework very difficult to use. The definition of datasource, keyset, etc, should separated from the assembly XML, and not cached in component implementations.
• Query framework: I find the query framework still quite ad hoc. There is no standard query mechanism. The query assembly binds a query name with a class, which implements an application specific query interface. The implementation of the query can use any method. I somehow find the concept of "inbound" vs. "outbound" in a data assembly unnecessary. We can provide standard datasource that makes SQL queries to a relation database, or XPATH queries on XML documents. These kinds of generic "data source" or "queries" can be used in inbound or outbound data collection.
• Other things that can be done for data collection:
  • Hello world example for using the dc framework
  • More useful data source and sinks:
    • GLA datasource: The current example uses an "embedded GLA source". The GLA source can be an external GLA instance.
    • Revisit CBE schema and decision regarding interoperability with TPTP.
  • a more user friendly interface (command line or UI) for interacting with the dc framework.
  • documentation

Joel's comments