SMILA/Documentation/Bundle org.eclipse.smila.processing.pipelets.xmlprocessing

General

All pipelets in this bundle support the configurable error handling as described in SMILA/Development_Guidelines/How_to_write_a_Pipelet#Implementation. When used in jobmanager workflows, records causing errors are dropped.

Read Type

• runtime: Parameters are read when processing records. Parameter value can be set per Record.
• init: Parameters are read once from Pipelet configuration when initializing the Pipelet. Parameter value can not be overwritten in Record.

org.eclipse.smila.processing.pipelets.xmlprocessing.XslTransformationPipelet

Description

This pipelet performs an XSL transformation on an attribute or attachment value and stores the transformed document in an attribute or attachment.

Configuration

Property	Data Type	Read Type	Description
inputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML input is found in an attachment or in an attribute of the record.
outputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the transformed output should be stored in an attachment or in an attribute of the record.
inputName	String	runtime	The name of the input attachment or the path to the input attribute (process literals of attribute).
outputName	String	runtime	The name of the output attachment or the path to the output attribute (store result as literals of attribute).
xslFile	String	runtime	The name (with relative or absolute path) of the XSL file to be used for transformation.
parameters	Map or Boolean	runtime	Either a map of XSL parameters or a boolean that indicates to add all attributes as XSL parameters.

Example

Pipelet configuration for XslTransformationPipelet

<proc:configuration>
    <rec:Val key="inputType">ATTRIBUTE</rec:Val>
    <rec:Val key="outputType">ATTRIBUTE</rec:Val>
    <rec:Val key="inputName">xmlIn</rec:Val>
    <rec:Val key="outputName">xmlOut</rec:Val>
    <rec:Val key="xslFile">./configuration/data/author.xsl</rec:Val>
</proc:configuration>

org.eclipse.smila.processing.pipelets.xmlprocessing.XPathExtractorPipelet

Description

This pipelet extracts elements selected by XPath, converts them to appropriate data types (Boolean, Double, String), and stores the transformed value in an attribute or attachment.

Configuration

Property	Data Type	Read Type	Description
inputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML input is found in an attachment or in an attribute of the record.
outputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the transformed output should be stored in an attachment or in an attribute of the record.
inputName	String	runtime	The name of the input attachment or the path to the input attribute (process literals of attribute).
outputName	String	runtime	The name of the output attachment or the path to the output attribute (store result as literals of attribute).
xpath	String	runtime	The XPath expression to be evaluated.
separator	String	runtime	The optional separator.
namespace	String	runtime	The optional XML namespace.

Example

Pipelet configuration for XPathExtractorPipelet

<proc:configuration>
    <rec:Val key="inputType">ATTRIBUTE</rec:Val>
    <rec:Val key="outputType">ATTRIBUTE</rec:Val>
    <rec:Val key="inputName">xmlIn</rec:Val>
    <rec:Val key="outputName">xmlOut</rec:Val>
    <rec:Val key="xpath">author/email</rec:Val>
    <rec:Val key="separator"></rec:Val>
    <rec:Val key="namespace"></rec:Val>
</proc:configuration>

---

org.eclipse.smila.processing.pipelets.xmlprocessing.XPathFilterPipelet

Description

This pipelet filters elements by XPath expressions (either using include or exclude mode) and stores the filtered elements as a new document in an attribute or attachment.

Configuration

Property	Data Type	Read Type	Description
inputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML input is found in an attachment or in an attribute of the record.
outputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the transformed output should be stored in an attachment or in an attribute of the record.
inputName	String	runtime	The name of the input attachment or the path to the input attribute (process literals of attribute).
outputName	String	runtime	The name of the output attachment or the path to the output attribute (store result as literals of attribute).
xpath	String	runtime	The XPath expressions to be evaluated (multi-valued property).
filterMode	String : INCLUDE, EXCLUDE	runtime	The filter mode, defining whether to include or exclude the elements matched by the XPath expressions.
namespace	String	runtime	The optional XML namespace.

Examples

Pipelet configuration for XPathFilterPipelet with multi-valued xpath

<proc:configuration>
    <rec:Val key="inputType">ATTRIBUTE</rec:Val>
    <rec:Val key="outputType">ATTRIBUTE</rec:Val>
    <rec:Val key="inputName">xmlIn</rec:Val>
    <rec:Val key="outputName">xmlOut</rec:Val>
    <rec:Seq key="xpath">
        <rec:Val>author/email</rec:Val>
        <rec:Val>author/name</rec:Val>
    </rec:Seq>
    <rec:Val key="filterMode">EXCLUDE</rec:Val>
    <rec:Val key="seperator"></rec:Val>
    <rec:Val key="namespace"></rec:Val>
</proc:configuration>

Pipelet configuration for XPathFilterPipelet with single-valued xpath

<proc:configuration>
    <rec:Val key="inputType">ATTRIBUTE</rec:Val>
    <rec:Val key="outputType">ATTRIBUTE</rec:Val>
    <rec:Val key="inputName">xmlIn</rec:Val>
    <rec:Val key="outputName">xmlOut</rec:Val>
    <rec:Val key="xpath">author/email</rec:Val>
    <rec:Val key="filterMode">EXCLUDE</rec:Val>
    <rec:Val key="seperator"></rec:Val>
    <rec:Val key="namespace"></rec:Val>
</proc:configuration>

---

org.eclipse.smila.processing.pipelets.xmlprocessing.RemoveElementFromXMLPipelet

Description

This pipelet removes a selected element from an XML document and stores the manipulated document in an attribute or attachment.

Configuration

Property	Data Type	Read Type	Description
inputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML input is found in an attachment or in an attribute of the record.
outputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the transformed output should be stored in an attachment or in an attribute of the record.
inputName	String	runtime	The name of the input attachment or the path to the input attribute (process literals of attribute).
outputName	String	runtime	The name of the output attachment or the path to the output attribute (store result as literals of attribute).
elementId	String	runtime	The ID of the XML element to be removed.

Example

Pipelet configuration for RemoveElementFromXMLPipelet

<proc:configuration>
    <rec:Val key="inputType">ATTRIBUTE</rec:Val>
    <rec:Val key="outputType">ATTRIBUTE</rec:Val>
    <rec:Val key="inputName">xmlIn</rec:Val>
    <rec:Val key="outputName">xmlOut</rec:Val>
    <rec:Val key="elementId">title</rec:Val>
</proc:configuration>

---

org.eclipse.smila.processing.pipelets.xmlprocessing.TidyPipelet

Description

This pipelet performs a Tidy transformation on an attribute or attachment value and stores the result in an attribute or attachment.

Configuration

Property	Data Type	Read Type	Description
inputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML input is found in an attachment or in an attribute of the record.
outputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the transformed output should be stored in an attachment or in an attribute of the record.
inputName	String	runtime	The name of the input attachment or the path to the input attribute (process literals of attribute).
outputName	String	runtime	The name of the output attachment or the path to the output attribute (store result as literals of attribute).
tidyFile	String	init	The name (with relative or absolute path) of the Tidy configuration file to be used by the transformation.

Example

Pipelet configuration for TidyPipelet

<proc:configuration>
    <rec:Val key="inputType">ATTRIBUTE</rec:Val>
    <rec:Val key="outputType">ATTRIBUTE</rec:Val>
    <rec:Val key="inputName">xmlIn</rec:Val>
    <rec:Val key="outputName">xmlOut</rec:Val>
    <rec:Val key="tidyFile">./configuration/data/tidy_config.txt</rec:Val>
</proc:configuration>

org.eclipse.smila.processing.pipelets.xmlprocessing.XmlSplitterPipelet

Description

This pipelet splits an XML stream into multiple XML snippets. For each snippet a new record is created where the XML snippet is stored in either an attribute or attachment. The created records are not returned as a PipeletResult (this is just the same as the incoming RecordIds) but are directly sent to the Bulkbuilder and are routed once more to the queue.

On each created record the attribute __isXmlSnippet=true is set to true. Incoming records with this attribute set are not splitted again, but returned as the Pipelet result. This way it's possible to add further processing steps that should be done on the splitted records to the same pipeline that does the splitting.

Namespaces visible in its scope are added to each snippet, thus the result is a valid XML document.

Configuration

Property	Type	Description
inputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML input is found in an attachment or in an attribute of the record. An input attribute is not interpreted as to contain XML content itself but rather as a file path or an URL to the XML document.
outputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML snippet should be stored in an attachment or in an attribute of the newly created record.
inputName	String	runtime	The name of the input attachment or the path to the input attribute containing a path to an external data source, e.g. an xml file name
outputName	String	runtime	The name of the output attachment or the path to the output attribute (store result as literals of attribute).
beginTagName	String	runtime	The name of the tag to start the XML snippet with.
beginTagNamespace	String	runtime	The namespace of the start tag. Namespaces are not checked, if not given (in that case any namespace matches).
endTagName	String	runtime	The name of the tag to end the xml snippet with, defaults to the value of beginTagName
endTagNamespace	String	runtime	The namespace of the end tag, defaults to the value of beginTagNamespace.
keyTagName	String	runtime	The name of the tag used to create a record ID.
maxBufferSize	Integer	runtime	The maximum size of the internal record buffer (optional, default is 20).
idSeparator	String	runtime	The separator used to create the record IDs of the split records (optional, default is "#").
xmlSnippetJobName	String	runtime	The JobManager job name to submit the split records to. It must be running when the pipelet is executed.

The first four attributes can be set only in the pipelet configuration. All other attributes can seperately customized for each single record by setting them as values of the the _parameters map in the record.

Example

Pipelet configuration for XmlSplitterPipelet

<proc:configuration>
    <rec:Val key="inputType">ATTRIBUTE</rec:Val>
    <rec:Val key="outputType">ATTRIBUTE</rec:Val>
    <rec:Val key="inputName">xmlIn</rec:Val>
    <rec:Val key="outputName">xmlOut</rec:Val>
    <rec:Val key="beginTagName">document</rec:Val>
    <rec:Val key="keyTagName">docId</rec:Val>
    <rec:Val key="idSeparator">#</rec:Val>
    <rec:Val key="xmlSnippetJobName">indexUpdateXml</rec:Val>
</proc:configuration>

The above configuration would split this XML format:

<sampleCollection>
    ...
    <document>
        <docId>4711</docId>
        <title>Some title</title>
        ...
        <text>Some text</text>
    </document>
    <document>
        <docId>0815</docId>  
        ...
    </document>
    ...
</sampleCollection>

into XML snippets like this one:

<document>
    <docId>4711</docId>
    <title>Some title</title>
    ...
    <text>Some text</text>
</document>

And for each snippet a record would be created and submitted to a job run of job "indexUpdateXml".

<Record xmlns="http://www.eclipse.org/smila/record" version="2.0">
  <Val key="_recordid">xmlsplitter:someBigXmlfile.xml#4711</Val>
  <Val key="_source">xmlsplitter</Val>
  <Val key="__isXmlSnippet">true</Val>
  <Val key="xmlOut">
         <document>
            <docId>4711</docId>
            <title>Some title</title>
            ...
            <text>Some text</text>
        </document>   
  </Val>
</Record>

org.eclipse.smila.processing.pipelets.xmlprocessing.XmlDocumentSplitterPipelet

Description

This is a variant of the XmlSplitterPipelet described above. It splits an XML text contained in an attribute or attachment of the input record, splits it at configurable tags into smaller XML documents and creates a "split record" for each of the fragments. The differences are:

• the split records are not submitted to another job, but replace the input records in the pipelet result.
• All attributes from the input records are copied to the split records, too.
• the "_recordid" of the input record is copied to attribute "_documentId" of each split record.
• the "_recordids" of the split record are created by appending "###" and a counter to the "_recordid" of the input record.

This is similar to what the DocumentSplitterPipelet is doing on structured records, but it works on XML text input instead.

Configuration

Property	Type	Description
inputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML input is found in an attachment or in an attribute of the record. An input attribute is not interpreted as to contain XML content itself but rather as a file path or an URL to the XML document.
outputType	String : ATTACHMENT, ATTRIBUTE	runtime	Defines whether the XML snippet should be stored in an attachment or in an attribute of the newly created record.
inputName	String	runtime	The name of the input attachment or the path to the input attribute containing a path to an external data source, e.g. an xml file name
outputName	String	runtime	The name of the output attachment or the path to the output attribute (store result as literals of attribute).
beginTagName	String	runtime	The name of the tag to start the XML snippet with.
beginTagNamespace	String	runtime	The namespace of the start tag. Namespaces are not checked, if not given (in that case any namespace matches).
endTagName	String	runtime	The name of the tag to end the xml snippet with, defaults to the value of beginTagName
endTagNamespace	String	runtime	The namespace of the end tag, defaults to the value of beginTagNamespace.