SMILA/Documentation/Feed Agent

Overview

The Feed agent offers the functionality to receive RSS and Atom feeds on a regular basis. The implementation uses ROME and ROME Fetcher to retrieve and parse the feeds. ROME supports the following feed formats:

• RSS 0.90
• RSS 0.91 Netscape
• RSS 0.91 Userland
• RSS 0.92
• RSS 0.93
• RSS 0.94
• RSS 1.0
• RSS 2.0
• Atom 0.3
• Atom 1.0.

Agent configuration

The example configuration file is located at configuration/org.eclipse.smila.connectivity.framework/feeds.xml.

Defining Schema: org.eclipse.smila.connectivits.framework.agent.feed/schemas/FeedDataSourceConnectionConfigSchema.xsd.

Agent configuration explanation

See SMILA/Documentation/Agent#Configuration for the generic parts of the configuration file.

The root element of the configuration is DataSourceConnectionConfig and contains the following sub elements:

• DataSourceID – the identification of a data source
• SchemaID – specifies the schema for the data source
• DataConnectionID – describes which agent or crawler should be used
  • Crawler – service ID a crawler
  • Agent – service ID of an agent
• CompoundHandling – specify if packed data (like a ZIP containing files) should be unpack and files within should be processed (YES or NO).
• Attributes – list all attributes provided by the data source
  • Attribute
    • attributes:
      • Type (required) – the data type (String, Integer or Date).
      • Name (required) – attributes name.
      • HashAttribute – specify if the attribute is used for the hash used for delta indexing (true or false). Must be true for at least one attribute which must always have a value.
      • KeyAttribute – specify if the attribute is used for creating the record ID (true or false). Must be true for at least one attribute. All key attributes used together must identify the entry uniquely, so usually the FeedAttribute Uri will be used.
      • Attachment – specify if the attribute return the data as attachment of record.
    • sub elements:
      • FeedAttributes content describes what to get from the feed entry. For allowed values see the table below.
• Process – contains parameters for the agent business logic.
  • UpdateInterval – the number of seconds to wait before reloading the feeds specified by FeedUrl.
  • FeedUrl – the URL of the news feed to load. You may specify multiple FeedUrls.

Here is a description of the attributes that the Feed agent offers. It provides attributes about the feed itself (using prefix Feed) and attributes for entries of the feed. Some attributes do not return literals (string, date) but nested objects like Person, Link, etc. These objects are all MObjects that contain attributes themselves. The nested MObjects and their attributes (the attribute names are hard coded and cannot be configured) are described below. Note that not all feeds necessarily provide values for all the attributes and that some values are provided that are not apperently part of the feed.

These are the attributes of the feed:

Attribute	Type	Description
FeedAuthors	List<Person>	Returns the feed authors
FeedCategories	List<Category>	Returns the feed categories
FeedContributors	List<Person>	Returns the feed contributors
FeedCopyright	String	Returns the feed copyright information
FeedDescription	String	Returns the feed description
FeedEncoding	String	Returns the charset encoding of the feed
FeedType	String	Returns the feed type
FeedImage	Image	Returns the feed image
FeedLanguage	String	Returns the feed language
FeedLinks	List<Link>	Returns the feed links
FeedPublishDate	Date	Returns the feed published date
FeedTitle	String	Returns the feed title
FeedUri	String	Returns the feed uri

And here are the attributes of feed entries:

Attribute	Type	Description
Authors	List<Person>	Returns a feed entry authors
Categories	List<Category>	Returns a feed entry categories
Contents	List<Content>	Returns a feed entry contents
Contributors	List<Person>	Returns a feed entry contributors
Description	Content	Returns a feed entry description
Enclosures	List<Enclosure>	Returns a feed entry enclosures
Links	List<Link>	Returns a feed entry links
PublishDate	Date	Returns a feed entry publish date
Title	String	Returns a feed entry title
Uri	String	Returns a feed entry uri.
UpdateDate	Date	Returns a feed entry update date.

MObject Person:

Attribute	Type	Description
Email	String	Returns the email of the person
Name	String	Returns the name of the person
Uri	String	Returns the uri of the person

MObject Image:

Attribute	Type	Description
Link	String	Returns the link of the image
Title	String	Returns the title of the image
Url	String	Returns the url of the image
Description	String	Returns the description of the image

MObject Category:

Attribute	Type	Description
Name	String	Returns the name of the category
TaxanomyUri	String	Returns the taxonomy uri of the category

MObject Enclosure:

Attribute	Type	Description
Type	String	Returns the type of the enclosure
Url	String	Returns the url of the enclosure
Length	Integer	Returns the length of the enclosure

MObject Link:

Attribute	Type	Description
Href	String	Returns the href of the link
Hreflang	String	Returns the hreflang of the link
Rel	Integer	Returns the rel of the link
Title	String	Returns the title of the link
Type	String	Returns the type of the link
Length	Integer	Returns the length of the link

MObject Content:

Attribute	Type	Description
Mode	String	Returns the mode of the content
Value	String	Returns the value of the content
Type	String	Returns the type of the content

Configuration example

<DataSourceConnectionConfig
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:noNamespaceSchemaLocation="../org.eclipse.smila.connectivity.framework.agent.feed/schemas/FeedDataSourceConnectionConfigSchema.xsd"
>
  <DataSourceID>feeds</DataSourceID>
  <SchemaID>org.eclipse.smila.connectivity.framework.agent.feed</SchemaID>
  <DataConnectionID>
    <Agent>FeedAgent</Agent>
  </DataConnectionID>
  <CompoundHandling>Yes</CompoundHandling>
  <Attributes>
    <Attribute Type="Date" Name="PublishDate" HashAttribute="true">
      <FeedAttributes>PublishDate</FeedAttributes>
    </Attribute>
    <Attribute Type="Date" Name="LastModifiedDate" HashAttribute="true">
      <FeedAttributes>UpdateDate</FeedAttributes>
    </Attribute>
    <Attribute Type="String" Name="Uri" KeyAttribute="true">
      <FeedAttributes>Uri</FeedAttributes>
    </Attribute>
    <Attribute Type="String" Name="Links">
      <FeedAttributes>Links</FeedAttributes>
    </Attribute>    
    <Attribute Type="MObject" Name="Contents">
      <FeedAttributes>Contents</FeedAttributes>
    </Attribute>
    <Attribute Type="String" Name="Title">
      <FeedAttributes>Title</FeedAttributes>
    </Attribute>
    <Attribute Type="MObject" Name="Authors">
      <FeedAttributes>Authors</FeedAttributes>
    </Attribute>    
  </Attributes>
  <Process>
    <UpdateInterval>300</UpdateInterval>
    <FeedUrl>http://dev.eclipse.org/newslists/news.eclipse.rt.smila/maillist.rss</FeedUrl>
    <FeedUrl>http://search.twitter.com/search.atom?q=smila</FeedUrl>
  </Process>
</DataSourceConnectionConfig>

Output example

A record created by the Feed agent using the default configuration may have the following or a similar structure:

<Record xmlns="http://www.eclipse.org/smila/record" version="1.0">
  <Val key="_recordid">feed:&lt;Url=http://dev.eclipse.org/mhonarc/newsLists/news.eclipse.rt.smila/msg00022.html</Val>
  <Val key="_source">feed</Val>
  <Val key="PublishDate"> type="datetime">2009-04-30T13:28:34+0100</Val>
  <Val key="Url">http://dev.eclipse.org/mhonarc/newsLists/news.eclipse.rt.smila/msg00022.html</Val>
  <Val key="MimeType">text/html</Val>
  <Val key="Title">[news.eclipse.rt.smila] Re: Semantic Software Engineering</Val>
  <Seq key="Contents">
    <Map>
      <Val key="Value">Hi J&#xFC;rgen, The idea is to support companies and projects that rely on semantic technologies (especially in RDF or OWL) with a set of plugins that they can reuse for their tooling. The first thing would be support for loading an ontology, searching for conc...</Val>
      <Val key="Type">text/html</Val>
    </Map>
  </Seq>
  <Seq key="Authors">
    <Map>
      <Val key="Name">lautenbacher@xxxxxxx (Florian Lautenbacher)</Val>
    </Map>
  </Seq>
</Record>

See also

• Agent
• Mock Agent