SMILA/Documentation/Importing/DeltaCheck

Workers for Importing: Delta Check

Delta Checking is about determining if a record has changed since the last import run and needs to be sent to the processing job again, e.g. to update the index.

Worker Description

• Worker name: deltaChecker
• Parameters:
  • deltaImportStrategy: configures usage of DeltaService. It has four possible values that select one of two behaviours for this worker (see DeltaDelete for an overview):
    • disabled or initial: DeltaService is not used at all by this worker, the input records are just written to the output unchanged. Actually, the worker could be removed from the workflow completely in this case, but for convenience it is possible to let the worker remain it in the workflow, if performance is not that critical, although the worker won't actually do any useful work in these modes.
    • additive or full: Perform normal operation: check state of input records, don't write unchanged records to output.
• Input Slot: recordsToCheck, a recordBulks bucket
• Output Slot:
  • updatedRecords, a recordBulks bucket. Output can be empty, if no record needs an update.
  • updatedCompounds, an optional recordBulks bucket. If connected, compound objects (records with attribute _isCompound set to true) are not written to updatedRecord but to this slot instead. Output can be empty, of course, if no changed or new compound objects were crawled.

The worker calls the DeltaService for each incoming record. The job run id is taken from a task property, while the source ID, record ID and hash code are taken from the record itself. The hash code is expected to be in attribute "_deltaHash" which can contain a single value. Cases are:

• DeltaService reports record as UPTODATE: record is not added to the output.
• DeltaService reports record as NEW: record is added to the output.
• DeltaService reports record as CHANGED: attribute "_update" is set to true and the record is added to the output.
• "_deltaHash" not set: DeltaService is not called and the record is added to the output.
• Error in DeltaService: record is not written to output.

ObjectStoreDeltaService

The DeltaCheck worker makes use of a DeltaService to check and update the state of a record. The bundle org.eclipse.smila.importing.state.objectstore provides an implementation of this service putting those state entries in the ObjectStore (and hence as separate files in a filesystem, if the filesystem implementation of objectstore is used), which should work well enough for a limited number of records per source.

The keys of the entries are created from the source ID, a '/' character and an entry key created from a digest calculated from the record ID. A small configuration file allows to customize this entry key, which may be necessary to manage a greater number of documents or to make use of advanced features of more sophisticated ObjectStore implementation.

The service uses store deltaservice.

Entry key calculation and configuration

Entries are stored in different "shards". This "sharding" is necessary to make it possible to parallelize the checking for deleted records after the import run has finished: For each shard one task will be generated to find the entries in this shard that have not been visited in this run. The shard part of the entry key is determined by taking the first shard.length characters of the record ID digest. The longer this shard part is the more shards can be created and the more the delete check can be parallelized. The default shard.length is 2 (which yields 256 shards, because the digest is a hexadecimal number).

The rest of the digest can be "segmented", i.e. additional '/' can be added so that not all entries of a shard are stored in a single directory. By default, 1 additional '/' is added after the second character of the digest.

The configuration file is org.eclipse.smila.importing.state.objectstore/deltastore.properties:

# Object ID configuration for delta entries in object store.
shard.length=2
segment.count=2
segment.length=1
 
# first argument: shard (first characters of record ID digest)
# second argument: segmented record ID digest
key.pattern=%s/%s

See the test case TestStateStoreConfiguration.java for examples of the effects of these settings.

DeltaService ReST API

Currently there is only a simple REST API for DeltaService that allows to show which sources have currently entries and delete entries of a single source or all entries.

Show active sources

• URL: /smila/importing/delta
• Method: GET
• Response Code: 200 OK, if successful,
• Response JSON:

{"sources": [
  {
    "id": "web",
    "url": "http://localhost:8080/smila/importing/delta/web"
  },
  {
    "id": "file",
    "url": "http://localhost:8080/smila/importing/delta/file"
  }
]}

Clear all sources

• URL: /smila/importing/delta
• Method: DELETE
• Response Code: 200 OK, if successful
• Response JSON: none

Get info about sources

• URL: /smila/importing/delta/<sourcename>
• Method: GET
• Response Code:
  • 200 OK, if successful,
  • 404 NOT FOUND, if source does not have entries currently.

• Response JSON:

Contains the ID of the source and the number of entries. If there are more than 10000 entries, the number is only estimated because exact counting could take a long time. To force an exact count, add ?countExact=true to the request URL.

{
  "id": "web",
  "count": "123456"
}

Clear a single source

• URL: /smila/importing/delta/<sourcename>
• Method: DELETE
• Response Code: 200 OK, if successful
• Response JSON: none