SMILA/Documentation/JobManagerFirstExample

This is a simple walkthrough on index building using the new JSON ReST APIs and the job management.

See SMILA/Documentation/Processing/JSON REST API for BPEL pipelines on how to use the JSON ReST API

.

Use a Workflow

You could create your own asynchronous workflow, but we use the "indexUpdate" workflow that is already provided with SMILA. It uses the BPEL pipelines from the standard configuration to add and delete index documents.

GET /smila/jobmanager/workflows/indexUpdate/
 
HTTP/1.x 200 OK
 
{
    "name": "indexUpdate",
    "parameters": 
    {
        "pipelineRunBulkSize": "20"
    },
    "startAction":
    {
        "worker": "bulkbuilder",
        "output":
        {
            "insertedRecords": "addBucket",
            "deletedRecords": "deleteBucket"
        }
    },
    "actions":
    [
        {
            "worker": "pipelineProcessor",
            "parameters": 
            {
                "pipelineName": "AddPipeline"
            },
            "input":
            {
                "input": "addBucket"
            }
        },
        {
            "worker": "pipelineProcessor",
            "parameters": 
            {
                "pipelineName": "DeletePipeline"
            },
            "input":
            {
                "input": "deleteBucket"
            }
        }
    ]
}

Create a Job

Now we have to create a job that uses this workflow:

POST /smila/jobmanager/jobs/
{
    "name": "exampleIndexUpdate",
    "workflow": "indexUpdate",
    "parameters": 
    {
        "tempStore": "tempStore"
    }
}

You get a reponse:

{
    "name": "exampleIndexUpdate",
    "timestamp": "2011-08-15T16:20:34.337+0200",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/"
}

Start a Job Run

Now this job has to be started:

POST /smila/jobmanager/jobs/exampleIndexUpdate/

The response is:

{
    "jobId": "20110815-162046851752",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/"
}

We will need the URL from this response later to finish the job run.

Add a Document

POST /smila/job/exampleIndexUpdate/record/
{
  "_recordid": "test.html",
  "_source": "handcrafted",
  "Title": "Hello Job World!",
  "Content": "This is the first document added to an SMILA index using the new job management",
  "MimeType": "text/plain",
  "Size": 42
}

Flush the bulk:

POST /smila/job/exampleIndexUpdate/record/

For both requests the response should be similar to:

{
    "workflowRunId": "1",
    "jobRunId": "20110815-162046851752",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/1/"
}

After a while (about a minute) the document can be found in the sample search site http://localhost:8080/SMILA/search. Hint: search for "first".

Delete a Document

DELETE /smila/job/exampleIndexUpdate/record/?_recordid=test.html

Flush the bulk:

POST /smila/job/exampleIndexUpdate/record/

Again, you get a response for both request like this:

{
    "workflowRunId": "2",
    "jobRunId": "20110815-162046851752",
    "url": "http://localhost:8080/smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/workflowrun/2/"
}

After a while (about a minute), the search should not return any results anymore.

Finish the Job Run

Look up the URL from the response of the start-job request and add "finish" to get the path for this POST request:

POST /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/finish/

The response will be empty, but you should get a response code of 202.

Finally you can request statistics about this job run:

GET /smila/jobmanager/jobs/exampleIndexUpdate/20110815-162046851752/

and get:

{
    "endTime": "2011-08-15T16:52:18.726+0200",
    "finishTime": "2011-08-15T16:52:18.714+0200",
    "jobId": "20110815-162046851752",
    "mode": "STANDARD",
    "startTime": "2011-08-15T16:20:46.920+0200",
    "state": "SUCCEEDED",
    "workflowRuns": {
        "activeWorkflowRunCount": 0,
        "canceledWorkflowRunCount": 0,
        "failedWorkflowRunCount": 0,
        "startedWorkflowRunCount": 2,
        "successfulWorkflowRunCount": 2
    },
    "tasks": {
        "canceledTaskCount": 0,
        "createdTaskCount": 4,
        "failedAfterRetryTaskCount": 0,
        "failedWithoutRetryTaskCount": 0,
        "obsoleteTaskCount": 0,
        "retriedAfterErrorTaskCount": 0,
        "retriedAfterTimeoutTaskCount": 0,
        "successfulTaskCount": 4
    },
    "worker": { ... },
    "jobDefinition": { ... }
}