SMILA/Documentation/Using The ReST API

Most functionality of SMILA is already accessible via an HTTP ReST API using JSON to represent data. This means basically that you can control and monitor a running SMILA server using your web browser (at least with some small plugins). Also, it is relatively simple to talk to SMILA programmatically or using scripts. On this page we will recommend some tools and best practices to work with SMILA.

Feel free to extend this page if you find interesting tools that you want to recommend to other SMILA users.

Basics

An HTTP ReST API consists of a set of URLs that can be invoked using standard HTTP requests. One URL refers to a "resource" in the system, which may be a service or some element managed by a service. Each resource may support several HTTP methods for different operations, but not each resource will support every method. In SMILA we use the following methods, usually with one of the described semantics:

• GET: get the content, definition or description of the resource; get statistic data or state information about the resource
• POST: execute the main functionality of the resource (e.g. execute a pipeline); add or update a sub element of the resource
• PUT: set the content, definition, or description of the resource
• DELETE: clear the contents of the resource; remove an element from the resource.

Most resources should at least support the GET method which is the method used by the web browser when you enter the URL, and the result should contain links to resources associated with this resource. So you can explore the system state with your web browser.

GET and DELETE requests never have a request body, parameters must be passed by using ...?param1=value1&param2=value2 in the URL. With POST and PUT requests the data is passed in the request body as a JSON object, i.e. a JSON string like this:

{
  "param1": "value1",
  "param2": "value2
}

In SMILA, this corresponds to the metadata part of a record. In POST requests it is also possible to add binary attachments, see Documentation of the SMILA HTTP server for details. Finally, some resources accept or produce so-called "JSON bulks". This means that it is possible to send or get a bunch of records in a single request. The JSON of the record metadata must then be printed on a single line, and the records are separated by newlines. However, it is not possible to add attachments to such records.

See SMILA/REST_API_Reference for a list of available resources and links to documentation.

Interactive Tools

Web Browser Add-Ons

For most web browsers there are free add-ons available that support pretty-printed viewing of JSON documents and the interactive invocation of ReST commands.

• JSON viewer: By default browsers do not know how to handle JSON results and will suggest to download them to a file. There are add-ons that display the JSON right in the browser, nicely formatted and highlighted and make contained URLs clickable, which makes it easy to explore the system state
  • Firefox: JSONView
  • Chrome: JSONView
  • Opera: JSONViewer
• ReST Client: ReST Client allow to compose other requests than GET requests with the browser. They allow to enter the resource URL, select the request method, and add the request body content, and then you can execute the request and view the result. There are lots of these add-ons out there, we can recommend the following:
  • Firefox: REST Client
  • Chrome:
    • cREST Client: Easy to use and keeps a nice history of the executed requests so that it is easy to repeat sequences of commands for test purposes.
    • Advanced REST client application: A bit harder to use, but the only one we currently know of that lets you create "multipart" requests containing record attachments: In the following screenshot we have created a request to submit a record (first file field contains name of a JSON file) with an attachment (second file field) to the "indexUpdate" job:
      
  • Opera: Simple REST Client: Very simple only.

Shell scripting

Basically, it is possible to work with the ReST API using generic HTTP tools like cURL. However, for improved usability you might want to check out resty. This is a simple wrapper (for bash and zsh) for cURL that allows you to write ReST requests like shell commands:

# initialize
> resty http://localhost:8080 
# get entry page
> GET /smila                   
# define a job
> POST /smila/jobmanager/jobs '{"name":"myJob","workflow":"myWorkflow","parameters":{ ... }}' 
# etc.

See the resty web site for more information.

Posting attachments from shell scripts

You can use cURL to send a record with some metadata and a binary file as an attachment: Use the -F option to send a multipart request to the specified URL. The part name "metadata" for the JSON part is arbitrary, the JSON part must be the first part in the request. The field name of the following attachment part (here "content") will be the name of the attachment in the created record. You can add multiple attachments with different names this way.

# send a metadata record and a binary attachment to a etlImport job
> curl -F "metadata=@record.json" -F "content=@binary-file.pdf" http://localhost:8080/smila/job/etlImport/record

As resty is based on cURL and passes all options to the cURL call, the same -F options should work there, too.

Programmatical Access

It's quite easy to use the SMILA API from external programs. In Java, basically everything you need is available in packages java.net. But it's easier using the SMILA data model classes and JSON utilities and a HTTP library like the Apache HttpClient. Below we provide some simple examples for doing requests with it.

There are also free Java libraries available to make working with JSON ReST APIs easier, e.g. Resty (yes, same name, but another tool ;-). However, we do not have experience with them.

SMILA RestClient

Since SMILA 1.1.0, we also provide a small library that encapsulates the JSON conversion and HTTP communication code for you. It's based on the SMILA data model classes and the Apache HttpCient 4.2, so it basically wraps up the code shown below. See SMILA/Documentation/HowTo/How_to_access_the_REST_API_with_the_RestClient for details.

Apache HttpClient 4.x

Using HttpClient 4.x (version 4.1 is included in SMILA since 1.1.0) and Apache Commons IO, invoking a pipeline could be done like this:

import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.commons.io.IOUtils;
 
HttpClient client = new DefaultHttpClient();
IpcAnyReader jsonReader = new IpcAnyReader();
IpcAnyWriter jsonWriter = new IpcAnyWriter();
 
AnyMap request = ...; // create request
String jsonString = jsonWriter.writeJsonObject(request);
HttpPost method = new HttpPost("http://localhost:8080/smila/pipelines/MyPipeline/process");
method.setEntity(new StringEntityjsonString, "application/json", "utf-8"));
try {
  HttpResponse response = client.execute(method);
  if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
    InputStream content = response.getEntity().getContent();
    try {
      AnyMap result = (AnyMap) jsonReader.readJsonStream(content);
      ... // use result.
    } finally {
      IOUtils.closeQuietly(content);
    }
  } else if (response.getEntity() != null) {
    EntityUtils.consume(response.getEntity()); // be sure to read the response in error cases, too.
    ... // handle error.
  }
} catch (Exception ex) {
  request.abort();
}

Apache HttpClient 3.1

Using HttpClient 3.1 (included in SMILA) and Apache Commons IO, invoking a pipeline could be done like this:

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.PostMethod;
import org.apache.commons.httpclient.methods.StringRequestEntity;
import org.eclipse.smila.datamodel.ipc.IpcAnyReader;
import org.eclipse.smila.datamodel.ipc.IpcAnyWriter;
import org.eclipse.smila.datamodel.AnyMap;
import org.apache.commons.io.IOUtils;
 
HttpClient client = new HttpClient();
IpcAnyReader jsonReader = new IpcAnyReader();
IpcAnyWriter jsonWriter = new IpcAnyWriter();
 
AnyMap request = ...; // create request
String jsonString = jsonWriter.writeJsonObject(request);
PostMethod method = new PostMethod("http://localhost:8080/smila/pipelines/MyPipeline/process");
method.setRequestEntity(new StringRequestEntity(jsonString, "application/json", "utf-8"));
try {
  client.executeMethod(postMethod);
  if (method.getStatusCode() == HttpStatus.SC_OK) {
    InputStream content = postMethod.getResponseBodyAsStream();
    try {
    AnyMap result = (AnyMap) jsonReader.readJsonStream(content);
    ... // use result.
    } finally {
      IOUtils.closeQuietly(content);
    }
  } else {
    ... // handle error.
  }
} finally {
  method.releaseConnection();
}