Tutorial: ECF Remote Services for Accessing Existing REST Services

Introduction

Remote Services are usually defined with both a 'host' component (aka 'server') and a 'consumer' component (aka 'client'). The host typically implements and exports a service for remote access, and the consumer discovers, imports, and then uses the remote service by calling methods on the service interface. ECF has a previous tutorial showing how to create a custom RESTful Remote Service Provider with both these host and consumer components, and there are also a number of other tutorials and examples showing how to use existing ECF providers to expose, discover, import, and use a remote service.

There are many situations, however, where it would be desirable to expose an existing (e.g. web-based REST) service as a Remote Service. There are of course many public and private web services already in place. These services are frequently not running within an OSGi framework, nor are implemented in Java, and therefore not exposed as a Remote Service host. For many such existing services, it would be valuable to have only the consumer of the service be running OSGi and using Remote Services, and not require that the host be running within an OSGi framework, nor even necessarily be implemented in Java.

With any Remote Service, what the consumer sees at the time of use is a dynamically-constructed proxy implementing one or more service interfaces. ECF's RSA implementation provides open, community-developed APIs allowing full customization of the both the creation and runtime behavior of the proxy, further enabling the creation of a 'consumer/client-only' remote service provider capable of interacting with any Internet-based service.

The use of a 'consumer/client-only' provider has several important advantages for consumer application development

Existing server/services don't change, and may be implemented in any language, using any framework or service infrastructure
Existing clients don't change, and may continue to be implemented in any language, using any framework
This Remote Service consumer can automatically benefit from advantages of using Remote Services
1. Clear separation between the remote service interface/API and the remote service implementation
2. Support for rich versioning (e.g. of the service interface)
3. Full dynamics of the remote service
4. Remote service discovery
5. Fine-grained service-level type-safe security
6. Standardized management agent via Remote Service Admin (RSA)
7. Synchronous and/or asynchronous/non-blocking access to the remote service without any implementation changes
8. Automatic integration with/use of a variety of injection frameworks (e.g. Spring/Blueprint, Declarative Services, others)
9. Automatic integration with any framework that uses OSGi Services
10. Use of open standards (OSGi Remote Services/RSA) and open, community-developed implementations (ECF Remote Services) to avoid vendor lock-in and provide easy customization

The tutorial below will focus on the creation of a remote service client provider that exposes an existing REST service as an OSGi Remote Service. The example remote service used in this tutorial is the Geonames Timezone service, which is a publicly available service to provide timezone information for a given point on Earth. However, the steps shown below, however, may be applied to creating a consumer/client for any remote service, whether exposed via REST/xml, REST/json, via http/https, or any other protocol. This allows a very powerful approach to integration of remote services into applications: The use of all of the advantages of ECF Remote Services listed above, without any required changes existing services or existing clients of those services.

The complete code for this tutorial is available at the ECF GitHub GeoNames repository.

Timezone Service Interface

The OSGi Services model is based upon a clear separation between the service interface (one or more java interface classes) and the service implementation (a POJO instance that implements those interfaces. The first step in making the Geonames Timeservice available as an ECF Remote Service is therefore to declare a new service interface class that exposes the appropriate semantics of the Geonames Timezone service.

package org.geonames.timezone;
/**
 * Service interface to represent consumer access to the Geonames timezone service. 
 * This web-service is documented here:
 * http://www.geonames.org/export/web-services.html#timezone
 *
 */
public interface ITimezoneService {
	/**
	 * Get a timezone instance, given latitude and longitude values.
	 * 
	 * @param latitude the latitude of the location to get the timezone for
	 * @param longitude the longitude of the location to get the timezone for
	 * @return the Timezone information for the given latitude and longitude.  Should
	 * return <code>null</code> if the latitude or longitude are nonsensical (e.g. negative values)
	 */
	Timezone getTimezone(double latitude, double longitude);
}

The getTimezone method returns an instance of the Timezone class. The Timezone class exposes the data returned by the Geonames Timezone service after converting the JSON to a Java Timezone instance. Here is the full source to both the ITimezoneService and the Timezone classes.

Asynchronous Remote Service

ECF Remote Services can automatically and dynamically create an asynchronous proxy to the remote service. This allows the consumer application (the code that calls the ITimezoneService.getTimezone method) to choose whether to make a synchronous remote call that will block the calling thread until the remote call is complete, or to make an asynchronous remote call that allows the application thread to continue while the remote call is made. As described in Asynchronous Remote Services all that's necessary to have ECF create an asynchronous proxy is to declare an asynchronous version of the ITimezoneService interface. Here is such a declaration for the ITimezoneService interface

package org.geonames.timezone;
 
import java.util.concurrent.CompletableFuture;
/**
 * Asynchronous version of ITimezoneService
 */
public interface ITimezoneServiceAsync {
	/**
	 * Immediately (without blocking) return a CompletableFuture to later retrieve a
	 * returned Timezone instance
	 * 
	 * @param latitude the latitude of the location to get the timezone for
	 * @param longitude the longitude of the location to get the timezone for
	 * @return the CompletableFuture<Timezone> to provide later access to an
	 * instance of Timezone information for the given latitude and longitude.  Should
	 * not return <code>null</code>.
	 */
	CompletableFuture<Timezone> getTimezoneAsync(double latitude, double longitude);	
}

Note the 'Async' suffix in both the interface class name (ITimezoneServiceAsync) and the method name (getTimezoneAsync). This is as described in Asynchronous Remote Services and is all that is necessary to get ECF's Remote Services implementation to dynamically construct a proxy exposing the ITimezoneServiceAsync method for use by consumer/application code.

These three classes: ITimezoneService, Timezone, and ITimezoneServiceAsync are all that are needed in this service API bundle. The manifest.mf and other meta-data to complete this bundle is available in our Github repository here. Note that there are no references in this API to ECF classes, or any other dependencies for that matter. As it should be, this API is completely independent of the distribution system, and only depends upon the semantics of the Geonames Timezone service.

Client Provider

ECF's implementation of Remote Services exposes open, community-created APIs to make it easy to create and use a custom Client Provider. What is a Client Provider? It's a custom implementation of the consumer/client-side of the remote services distribution system. An ECF client provider is able to substitute custom protocols/transports (e.g. http/https, MQTT, JMS, private protocols) and custom serialization formats (e.g. xml, JSON, object serialization, others), without having to implement the entire OSGi Remote Services and Remote Service Admin specification. In an open, modular fashion, an ECF Client Provider can reuse any/all existing ECF or third party remote service provider functionality. A Client Provider automatically is fully compliant with the OSGi Remote Services/RSA standards, while providing the flexibility to use new, custom, or even private transports. Here is an architecture diagram showing the relationship between distribution providers, the Remote Services standards and ECF's extensible and open implementation.

Remote Service Admin Integration

The Remote Service Admin specification standardizes an API and mechanism for dynamic discovery of remote services. ECF's implementation of this standard provides a way for new Client Providers to dynamically insert themselves into this discovery process. The way this is done is to implement a new container class along with a container instantiator, and to register a container type description instance as a local OSGi service using the whiteboard pattern.

For the example Timezone remote service the container class is called TimezoneClientContainer. The full source for this class is available here.

The role of the container instantiator is to create an appropriately typed instance of the container when needed by RSA. When a remote service is discovered and then imported, the ECF RSA implementation takes the OSGi-define remote service properties and passes them to every available container instantiator, essentially asking: 'a container of given type has been discovered, and a new instance is needed, can you provide such an instance?'. The implementation of this query is via the TimezoneClientContainer.Instantiator.getImportedConfigs method (labelled in comment with '1').

	public static class Instantiator extends RestClientContainerInstantiator {
		/**
		 * 1. This method is called by the ECF RSA implementation when a remote
		 * service is to be imported. The exporterSupportedConfigs parameter
		 * contains the exported config types associated with the remote
		 * service. The implementation of this method decides whether we are
		 * interested in this remote service config type. If we are
		 * (exporterSupportedConfigs contains CONTAINER_TYPE_NAME, then we
		 * return an array of strings containing our CONTAINER_TYPE_NAME
		 */
		public String[] getImportedConfigs(
				ContainerTypeDescription description,
				String[] exporterSupportedConfigs) {
			/**
			 * If the exporterSupportedConfigs contains CONTAINER_TYPE_NAME,
			 * then return that CONTAINER_TYPE_NAME to trigger RSA usage of this
			 * container instantiator
			 */
			if (Arrays.asList(exporterSupportedConfigs).contains(
					CONTAINER_TYPE_NAME))
				return new String[] { CONTAINER_TYPE_NAME };
			return null;
		}
		/**
		 * 2. This method is called by the ECF RSA to create a new instance of
		 * the appropriate
		 * container type (aka OSGi config type)
		 */
		public IContainer createInstance(ContainerTypeDescription description,
				Object[] parameters) throws ContainerCreateException {
			return new TimezoneClientContainer();
		}
	}

. The getImportedConfigs method logic is summarized as: if my CONTAINER_TYPE_NAME matches one of the exporterSupportedConfigs then return a String[] including my CONTAINER_TYPE_NAME. This approach provides the flexibility to support multiple exporterSupportedConfigs with a single container type. The ECF RSA implementation then turns around and calls the TimezoneClientContainer.'Instantiator.createContainer method, which in the implementation above (2) returns a new instance of the TimezoneClientContainer.

Once the TimezoneClientContainer instance is created, ECF RSA calls it's connect method with the targetID defined in the standardized meta-data for the remote service. Here are the lines of the TimezoneClientContainer connect method explained

super.connect(targetID, connectContext1);

This line passes the RSA-created targetID to the RestClientContainer super class, which simply sets a member variable to the value of targetID.

setAlwaysSendDefaultParameters(true);

This line configures the container instance to always send the default parameters on the remote call. This is necessary for the Geonames Timezone service because one of the required http request parameters is the username parameter. Setting this to 'true' means that the default parameter values (username) are always sent as required by the Timezone service.

To automatically construct the proxy instance of the ITimezoneService and ITimezoneServiceAsync it's necessary to define the association between the getTimezone method and the Geonames Timezone request. As documented here, the Timzeone request consists of

A complete URL for the request (protocol/hostname/port + path)
The required parameters. In the case of the Geonames Timezone service the parameters are
1. latitude (aka 'lat')
2. longitude ('lng')
3. username (for authentication)

First we construct the necessary remote call parameters using a RemoteCallParameter.Builder

RemoteCallParameter.Builder parameterBuilder = new RemoteCallParameter.Builder()
		.addParameter("lat").addParameter("lng")
		.addParameter("username", USERNAME);

Then we construct an IRemoteCallable, defining the association between the getTimezone method and the /timezoneJSON remote service path, including the above-created parameters, and setting the request to use HTTP GET.

RemoteCallable.Builder callableBuilder = new RemoteCallable.Builder(
		"getTimezone", "/timezoneJSON").setDefaultParameters(
		parameterBuilder.build()).setRequestType(
		new HttpGetRequestType());

Finally, we register this IRemoteCallable and associate it with the ITimezoneService.class service interface

tzServiceRegistration = registerCallables(ITimezoneService.class,
		new IRemoteCallable[] { callableBuilder.build() }, null);

The above registration is all that's needed for the ECF remote service code to construct a proxy implementing ITimezoneService and ITimezoneServiceAsync. When a consumer application actually calls either the getTimezone or getTimezoneAsync methods on the proxy, the following will occur automatically

Construct the appropriate URL for calling the Geonames Timezone service:
```
http://api.geonames.org/timezoneJSON
```
Serialize the parameters passed to the getTimezone method (i.e. 'lat' and 'lng') and add them as URL request parameters, resulting in the URL:
```
http://api.geonames.org/timezoneJSON?lat=<lat param value>&lng=<lng param value>&username=<username default value>
```
Use the resulting URL to issue an HTTP GET request to the Geonames service

When the Geonames service responds to the GET request, it returns a single instance of a JSON Object container several fields with various information associated with the timezone...e.g. countryCode, countryName, lat, lng, timezoneId, dstOffset, etc. The following code sets up a response deserializer, that parses the returned String using a common JSON parser. Then given the data, it creates an instance of the Timezone class and returns it. At remote method call time, when the Geonames service responds this method will be called to deserialize the HTTP response and return a proper Timezone instance to the caller of getTimezone.

		setResponseDeserializer(new IRemoteResponseDeserializer() {
			@Override
			public Object deserializeResponse(String endpoint,
					IRemoteCall call, IRemoteCallable callable,
					@SuppressWarnings("rawtypes") Map responseHeaders,
					byte[] responseBody) throws NotSerializableException {
				try {
					// Convert responseBody to String and parse using org.json
					// lib
					JSONObject jo = new JSONObject(new String(responseBody));
					// Check status for failure. Throws exception if
					// error status
					if (jo.has("status")) {
						JSONObject status = jo.getJSONObject("status");
						throw new JSONException(status.getString("message")
								+ ";code=" + status.getInt("value"));
					}
					// No exception, so get each of the fields from the
					// json object
					String countryCode = jo.getString("countryCode");
					String countryName = jo.getString("countryName");
					double lat = jo.getDouble("lat");
					double lng = jo.getDouble("lng");
					String timezoneId = jo.getString("timezoneId");
					double dstOffset = jo.getDouble("dstOffset");
					double gmtOffset = jo.getDouble("gmtOffset");
					double rawOffset = jo.getDouble("rawOffset");
					String time = jo.getString("time");
					String sunrise = jo.getString("sunrise");
					String sunset = jo.getString("sunset");
					// Now create and return Timezone instance with all the
					// appropriate
					// values of the fields
					return new Timezone(countryCode, countryName, lat, lng,
							timezoneId, dstOffset, gmtOffset, rawOffset,
							dateFormat.parse(time), dateFormat.parse(sunrise),
							dateFormat.parse(sunset));
					// If some json parsing exception (badly formatted json and
					// so on,
					// throw an appropriate exception
				} catch (Exception e) {
					NotSerializableException ex = new NotSerializableException(
							"Problem in response from timezone service endpoint="
									+ endpoint + " status message: "
									+ e.getMessage());
					ex.setStackTrace(e.getStackTrace());
					throw ex;
				}
			}
		});
	}

Note that deserialization code may do whatever is appropriate (e.g. parse xml, parse JSON, deserialize java Objects, use some other serialization technique like protocol buffers, etc) to convert from appropriate remote response (JSON) to an instance of the service interface (Timezone).

The connect implementation above is sufficient for the TimezoneClientContainer to create and execute the Geonames Timezone service request, and handle/deserialize the JSON response, and associate this implementation with the ITimezoneService.getTimezone method and provided parameters. Note that it's also possible to reuse other codebases (other than the RestClientContainer superclass) if the remote service is not REST-based, or requires more control and customization of the remote invocation process.

Registering the Container and Namespace

In the Activator for this Client Provider, there are two OSGi service registrations, which are necessary to hook into the ECF RSA implementation as described above

	public void start(BundleContext bundleContext) throws Exception {
		Activator.context = bundleContext;
		// Register an instance of TimezoneNamespace
		bundleContext.registerService(Namespace.class, new TimezoneNamespace(),
				null);
		// Register an instance of TimezoneContainerTypeDescription (see class
		// below)
		bundleContext.registerService(ContainerTypeDescription.class,
				new TimezoneContainerTypeDescription(), null);
	}
 
	class TimezoneContainerTypeDescription extends ContainerTypeDescription {
		public TimezoneContainerTypeDescription() {
			super(TimezoneClientContainer.CONTAINER_TYPE_NAME,
					new TimezoneClientContainer.Instantiator(),
					"Geonames Timezone Remote Service Client Container");
		}
	}

As described in comments, these two service registrations (Namespace and ContainerTypeDescription) setup the necessary structures for RSA to create the TimezoneClientContainer, call connect and other methods on the container at the appropriate times during remote service discovery, import, proxy creation, and remote service call/invocation.

The entire/complete provider, along with all meta-data are available in the ECF Github repository here.

Testing the Remote Service

Example Consumer/Application Code

To test the above defined Timezone Client Provider, we can use an OSGi ServiceTracker to create some code that uses the ITimezoneService. The Remote Service consumer (application) code is appropriately in a separate bundle from both the Timezone service bundle (which contains the ITimezoneService and ITimezoneServiceAsync interface classes), and the Client Provider bundle defined above. Here is a ServiceTracker.addingService method implementation, where the proxy will be made available when the proxy is created

public ITimezoneServiceAsync addingService(
		ServiceReference<ITimezoneServiceAsync> reference) {
	ITimezoneServiceAsync service = getContext().getService(reference);
	System.out.println("Got ITimezoneServiceAsync");
	// Get completable future and when complete
	service.getTimezoneAsync(47.01, 10.2).whenComplete(
			(result, exception) -> {
				// Check for exception and print out
				if (exception != null) {
					System.out.println(exception.getMessage());
					exception.printStackTrace();
				} else
					// Success!
					System.out.println("Received response:  timezone="+ result);
			});
	// Report
	System.out.println("Returning ITimezoneServiceAsync");
	return service;
}

Once the ITimezoneServiceAsync proxy has been retrieved, the getTimezoneAsync method is called with parameters latitude=47.01 and longitude=10.2. With the whenComplete method and Java8 lambda, we provide the block of code that is executed asynchronously when the remote call completes

	(result, exception) -> {
		// Check for exception and print out
		if (exception != null) {
			System.out.println(exception.getMessage());
			exception.printStackTrace();
		} else
			// Success!
			System.out.println("Received response:  timezone="+ result);
	}

As from the above code, if an exception occurs then the message and stack trace are printed out. If successful, the returned Timezone instance is printed out.

Triggering Discovery

The Remote Service Admin specification standardizes the meta-data to dynamically discover remote services. One way to trigger discovery at consumer runtime is to provide a file in the standardized format called EDEF for Endpoint Description Extender Format. For reference, here is the EDEF for the Timezone remote service

<?xml version="1.0" encoding="UTF-8"?>
<endpoint-descriptions xmlns="http://www.osgi.org/xmlns/rsa/v1.0.0">
  <endpoint-description>
    <property name="ecf.endpoint.id" value-type="String" value="http://api.geonames.org/"/>
    <property name="ecf.endpoint.id.ns" value-type="String" value="ecf.geonames.timezone.namespace" />
    <property name="ecf.endpoint.ts" value-type="Long" value="1387233380373"/>
    <property name="endpoint.framework.uuid" value-type="String" value="20cc5d57-e8f0-0012-192b-c570b422d1f9"/>
    <property name="endpoint.id" value-type="String" value="87d3ef4f-8e8f-4187-873e-166dcc58c9ea"/>
    <property name="endpoint.package.version.org.geonames.timezone" value-type="String" value="1.0.0"/>
    <property name="endpoint.service.id" value-type="Long" value="0"/>
    <property name="ecf.exported.async.interfaces" value-type="String" value="*"/>
    <property name="objectClass" value-type="String">
      <array>
        <value>org.geonames.timezone.ITimezoneService</value>
      </array>
    </property>
    <property name="remote.configs.supported" value-type="String">
      <array>
        <value>ecf.container.client.geonames.timezone</value>
      </array>
    </property>
    <property name="remote.intents.supported" value-type="String">
      <array>
        <value>passByValue</value>
        <value>exactlyOnce</value>
        <value>ordered</value>
      </array>
    </property>
    <property name="service.id" value-type="Long" value="66"/>
    <property name="service.imported" value-type="String" value="true"/>
    <property name="service.imported.configs" value-type="String">
      <array>
        <value>ecf.container.client.geonames.timezone</value>
      </array>
    </property>
  </endpoint-description>
</endpoint-descriptions>

and here is the complete bundle containing this EDEF file.

Running the Consumer/Application

Now we may run this example by adding these four bundles to an OSGi Framework with ECF Remote Services installed

org.geonames.timezone - The Geonames Timezone API
org.eclipse.ecf.provider.geonames.timezone.client - The Timezone Client Provider
org.eclipse.ecf.geonames.timezone.consumer - The consumer/application example code (i.e. ServiceTracker.addingService method implementation)
org.eclipse.ecf.geonames.timezone.consumer.edef - The bundle containing the EDEF file used to trigger discovery

Once started, we can trigger the discovery of the Timezone remote service by simply starting the org.eclipse.ecf.geonames.timezone.consumer.edef bundle (20)

osgi> start 20
Got ITimezoneServiceAsync
Returning ITimezoneServiceAsync
Received response:  timezone=Timezone [countryCode=SO, countryName=Somalia, latitude=10.2, longitude=47.01, timezoneId=Africa/Mogadishu, dstOffset=3.0, gmtOffset=3.0, rawOffset=3.0, time=Tue Dec 09 22:23:00 PST 2014, sunriseTime=Tue Dec 09 05:57:00 PST 2014, sunsetTime=Tue Dec 09 17:30:00 PST 2014]
osgi>

As can be seen from the order of the above lines of output, the 'Returning ITimezoneServiceAsync' is printed prior to the 'Received response: ...' line, indicating that the whenComplete block was executed after executing the remote http get call to the api.geonames.org timezone service.

The complete source and meta-data for all of above are located in the ECF Geonames Github Repository.