Aperi Architecture Document

Ted Slupesky (slupesky@us.ibm.com) and Todd Singleton (toddsing@us.ibm.com)

Latest Update: August 2006

Architecture of Aperi Storage Manager

Introduction

This document describes the architecture of the Aperi Storage Manager, an open source storage management platform owned by a community of storage vendors. The Aperi Storage Manager is based on an initial contribution from IBM of a large portion of the IBM TotalStorage Productivity Center 3.1 product. Leveraging the Eclipse Platform, the Aperi project intends to develop an extensible storage management application framework and an initial set of exemplary, exploiting applications. The framework will include standards-based services for control, discovery, and monitoring of storage resources. Initial candidate exploiters include file system, fabric, tape, and disk management applications.

The platform should be extensible, so that community members can easily add new features, and sufficiently componentized to allow portions of the platform to be used in other products. In addition, it is expected that the Aperi platform will have a commercially-friendly license that will allow community members to develop commercial offerings that combine open source Aperi code with proprietary, closed source functions. This is intended to support the development of an ecosystem similar to Eclipse, in which it is possible to:

Download and use the vanilla Aperi Storage Manager as a complete (if basic) application
If desired, extend the capabilities of Aperi through freely-available (or other) plug-ins
Alternatively, commercial products can be purchased, which contain Aperi as well as code that is proprietary to a vendor
In addition, internal components of Aperi, such as the discovery engine, can be included in other products (open or closed source) that have little to do with the Aperi Storage Manager.

This document describes the desired architecture of the Aperi Storage Manager. It will be seen that IBM's initial contribution needs to evolve in various ways to achieve the vision of the desired architecture. At times specific deviations between IBM's contribution and the desired architecture are mentioned, but this is not exhaustive. Further work can be done to provide specific bridging plans between the initial contribution and the desired architecture.

Technology & Component Model

Java

The Aperi Storage Manager is primarily written in Java with some native code (written in C) as required. The intent is that Aperi can run on either IBM or Sun JREs, although the initial IBM contribution has been tested only with the Sun JRE. Compatibility issues are not anticipated between JREs except in the area of encryption libraries. Specific JRE versions are not identified in this document; however, each release of Aperi will be associated with a particular JRE version (e.g. Java 1.5). In general, Aperi should refrain from upgrading from one JRE version to another whenever possible, because upgrading Java versions can require an upgrade on all agent and GUI machines. This should be avoided whenever possible; decisions to upgrade Java versions should be made carefully.

OSGi (Service Model)

The OSGi/Eclipse service platform provides an open and standard architecture to develop, deploy,and manage services in a coordinated fashion. In the OSGi Service platform, bundles are the entities that are utilized for deploying java-based applications. A bundle is comprised of a set of java classes which can provide functionality to end users and can provide components or services to other bundles in the environment.

The following sections will provide more detail on both the OSGi and/Eclipse component models.

OSGi Bundle Lifecycle

A bundle has a prescribed lifecycle as the following diagram shows:

File:BundleLifecycle.gif

Installed. A bundle is in this state when it is installed in the framework but cannot run. The bundle's code dependencies are not resolved.

Resolved. A bundle is in this state when the framework has successfully resolved the bundle's code dependencies.

Starting. A bundle is in this state when the start method is active. If the start method completes without exception, the bundle has successfully started and will move to the Active state.

Active. A bundle is in this state when it has been successfully started.

Stopping. A bundle is in this state when the stop method is active. When the stop method completes, the bundle is stopped and will move to the "Resolved" state.

Uninstalled. A bundle is in this state after it is uninstalled. The bundle is in an unusable state and all references to the Bundle object should be released immediately.

Eclipse (Lifecycle Management, Extensions)

The Eclipse platform is a universal platform for integrating components. With the release of Eclipse 3.0 the Eclipse architecture and implementation utilizes the OSGi component model as a base for its plug-in architecture. A plug-in represents the smallest unit of function/component that is deployed within the Eclipse platform. In addition, the Eclipse 3.0 plug-in model further extends the base OSGi bundle interfaces with its own features and enhancements. Some of the core features deal with the concepts of "extensions" and "extension points" in addition to an enhanced deployment and installation model.

Plug-ins

A plug-in can be described as a JAR file with a plug-in manifest file named plugin.xml. The plug-in manifest describes the plug-in to the framework and enables a plug-in to consume and/or provide extensions from/to other plug-ins. On the other hand, a bundle is a JAR file with a bundle manifest file named MANIFEST.MF. The bundle manifest describes the bundle to the service framework and enables a bundle to consume or provide packages and services from/to other bundles. Since the Eclipse framework is built on the OSGi Service Framework, each component in the applications can be defined as a plug-in, a bundle, or both depending on your requirements

Relative to Aperi's usage, the relationship of an eclipse plug-in to an OSGi bundle can be represented by whether or not extensions and extension points are defined. If so, plug-in capability is leveraged. If not, it is a pure OSGi bundle.

The MANIFEST.MF file below corresponds to the Myapp application, which exports a service definedin the IMyappService Interface and implemented in the MyappServiceImpl class.

Error creating thumbnail: File missing

The Bundle-Activator header in the bundle manifest file identifies the class to the framework. At startup time, the framework creates an instance of this class and calls its start() method. The Bundle Activator can then publishes services, starts its own threads, and so on. When the bundle shuts down, the framework calls the activator's stop() method. While the bundle shuts down, the Bundle Activator can release resourcesthat are obtained since the start method was called and revoke any services it has published. What follows are some relevant pieces from the MyappPlugin bundle activator class:

public class MyappPlugin extends Plugin {
   // ServiceRegistration
   private ServiceRegistration mServRegistration;

   /**
    * This method is called upon plug-in activation
    */
    public void start(BundleContext context) throws Exception {
       super.start(context);
       // Register the service
       mServRegistration= context.registerService(IMyappService.class.getName(),
                             new  MyappServiceImpl(), null);
    }

   /**
    * This method is called when the plug-in is stopped
    */
    public void stop(BundleContext context) throws Exception {
       super.stop(context);
       // Unregister the service
       mServRegistration.unregister();
    }
}

Fragments

The full capability of a service does not need to be delivered as a plug-in. Additional functionality can be delivered as a plug-in fragment. A plug-in fragment is used to provide additional plug-in functionality to an existing plug-in after it has been installed.

File:Fragment.jpg

Fragments are ideal for shipping features like language or maintenance packs that typically trail the initial products for a few months. Another frequent use of fragments is to deliver OS or windowing system-specific features.

The relationship between the fragment and the Fragment-Host plug-in is defined in the fragment MANIFEST.MF file:

Manifest-Version: 1.0
Bundle-Name: Fragment_A Fragment
Bundle-SymbolicName: fragment_A
Bundle-Version: 1.0.0
Bundle-ClassPath: fragment_A.jar
Fragment-host: pluginHost;bundle-version=[1.0.0,2.0.0)
Bundle-Localization: plugin

When a fragment is detected by the platform and its target plug-in is found, the function in the fragment is "merged" with the original function in the target plug-in. If you query the plug-in registry, you will see the features defined in a fragment as if they were contributed by the original plug-in. Fragments do not contain BundleActivator classes. Fragments are only extensions to a plug-in or bundle, therefore fragments cannot be required or imported by another plug-in or bundle.

Features

Features are groups of plug-ins that define a logical product feature. Lifecycle management in the form of patches and fixes are made at the feature level. In other words, features provide a logical entity for remote deployment and updates of plug-ins.

Extension Points

The process of adding some processing element or elements to a plug-in is known as an extension. An extension is defined by an extender plug-in and causes a host plug-in to modify its behavior. Typically, this modification of behavior includes the addition of processing elements to the host plug-in (e.g., the addition of new menu items to the Eclipse workbench), and the customization of the behavior of these additional elements by services provided by the extender plug-in (e.g., the customization of new menu items by specific menu event handlers).

In the context of a particular extension, a plug-in that stands in the host role provides the extension-point and is extended. In addition to providing services in its own right, such a plug-in also acts as the coordinator and controller of a number of extensions.

As an example, the following plugin.xml declares an extension-point element:

<?xml version=1.0 encoding=UTF-8?>
<?eclipse version=3.0?>
<plugin>
	<extension-point id=org.aperi.sample.myapp
	   name=MyApp
	   schema=schema/org.aperi.sample.myapp.exsd/>
	   <!Other specifications omitted. -- >
</plugin>

In the context of a particular extension, a plug-in that stands in the extender role defines the extension, typically making certain aspects of itself available to a host plug-in through the extension. It also causes the host plug-in to add certain processing elements to its environment. As an example, the following plugin.xml corresponds to a plug-in that consumes the org.aperi.myapp extension.

<?xml version=1.0 encoding=UTF-8?>
<?eclipse version=3.0?>
<plugin>
	<extension
	   point=org.aperi.myapp >
	 </extension>
</plugin>

In a similar fashion as a plug-in, a fragment can consume or provide extensions from/to other plug-ins. These capabilities are defined in the fragment.xml file, which is similar to the plugin.xml for a plug-in, and is part of every fragment. An example of a fragment.xml that corresponds to a fragment that consumes the org.aperi.myapp extension follows:

<?xml version=1.0 encoding=UTF-8?>
<?eclipse version=3.0?>
<fragment>
	<extension
	   point=org.aperi.myapp >
	 </extension>
</fragment>

The act of extension is quite a general concept in Eclipse, and to understand its full generality, it is useful to summarize the types of relationships that may exist between plug-in objects, extension-points, and callback objects.

Multiple extension-points may exist in a host plug-in.
A plug-in may act both as a host plug-in, exposing some extension-points, and as an extender plug-in, extending some plug-ins.
Multiple plug-ins may extend a given extension-point.
A given plug-in may extend a given extension-point multiple times.
An extender plug-in may include different extensions of different host plug-ins.
A single act of extension of an extension-point by a particular extension of a particular plug-in may create multiple callback objects.
A plug-in can define extensions of its own extension-points.

For more information on the Eclipse plug-in architecture, see Notes on the Eclipse Plug-in Architecture (http://www.eclipse.org/articles/Article-Plug-in-architecture/plugin_architecture.html).

High Level Architecture

Conceptual View

The conceptual view of the Aperi architecture consists of two layers:

The Platform layer provides basic storage management functions such as discovery, monitoring, and control of storage assets. It also includes the basic server infrastructure needed to support an enterprise application (logging, threading, transport, container, etc).
The Application layer provides high-level functions, such as filesystem capacity reporting, fabric zone control, and so forth.

Process View

At a high level the Aperi Storage Manager consists of the following processes:

GUI
CLI
Server
Host agent
Database

In addition to the host agent the architecture depends upon SMI-S CIMOMs and SNMP agents to be deployed in the environment. Furthermore, while Aperi includes a rich-client GUI, this is not meant to preclude the development of alternative user interfaces. The components are shown in the image below:

GUI	Renders GUI panels for the user and receives user input
CLI	Provides scriptable command-line interface for Aperi functions
Server	Contains core infrastructure, including discovery, control, and monitoring services; responds to GUI requests; carries out scheduled actions and receives notifications from agents
Database	Repository of all information collected by Aperi
Host agent	Provides detailed information about host filesystems and SAN visibility from the host perspective. May be replaced over time by SMI-S host agents. Responds to requests from the server and generates asynchronous notifications to the server
SMI-S CIMOM	Provides low-level management and monitoring capability for storage devices. Responds to requests from server and generates asynchronous notifications to the server
SNMP Agent	Provides low-level monitoring capability for (some) storage devices and generates asynchronous notifications to the server

The following sequence diagrams illustrate various typical flows of communication among the various components.

The first diagram illustrates the flow during a scheduled discovery:

The server's scheduler kicks off the discovery process
The server queries the agents (Aperi host agents, SMI-S CIMOMs, or SNMP agents) required as part of the discovery
The agents query the devices they represent
The devices return data to the agent
The agents return data to the server
The server stores the data in the database

The next sequence diagram illustrates the flow when an agent sends an event that triggers a re-discovery (for example, a switch may broadcast a "configuration changed" event, and Aperi needs to rediscover the configuration):

The agent sends an event to the server (either because the agent noticed a change or because it received a notification from the device, which is not shown here)
The server stores a record of the event in the database
The server queries the agents (Aperi host agents, SMI-S CIMOMs, or SNMP agents) required as part of the re-discovery
The agents query the devices they represent
The devices return data to the agents
The agents return data to the server
The server stores the data in the database

The next sequence diagram illustrates the flow when a GUI user invokes a configuration operation, such as creating a volume:

The GUI invokes API on the server (in response to user action not shown here)
The server invokes API on the appropriate agent, such as an SMI-S CIMOM
The agent invokes API on the device to execute the change
The device returns data to the agent
The agent returns data to the server
The server stores the data in the database
The server returns data to the GUI

Mapping Aperi Function to the OSGi/Eclipse Component Model

Each major component of the Aperi platform is represented as an OSGi bundle and/or Eclipse plug-in. The components are organized into logical groupings called features (collections of plug-ins). Many of the plug-ins expose packages or services so that other plug-ins may leverage them. Several plug-ins also define extension points to allow exploiting applications (also plug-ins) to extend the underlying function. Notice that these features correspond to the architectural layers identified above.

Server Plug-ins and Features

Feature	Included Plug-ins	Comments
org.eclipse.aperi.runtime	javax.help javax.jms org.apache.derby org.apache.soap org.apache.xerces org.eclipse.aperi.log org.eclipse.equinox.common org.eclipse.equinox.jakarta.common.logging org.eclipes.equinox.jetty org.eclipse.equinox.jetty.http org.eclipse.equinox.registry org.eclipse.equinox.servlet.api org.eclipse.equinox.servlet.bridge.http org.eclipse.osgi org.sblim.cim.client org.sblim.slp org.eclipse.osgi.services org.eclipse.update.scheduler org.eclipse.update.core org.eclipse.update.configurator org.eclipse.aperi.common org.eclipse.aperi.connector	This feature contains the core Eclipse runtime. That includes the base OSGi container, OSGi service definitions, the plug-in framework, and support for plug-in deployment and updates (lifecycle mgmt). This feature also contains the implementation of the OSGi HTTP Service. This service contains the servlet container and the soap implementation needed to support web services.
org.eclipse.aperi.server	org.eclipse.aperi.cbe org.eclipse.aperi.database org.eclipse.aperi.jobs org.eclipse.aperi.discovery org.eclipse.aperi.monitor org.eclipse.aperi.control org.eclipse.aperi.events org.eclipse.aperi.authorization org.eclipse.aperi.config org.eclipse.aperi.alerts	This feature provides the platform layer of the Aperi server. Storage applications are built on top of this feature.
org.eclipse.aperi.app.disk		Contains disk application (server side)
org.eclipse.aperi.app.fabric		Contains fabric application (server side)
org.eclipse.aperi.app.filesystem		Contains filesystem application (server side)
org.eclipse.aperi.app.tape		Contains tape application (server side)

Agent Plug-ins and Features

Feature	Included Plug-ins	Comments
org.eclipse.aperi.runtime	see above	see above
org.eclipse.aperi.agent.fabric		This feature includes the fabric subagent.
org.eclipse.aperi.agent.filesystem		This feature includes the filesystem subagent.

GUI Plug-ins and Features

Feature	Included Plug-ins	Comments
org.eclipse.aperi.runtime	see above	see above
org.eclipse.aperi.gui		This feature contains the gui framework
org.eclipse.aperi.gui.disk		This feature contains the Disk application GUI
org.eclipse.aperi.gui.fabric		This feature contains the Fabric application GUI
org.eclipse.aperi.gui.filesystem		This feature contains the Filesystem application GUI
org.eclipse.aperi.gui.tape		This feature contains the Tape application GUI

Standards

Standards, particularly storage management standards, are critical to Aperi. The Aperi Storage Manager is intended to be an exemplary consumer of the SNIA SMI-S standard for storage device management. Aperi should use SMI-S to the fullest extent possible and should serve as a model to others on how to use SMI-S successfully. SMI-S should be the first and foremost mechanism used by Aperi, but Aperi can also use other standard mechanisms such as SNMP and GS3. Finally, it is also important that Aperi be a competitive storage management application, and it is possible that there is some important management function that cannot be achieved via SMI-S or other standard; in other words, a proprietary mechanism would have to be used. In these circumstances Aperi should strive to get the needed function implemented in a standard, but can consider using the proprietary mechanism as a stopgap.

Finally, Aperi will provide a set of interfaces between its framework and application levels. The Aperi community will work within SNIA to standardize these interfaces as appropriate.

Platform Layer

The base layer includes the underlying OSGi runtime container and a few generic services for managing components (bundles). The base layer does not include any storage-specific function.

OSGi and Eclipse Container

The current base runtime container is based on the Eclipse Equinox 3.2 release. This release implements the OSGi R4 Specification.

Lifecycle Management (Eclipse Updater)

The Eclipse update functions are leveraged to provide the capability to remotely deploy new or update existing features, plug-ins, or fragments. With this capability, the Aperi server, agent, or console can occasionally download updates from an 'update site' (a well known url). This simplifies deployment and patch management.

Servlet Container

The servlet container is the Jetty implementation of the OSGi HTTP Service. This service provides 2.4 Servlet support.

SSL Support

The HTTP Service support http and https protocols. An X509 certificate may be used to provide SSL communication over https.

SOAP & WSDL Support

Apache Axis 2.0 is a candidate for use to provide SOAP and WSDL Support. In the meanwhile, Apache Soap RPC will be used.

The Axis bundle is an OSGi bundlification of a standard Axis server. The standard Axis server is designed to be running in a servlet version 2.4 environment. In addition to the bundlification of the Axis server the axis-osgi bundle offers the ability to export services, registered on the OSGi framework service registry, as SOAP services. The exported services must not expose any data types not supported by SOAP. In order to export a service object as a SOAP service, the only thing needed is to set a property named "SOAP.service.name" on the registered service. The property value should be of type java.lang.String and its value will be used as the name of the exposed SOAP service.

Out of the box, axis supports the serialization and deserialization of a set number of object types. As far as objects are concerned, objects that conform to the java bean paradigm can easily be serialized using Axis. Since some of the objects sent via Aperi are not Java beans, custom serializers will be needed. Axis gives you the ability to write custom serializers/deserializers, and some tools to help make your life easier when you do so. For more on serialization see, http://ws.apache.org/axis/java/user-guide.html#XMLJavaDataMappingInAxis.

Common (org.eclipse.aperi.common plug-in)

This plug-in contains common code that is shared across implementation plug-ins. Three categories of code are included in this plug-in: service interfaces, data objects, utility classes.

Utility Classes

Utility classes represent helper logic to be exploited by the rest of the platform.

Dynamic Plug-in Support

Given the nature of an extensible architectural, plug-ins and dynamic loading/unloading, it is very important to support dynamic plug-ins. Basically, this refers to the fact that plug-ins must be aware of the fact that different portions of the code (other plug-ins) have different life cycles. As a result, when plug-ins are uninstalled, stale references to its classes must be released throughout the entire application. Aperi will take special care to support dynamic plug-ins by encouraging the following design paradigms:

Use stateless objects whenever possible (minimize use of instance variables)
Enforce lifecycle management whenever object caches are used (especially when caching extensions)

Object caches should implement the IDisposable interface.

/**
 * Provides interface to support cleanup of objects when the defining 
 * plug-in is removed from the container
 */
public interface IDisposable {

	/**
	 * Method called to allow the implementing object
	 * to cleanup its instance variables
	 */
	void dispose();
}

Bundle activators (which implement plug-in lifecycle calls) should contain the following logic:

import java.util.Vector;

import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;


public class Activator implements BundleActivator {	
	private static Vector disposables = null;
	
	public Activator(){
		if(disposables == null){
			disposables = new Vector();
		}
	}
	public void start(BundleContext context) throws Exception {
		//...
	}

	public void stop(BundleContext context) throws Exception {
		//...
		if(disposables != null){
			for(int i = 0; i < disposables.size(); i++){
				IDisposable trash = (IDisposable)disposables.get(i);
				trash.dispose();
			}
			disposables = null;
		}
	}
	
	public static boolean addDisposable(IDisposable disposable){
		boolean willManage = false;
		if(disposables != null){
			disposables.add(disposable);
			willManage = true;
		}
		return willManage;
	}
}

AbstractExtensionMgr


import org.eclipse.core.runtime.IRegistryChangeEvent;
import org.eclipse.core.runtime.IRegistryChangeListener;


/**
 * This class should be used to manage extensions.  For 
 * every custom extension point, a subclass of this abstract
 * class should be used to manage a cache of implementing 
 * extensions.  The extension point MUST conform to a particular
 * extension point schema:
 * <element name="[ELEMENT_NAME]">
      <complexType>
         <attribute name="impl" type="string">
            <annotation>
               <appInfo>
                  <meta.attribute kind="java" basedOn="[CLASS OR INTERFACE NAME]"/>
               </appInfo>
            </annotation>
         </attribute>
      </complexType>
   </element>
   The "impl" attribute must contains the java implementation of a particular
   interface or parent.  An instance of this impl is returned to the client
   of this subclass.  
 */
public abstract class AbstractExtensionMgr implements IRegistryChangeListener, IDisposable{
	/**
	 * @return The extension point id as found in the extension
	 * registry.  This represents the extension point to be 
	 * managed by this instance.
	 */
	public abstract String getExtensionPointId();
	
	/**
	 * This method should be called when attempting obtain a reference
	 * to an extension implementation.  Extensions will not be loaded until 
	 * this method is called.  This method will first check the cache; if 
	 * the extension is not found in the cache, the Extension Registry 
	 * will be searched.  Once found, a reference will be placed in the cache.
	 * 
	 * If the extension is removed from the runtime environment (extending
	 * plug-in is removed), the cache will be updated to reflect the 
	 * removal.  
	 * @param extensionId
	 * @return
	 */
	public Object getExtensionImpl(String extensionId){
		...
	}

	/**
	 * Called by the container when an extension is changed in the 
	 * registry.  This object is interested only in extensions
	 * that implement the extension point id found in 
	 * getExtensionPointId().  It is also only interested in 
	 * removal events.  
	 * @param event - the change to the registry.
	 */
	public void registryChanged(IRegistryChangeEvent event) {	
	}
        
        public dispose(){
             //... cleanup cache of extensions
        }

}

Below is an example of an extension point manager that makes use of the abstract helper above:

public class ExampleExtensionMgr extends AbstractExtensionMgr {
	private static final String EXTENSION_POINT_ID = "aperi.example";
	private static ExampleExtensionMgr mgr = null;
	
	private ExampleExtensionMgr(){
		super();
		Activator.addDisposable(this);
	}
	
	public ExampleExtensionMgr getInstance(){
		if(mgr == null){
			mgr = new ExampleExtensionMgr();
		}
		return mgr;
	}
	
	public String getExtensionPointId() {
		return EXTENSION_POINT_ID;
	}

	public ExampleExtension getExampleExtension(String extensionId){
                ExampleExtension ext = null;
		Object obj = super.getExtensionImpl(extensionId);
                if(obj != null){
                        ext = (ExamplExtension)obj;
                }
                return ext;
	}
}

Database (org.eclipse.aperi.database plug-in)

The database is an essential component of the overall architecture. In addition to being a respository of information, it is also an essential data transfer mechanism between components. For example, a component can make an entry for a task that needs to be scheduled in the appropriate table; the Scheduler service will then execute the task.

RDBMS

Aperi is designed to support multiple Relational Database Management Systems (RDBMS). The 'vanilla' Aperi offering works with the Derby open source RDBMS. Other RDBMS may work but have not been tested. Although by design Aperi does not include RDBMS vendor-specific code, it is likely that adding new RDBMS will require some modifications to SQL queries. We expect that commercial adopters of Aperi would likely bundle commercial RDBMS with their products.

Schema

The schema can be understood conceptually has having three parts:

Tables that model the entites available on the storage network. As examples, key tables used by the Filesystem component are T_RES_COMPUTER and T_RES_FILESYSTEM, which contain records for each discovered computer (host) and filesystem; the Disk component relies on T_RES_STORAGE_SUBSYSTEM and T_RES_STORAGE_VOLUME; the Fabric component makes heavy use of T_RES_FABRIC, T_RES_PORT, T_RES_SWITCH, and T_RES_ZONE.
Tables that contain information internal to Aperi. Examples include T_SCHEDULE and T_RUNS, which track schedule entries for jobs and the results of scheduled runs, respectively.
Indexes and views over the tables

It should be noted that there is not necessarily a strict containment relationship between a given architectural component and a given set of tables. Both the Fabric and Disk components, for example, can update the T_RES_STORAGE_SUBSYSTEM table, since both components can discover storage subsystems (using different means). In most cases, however, it is appropriate to think of a given table as "belonging" to a given functional component.

The schema uses vendor-neutral data definition language (DDL). Specific RDBMS vendor extensions are not used in the DDL.

In many cases, an entity on the storage network has a 'natural' key that would be modeled in SQL as a variable length string (that is, something that might be derived from some combinations of names and serial numbers). For efficiency of database access, the schema provides an autogenerated integer key for such entities, where the necessary information required to generate a user-visible string name of the entity is also available in the database. For example, the user interface may refer to a given subsystem as "IBM-ESS-2105-123456", while the primary key in T_RES_STORAGE_SUBSYSTEM corresponding to that subsystem is an arbitrary integer.

In addition, certain other frequently used string enumeration values (such as 'Vendor' fields) are represented in database keys as numeric values, with translation to strings available as needed.

Database Interface

The Data and Device Servers contain a Database Interface component, which itself consists of three parts.

First, the Database Interface maintains a pool of database connections that are leased by other components as needed.

Second, the 'Helpers' are automatically generated Java code that provide generic access to tables via an object-oriented interface. In other words there is conceptually one Helper per table, with methods corresponding to the fundamental SQL operations of insert, update, delete, and retrieve.

Finally, the 'Mappers' provide a fundamental insulation layer between discovery code and the schema itself. There is, generally speaking, one Mapper for each kind of discoverable entity; the mapper takes as input the data returned by the discovery layer for that kind of component, and efficiently converts that data to the form required by the schema.

The intent of the Helpers and Mappers is minimize the coupling between the details of the schema and the rest of the system. Helpers provide a generic mechanism for general database access; Mappers provide special access for the performance-critical discovery code while still insulating the discovery code itself from the details of the schema. As a general rule, code is expected to use a Helper or Mapper for database access; however, when appropriate, a direct JDBC query can be developed and used for a particular purpose. This would normally be done, say, for a report or some other part of the user interface that has specific characteristics that make it unsuitable for generalization, and for which the generic access provide by the appropriate Helpers does not meet performance requirements. In general it is preferable to create a view for these cases, but some use of 'direct JDBC' does occur.

Jobs & Scheduler (org.eclipse.aperi.jobs plug-in)

The Aperi Job scheduler provides the ability run 'jobs' on a schedule; these jobs are primarily data collection jobs, but can also be long-running control operations. The Scheduler can distribute work among many agents when appropriate. The primary interaction mechanism with the job scheduler is the database itself: clients insert rows describing work to be done, and retrieve status of those jobs from the database as well.

Discovery (org.eclipse.aperi.discovery plug-in)

The Discovery service provides a framework for discovery of entities on a storage network. The Discovery service does not itself prescribe the mechanisms by which discovery is performed: instead, it provides extension points for 'scanners' (that initiate discovery requests) and 'parsers' (that handle the returned data). Applications are expected to extend the Discovery service in appropriate ways. For example, the Disk application adds discovery of storage subsystems using SMI-S; the Fabric application adds discovery of SAN topology information and zone configuration via SNMP and GS3.

Monitoring (org.eclipse.aperi.monitor plug-in)

The Monitoring service receives external notifications such as SNMP traps, GS3 events, and CIM indications. It includes a damping mechanism to resolve an 'event storm', as when many notifications are received that pertain to the same underlying condition. The Monitoring service translates from received notifications (after any damping) to the Aperi Event service: that is, received notifications are posted as events.

Control (org.eclipse.aperi.control plug-in)

The Control service provides a framework for invoking control (change) operations either synchronously or asynchronously. Applications are expected to extend this service: for example, the Disk application adds a "create volume" control operation.

Events (org.eclipse.aperi.events plug-in)

The Event service implements a pub/sub JMS bus that is used for loosely-coupled communication among components. Events can be subscribed to by supplying the name of an Event class and an id for a class that implements MessageListener. Events can be received via onMessage() and sent via createMessage(), as specified by JMS.

Authorization (org.eclipse.aperi.authorization plug-in)

The Authorization service provides a role-based gatekeeper service. All external Aperi APIs are associated with a required role. Users must possess this role in order to invoke the API. The Authorization service determines whether a given user has the authority to invoke a given operation.

Configuration (org.eclipse.aperi.config plug-in)

The Configuration service implements a simple set and get interface for storing and retrieving configuration values, such as log or trace levels, port numbers, and so forth. The Configuration service has no special knowledge of what these parameters are, with one exception: JDBC connectivity information is treated specially, in that it is stored in a .properties file. Other parameters are stored in the database itself. In addition the Configuration service has the ability to encrypt passwords before storing them, and decrypt them on retrieval. This prevents Aperi from storing plain-text passwords in repositories.

Alerts (org.eclipse.aperi.alerts plug-in)

The Alert service is general-purpose mechanism for generating user-visible alerts and delivering them via a variety of mechanisms. A number of mechanisms (e.g., email and SNMP) are included in the Aperi base, and in addition the Alert service provides an extension point so that new event destinations can be added by plug-ins.

Application Layer

Disk Application

The Disk application provides functions that enable discovery, monitoring, and control of storage subsystems (such as disk arrays and block virtualization engines), and asset reporting, through SMI-S. The Disk application extends the Discovery engine to provide SMI-S based discovery of these devices, and extends the Monitoring engine to receive SMI-S indications from the discovered CIMOMs. It also extends the Control service with a limited number of control operations, specifically:

Create volume
Assign volume to host
Unassign volume from host
Delete volume

The Disk application does not involve the host agent as it relies completely on SMI-S CIMOMs as agents. It does have a server-side component for extending the Discovery, Control, and Monitoring services as described above, and provides a Web Services-based API for those functions as well. In addition, a set of GUI panels that correspond to these functions are also provided.

The Disk application introduces a set of tables that are concerned with maintaining information about the discovered storage subsystems, volumes, and other objects. Some of these tables are shared with Fabric (see below) as, for example, storage subsystems can be discovered by both applications.

Fabric Application

The Fabric application provides functions that enable discovery, monitoring, and control of storage area networks, and asset reporting, through a variety of mechanisms. The Fabric application extends the Discovery engine to provide discovery of SANs, their topology, and zone configurations through SNMP and inband GS3 mechanisms; in the future SAN discovery through SMI-S will be added. In addition extends the Monitoring engine to receive SNMP traps (and in the future SMI-S indications) from the discovered objects. In addition it extends the Control service with a number of control operations relating to creating and updating the zone configuration.

The Fabric application provides a host-based subagent to perform the inband GS3 discovery and control operations; in addition it provides a server-based component to perform equivalent discovery via SNMP. The host agent is not required for fabric function except for zone control operations; zone control is not possible via SNMP. Complete fabric SMI-S support will allow full discovery and zone control without relying on the host-based agent. Note that the current implementation of SNMP support is primarily aimed at the standard fabric MIB, but does include some vendor-specific (but published) calls to handle Cisco Virtual SAN support. Again, full fabric SMI-S support will allow us to sunset this vendor-specific SNMP code. Note also that the Brocade API is not included in Aperi, so zone control of Brocade switches is not possible until fabric SMI-S support is available.

In addition the Fabric application provides a server-side component for extending the Discovery, Control, and Monitoring services as described above, and provides a Web Services-based API for those functions as well. In addition, a set of GUI panels that correspond to these functions are also provided.

The Fabric application introduces a set of tables that are concerned with maintaining information about the discovered storage subsystems, volumes, and other objects. Some of these tables are shared with Disk (see below) as, for example, storage subsystems can be discovered by both applications.

Filesystem Application

The primary functions provided by the Filesystem application are the discovery and collection of capacity information regarding host filesystems.

The Filesystem application completely depends on the host-based Filesystem subagent to perform discovery and data collection. In the future it might be possible to migrate away from this host agent and toward SMI-S based host agents, although the current level of function provided by the Filesystem application is probably beyond what SMI-S based agents provide today.

In addition to the host subagent, the Filesystem application provides a server component for implementing its data collection and GUI panels that provide its functions and reports as well. For legacy reasons, the Filesystem application does not use the Discovery, Control, and Monitoring engine described above. No control operations are performed.

Finally, the Filesystem application manages a significant portion of schema that is unique to it.

Tape Application

The Tape application is very similar to the Disk application, except that its functionality is specifically tied to tape libraries. It introduces new components analogous to the Disk components, but targeted for tape; also, it does not (yet) provide control functions.

User Interface Layer

Graphical Console

The initial contribution includes a GUI that is a rich client, Java Swing-based user interface. We are beginning development of an Eclipse-based GUI alternative; and certainly other, perhaps web-based, alternatives could be developed by the community.

Initial Contribution GUI – External Design

The TPC console framework presents a modified Explorer view, in which most navigation activity is performed through the tree control, and most other interaction is perfomed on the right-hand side "work area."

Items in the tree control are grouped under several top-level navigation nodes. The first node, 'Administrative Services', governs items related to the configuration of TPC itself. The second node is currently labeled TotalStorage Productivity Center, and governs functions that span the functional divisions of the product, such as the topology viewer. This node will need to be relabeled as part of the contribution to Aperi, perhaps to a name like "General" or "Core", or even to "Aperi".

The remaining top-level items correspond to the major functional divisions outlined above, such as Disk, Tape, and Fabric (referred to as the 'Disk Manager', etc.). Note that the commercial TPC product contains a node labeled 'Data Manager' that contains, among other things, the functions referred to as 'Filesystem' here.

Initial Contribution GUI – Internal Design

The GUI is architected to be relatively lightweight. The only process it connects to is the Data Server (see below); it does not connect directly to the RDBMS or to agents. This is to enforce the responsibility of the Data Server to perform the more complex aggregation and analysis that is implied by agent or RDBMS access, and prevent this load from be shifted onto end-user workstations. In addition, allowing the GUI to connect only to a single server process minimizes the potential impact to firewall configuration at customer sites. It uses a private protocol to communicate with the Data Server; a future work item could include redesigning this protocol, although it is possible to host alternative GUIs (such as web-based GUIs) using the protocol as it is.

The GUI uses JavaHelp for its online help framework. The TPC GUI also uses a licensed third-party component for charting; this component will be replaced in the contribution to Aperi with the Eclipse Chart Engine (from BIRT).

The TPC 3.1 GUI offers an extensibility mechanism that interacts closely with its licensing enforcement scheme. Since licensing is not appropriate for open source products, the TPC licensing scheme will be removed prior to contribution.

Eclipse GUI

We are beginning development of alternative GUI for Aperi that uses the Eclipse framework. The initial effort in this area is focused on 'porting' the existing topology viewer to Eclipse, and using Eclipse BIRT as a report designer and viewer. Thus, many existing Aperi reports will be refactored to use BIRT.

Followon work will address the proper Eclipse representation of other functions of Aperi. Our initial thinking is that major functional areas should be organized as distinct Eclipse perspectives, perhaps:

Topology
Reporting
Data Collection
Server Administration
Alerting

Much work remains to be done to flesh this out. Naturally, the resulting GUI will be extensible in the normal Eclipse manner.

Topology

Aperi includes a graphical topology viewer, written using Swing components, that is specifically designed to redner large storage area networks efficiently. This component has extensive 'special knowledge' of how to render SANs and also makes use of specially-written database queries for efficiency (in other words, it does not go through database 'helpers'). Because of this special knowledge, the topology viewer is not a general-purpose topology viewing component that could be easily included in other applications, although it may well have portions that could be adapted to such purposes.

Command Line Interface

The Aperi Storage Manager should have a command line interface (CLI) program that provides scriptable access to the full functionality of the platform as exposed in the GUI. This CLI is not included in the initial contribution. It can be expected to be a simple Web Services client that invokes Aperi APIs, possibly remotely (it should not be required to run on the server host).

Web-based Console

The presence of the GUI layer in the Aperi platform should not be understood as precluding development of alternative user interfaces, such as a web-based console for Aperi functions. All the functions available in the "fat" Aperi GUI are conceivably able to be implemented in a web-based form, although the GUI layer itself is unlikely to provide much leverage in developing alternative forms. However, the remaining layers of the architecture should be usable by any user interface, and in fact IBM has prototyped web-based access to the TPC 3.1 application, which is the foundation of the Aperi platform.

Other Areas

Security

A general description of security within Aperi is forthcoming.

Installation & Patch Management

Aperi does not include an installer in the traditional sense. At the time of initial contribution, Aperi is distributed as a zip file. After unzipping, the 'cfgaperi' script is run, which configures the system to run properly.

Commercial distributors are expected to develop their own installers, probably using tools like InstallShield.

It is an open question whether Aperi, as an open source project, needs a full-featured graphical installer or if the cfgaperi script is sufficient.

Platforms

The primary initial development platforms for the Aperi Storage Manager are Windows and Linux. IBM's TPC provides support for these two platforms as well as AIX (for the server) and many others (for the GUI and host agent). It is not anticipated that any Aperi development work would create a technical limitation to supporting the full set of TPC platforms, or to adding additional platforms. However, at the time of initial contribution, build and test are performed only on those two platforms.

Certainly, the Java code that comprises the majority of Aperi is expected to be easily portable to other platforms. There is a fair amount of C code as well that presents a porting challenge, although in all cases the code has already been ported to at least Windows, Linux, and AIX. This should ease the potential burden of porting to new platforms.

Breadcrumbs

Notice: This Wiki is now read only and edits are no longer possible. Please see: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/wikis/Wiki-shutdown-plan for the plan.