SMILA/Project Concepts/Concept for a Performance Measurement Framework

Goals of the Performance Measurement Framework:

Framework for:

• Measurements of Application Metrics like
  • Response time
  • Throughput
  • Resource Utilization
  • Workload
• A totally distributed application
• A heterogeneous Components (Hardware / Operation System)
• Applications are based on Java
• Profiling (Monitoring on instruction level) is not important

Survey

The Concept for the Framework can be divided into two components:

• Measurement Interface/API - Component: Measurements are taken from application\-specific results. Therefore the Applications have to measure the metric itself. The results can be delivered to the Measurement interface/API which can be used from the Data Collection Component.
• Data Collection Component: The Data Collection Component can contact the Measurement Interface/API that is used in parts of the distributed application to collect the results of the measurement. Furthermore this component can analyze/convert the data and can create statistics or graphs.

Survey of techniques for the Measurement Interface/API:

• Logging
  • Every Part of the distributed Application can use logging to save results of the measurement.
    • Advantages: Simple approach
    • Disadvantages: Synchronization of Measurement entries
    • Unsolved: the Collection of the logging files
      • Can be solved with an implementation over JMS /ActiveMQ
• JMX (Java Management Extensions)
  • Framework for Management and Monitoring of Java\-Applications
  • Advantages:
    • Simple publication of application properties (can be measurement results)
      • MBeans ( like Pojo\-Objects with getter/setter)
      • Notifications possible
  • Disadvantages:
    • MBeans Properties has to be polled (probably, otherwise use of notifications)
• ARM (Application Response Measurement)
  • Open standard ([[1]])
  • "for monitoring and diagnosing performance bottlenecks within complex enterprise applications that use loosely\-coupled designs or service\-oriented architectures"
  • Standard contains the API for C and java that allows timing information associated with each step in processing a transaction to be logged to a remote server for later analysis
  • Contains Functions for Performance Counters, Monitoring of Transactions, etc.
  • Implementations:
    • Open Source: open\-arm.sourceforge.net
      • Alpha/Beta' status (version 0.009)
      • Contains for the distribution of results:
        • Logging
        • SNMP
    • ARM\-SDK4 (distributed from the standardization group)
      • Uses the C Lib and JNI for the java API
      • Measurement functions are implement
      • Implementation of the Distribution of measurement results
        • ONLY: logging
    • Commercial: tang\-it.com
      • Commercial!
      • E.g.: Supports the collection/distribution of measurement data to a database server
      • Advantages: best solution for the Performance Measurement Framework
      • Disadvantages: there is no (open source, non-commercial) implementation that supports distributed applications and distributed measurements

nSurvey of techniques for the Data Collection Component:

Most Techniques are developed for the monitoring of J2EE\-Applications/Servers and for profiling Java\-Applications

• JManage (jmanage.org): Client for JMX: runs in a Web server:

Can be used to access applications that use JMX Can create graphs from the returned data

• Commercial Tools:
  • DynaTrace
  • Intrascope

• Eclipse Test & Performance Tools Platform Project ([[2] www.eclipse.org/tptp]) :
• TPTP provides powerful frameworks and services for an open platform upon which developers build unique test and performance tools

• TPTP can be used for profiling of Java\-Applications and JMX\-Application can be managed:

• Advantages:
  • Logging concept : TPTP supports CBE (Common Base Events). It can read logs and can convert them to CBE. CBE can be used for statistical analyze of the data

• • Open Platform: The TPTP\- Framework can be extended
    • Agent Controller specific:
      • E.g. Eclipse Help:
        • TPTP Data Collection Framework / How to write a TPTP Data Collection Agent
        • Extending the monitoring, profiling, and testing functions

• Problems/Disadvantages:
  • JMX can used for managing JMX\-Applications, but JMX information cannot be used to creating graphs/statistical
  • For the "Statistical" Launch Configuration there is no Agent (Agent is a collecting service that can be started in the Agent Controller) in the Agent Controller that can use ARM or JMX - Clients (JMX only for specific JBoss/Jonas, no JSR\-160').
  • And only one agent can be selected in eclipse TPTP( but: according to the documentation it should be possible to use more agents [[3]] )
  • Therefore for the use of TPTP for this Framework there have to be some further development ( a special agent that can address JMX or ARM and simultaneously the windows PerfMon functionality)

• Eclipse Cosmos:

The COSMOS (Community\-driven Systems Management in Open Source) project aims to provide an extensible, standards\-based framework upon which software developers can create specialized, differentiated and inter\-operable offerings of tools for system management.

• • Review: Cosmos is no option. Release V1.0 is Sommer 2008

Review:

• the non-commercial arm - software allows no distributed application (only the logging facility is implemented, with open-arm snmp, probably this is a option!?)
• JMX is used by many applications to allow the management of (runtime) properties, but it isnt often used for measurement functionality (but it can be used for it)

Further Steps:

• Open Issue: How can JMX/ARM integrated into the TPTP-Framework to use the "statisticals" launch configuration (for Graphs)?
  • Own Implementation of an agent for the agent controller (work?)
  • Can this agent support JMX/ARM and Windows Performace Information?
• Open Issue: Alternatives?