UGAccessingDerivedInformation

Accessing Derived Information

Configuring Active Colletions

An Active Collection is similar to a standard collection, but with the ability to report change notifications when items are inserted, updated or removed. The other main difference is that they cannot be directly updated - their contents is managed by an Active Collection Source which acts as an adapter between the collection and the originating source of the information.

This section will explain how to define an Active Collection Source and register it to indirectly create an Active Collection.

Defining the Source

The source can be defined as an object model or specified as a JSON representation for packaging in a suitable form, and subsequently de-serialized when deployed to the runtime governance server.

The following is an example of the JSON representation that defines a list of Active Collection Sources - so more than one source can be specified with a single configuration:

[
  {
    "@class" : "org.overlord.rtgov.active.collection.epn.EPNActiveCollectionSource",
    "name" : "ServiceResponseTimes",
    "type" : "List",
    "itemExpiration" : 0,
    "maxItems" : 100,
    "subject" : "ServiceResponseTimes",                           // Attribute specific to the source implementation
    "aggregationDuration" : 1000,
    "groupBy" : "serviceType + \":\" + operation + \":\" + fault",
    "aggregationScript" : "AggregateServiceResponseTime.mvel"
  },{
    "@class" : "org.overlord.rtgov.active.collection.epn.EPNActiveCollectionSource",
    "name" : "ServiceDefinitions",
    "type" : "Map",
    "itemExpiration" : 0,
    "maxItems" : 100,
    "subject" : "ServiceDefinitions",
    "scheduledScript" : "TidyServiceDefinitions.mvel",
    "scheduledInterval" : 60000,
    "properties" : {
        "maxSnapshots" : 5
    },
    "maintenanceScript" : "MaintainServiceDefinitions.mvel"
  },{
    "@class" : "org.overlord.rtgov.active.collection.epn.EPNActiveCollectionSource",
    "name" : "Situations",
    "type" : "List",
    "itemExpiration" : 40000,
    "maxItems" : 0,
    "subject" : "Situations",                                    // Attribute specific to the source implementation
    "activeChangeListeners" : [ {
        "@class" : "org.overlord.rtgov.active.collection.jmx.JMXNotifier",
        "objectName" : "overlord.rtgov:name=Situations",
        "descriptionScript" : "SituationDescription.mvel",
        "insertTypeScript" : "SituationType.mvel"
    } ]
  },{
    "@class" : "org.overlord.rtgov.active.collection.ActiveCollectionSource",
    "name" : "Principals",
    "type" : "Map",
    "lazy" : true,
    "visibility" : "Private",
    "factory" : {
        "@class" : "org.overlord.rtgov.active.collection.infinispan.InfinispanActiveCollectionFactory",
        "cache" : "Principals"
    }
  }
]

This configuration shows the definition of two Active Collection Sources. The top level elements for a source, that are common to all active collection sources, are:

Field	Description
@class	This attribute defines the Java class implementing the Active Collection Source. This class must be directly or indirectly derived from org.overlord.rtgov.active.collection.ActiveCollectionSource.
name	The name of the source and also associated Active Collection.
type	The type of active collection. The currently supported values (as defined in the org.overlord.rtgov.active.collection.ActiveCollectionType enum are: List (default) Map
visibility	The visibility of active collection, i.e. whether accessible via the remote access mechanisms such as REST. The currently supported values (as defined in the org.overlord.rtgov.active.collection.ActiveCollectionVisibility enum are: Public (default) Private
lazy	Whether active collection should be created on startup, or lazily instantiated upon first use. The default is false.
itemExpiration	If not zero, then defines the number of milliseconds until an item in the collection should expire (i.e. be removed).
maxItems	If not zero, defines the maximum number of items that the collection should hold. If an insertion causes the size of the collection to increase above this value, then the oldest item should be removed.
aggregationDuration	The duration (in milliseconds) over which the information will be aggregated.
groupBy	An expression defining the key to be used to categorize the information being aggregated. The expression can use properties associated with the information being aggregated.
aggregationScript	The MVEL script to be used to aggregated the information. An example will be shown in a following sub-section.
scheduledInterval	The interval (in milliseconds) between the invocation of the scheduled script.
scheduledScript	The MVEL script invoked at a fixed interval to perform routine tasks on the collection.
maintenanceScript	By default, events received by the active collection source will be inserted into the associated active collection. If a MVEL maintenance script is specified, then it will be invoked to manage the way in which the received information will be applied to the active collection.
properties	A set of properties that can be access by the various scripts.
activeChangeListeners	The list of active change listeners that should be instantiated and automatically registered with the Active Collection. The listeners must be derived from the Java class org.overlord.rtgov.active.collection.AbstractActiveChangeListener.
factory	The optional factory for creating the active collection, derived from the class org.overlord.rtgov.active.collection.ActiveCollectionFactory.

The additional attributes associated with the EPNActiveCollectionSource implementation will be discussed in a later section.

Scripts

The aggregation script is used to (as the name suggests) aggregate information being provided by the source, before being applied to the collection. The values available to the MVEL script are:

Variable	Description
events	The list of events to be aggregated.

The scheduled script is used to perform regular tasks on the active collection, independent of any information being applied to the collection. The values available to the MVEL script are:

Variable	Description
acs	The active collection source.
acs.properties	The properties configured for the active collection source.
variables	A map associated with the active collection source that can be used by the scripts to cache information.

The maintenance script is used to manage how new information presented to the source is applied to the active collection. If no script is defined, then the information will be inserted by default. The values available to the MVEL script are:

Variable	Description
acs	The active collection source.
acs.properties	The properties configured for the active collection source.
key	The key for the information being inserted. May be null.
value	The value for the information being inserted.
variables	A map associated with the active collection source that can be used by the scripts to cache information.

An example script, showing how these variables can be used is:

int maxSnapshots=acs.properties.get("maxSnapshots");

snapshots = variables.get("snapshots");

if (snapshots == null) {
	snapshots = new java.util.ArrayList();
	variables.put("snapshots", snapshots);
}

// Update the current snapshot
currentSnapshot = variables.get("currentSnapshot");

if (currentSnapshot == null) {
	currentSnapshot = new java.util.HashMap();
}

snapshots.add(new java.util.HashMap(currentSnapshot));

currentSnapshot.clear();

// Remove any snapshots above the number configured
while (snapshots.size() > maxSnapshots) {
	snapshot = snapshots.remove(0);
}

// Merge snapshots
merged = org.overlord.rtgov.analytics.service.util.ServiceDefinitionUtil.mergeSnapshots(snapshots);

// Update existing, and remove definitions no longer relevant
foreach (entry : acs.activeCollection) {
	org.overlord.rtgov.analytics.service.ServiceDefinition sd=null;

	if (merged.containsKey(entry.key)) {
		acs.update(entry.key, merged.get(entry.key));
	} else {
		acs.remove(entry.key, entry.value);
	}

	merged.remove(entry.key);
}

// Add new definitions
for (key : merged.keySet()) {
	acs.insert(key, merged.get(key));
}

This example shows the script accessing the Active Collection Source and its properties, as well as accessing (and updating) the 'variables' cache associated with the source.

Active Change Listeners

The activeChangeListeners element defines a list of Active Change Listener implementations that will be instantiated and registered with the active collection.

The fields associated with this component are:

Field	Description
@class	The Java class that provides the listener implementation and is directly or indirectly derived from org.overlord.rtgov.active.collection.AbstractActiveChangeListener.

The remaining attributes in the example above will be discussed in a subsequent section related to reporting results via JMX notifications.

Factory

The factory element defines an Active Collection Factory implementation that will be used to create the active collection.

The fields associated with this component are:

Field	Description
@class	The Java class that provides the factory implementation and is directly or indirectly derived from org.overlord.rtgov.active.collection.ActiveCollectionFactory.

The current list of factory implementations are defined below.

Infinispan

The fields associated with the org.overlord.rtgov.active.collection.infinispan.InfinispanActiveCollectionFactory component are:

Field	Description
cache	The name of the cache to be presented as an Active Map.
container	The optional JNDI name used to obtain the cache container. If not defined, then the default container will be obtained from the 'infinispan.container' property from overlord-rtgov.properties file in the $JBOSS_HOME/standalone/configuration folder. If the default container is not defined, then cache details will be obtained from the rtgov_infinispan.xml file contained within the overlord-rtgov.war.

Registering the Source

JEE Container

The Active Collection Source is deployed within the JEE container as a WAR file with the following structure:

warfile
|
|-META-INF
|    |- beans.xml
|
|-WEB-INF
|    |-classes
|    |    |-acs.json
|    |    |-<custom classes/resources>
|    |
|    |-lib
|       |-acs-loader-jee.jar
|       |-<additional libraries>

The acs.json file contains the JSON representation of the Active Collection Source configuration.

The acs-loader-jee.jar acts as a bootstrapper to load and register the Active Collection Source.

If custom active collection source and/or active change listeners are defined, then the associated classes and resources can be defined in the WEB-INF/classes folder or within additional libraries located in the WEB-INF/lib folder.

A maven pom.xml that will create this structure is:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  	<modelVersion>4.0.0</modelVersion>
	<groupId>....</groupId>
	<artifactId>....</artifactId>
	<version>....</version>
	<packaging>war</packaging>
	<name>....</name>

	<properties>
		<rtgov.version>....</rtgov.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.overlord.rtgov.active-queries</groupId>
			<artifactId>active-collection</artifactId>
			<version>${rtgov.version}</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>org.overlord.rtgov.active-queries</groupId>
			<artifactId>acs-loader-jee</artifactId>
			<version>${rtgov.version}</version>
		</dependency>

		<!-- Other dependencies relevant to the Active Collection Source or Active Change Listener implementations -->

		<dependency>
			<groupId>javax</groupId>
			<artifactId>javaee-api</artifactId>
			<version>6.0</version>
			<scope>provided</scope>
		</dependency>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>${junit.version}</version>
			<scope>test</scope>
		</dependency>
	</dependencies>

</project>

If deploying in JBoss Application Server, then the following fragment also needs to be included, to define the dependency on the core Overlord rtgov modules:

.....
	<build>
		<finalName>....</finalName>
		<plugins>
			<plugin>
				<artifactId>maven-war-plugin</artifactId>
				<configuration>
					<failOnMissingWebXml>false</failOnMissingWebXml>
 					<archive>
						<manifestEntries>
							<Dependencies>org.overlord.rtgov</Dependencies>
						</manifestEntries>
					</archive>
				</configuration>
			</plugin>
		</plugins>
	</build>
	.....

Presenting Results from an Event Processor Network

As discussed in the preceding section, an Active Collection Source can be configured to obtain information from an Event Processor Network, which is then placed in the associated Active Collection. This section will explain in more detail how this can be done using the specific Active Collection Source implementation.

[
  {
    "@class" : "org.overlord.rtgov.active.collection.epn.EPNActiveCollectionSource",
    "name" : "Situations",
    "type" : "List",
    "itemExpiration" : 40000,
    "maxItems" : 0,
    "subject" : "Situations",
    "activeChangeListeners" : [ {
        "@class" : "org.overlord.rtgov.active.collection.jmx.JMXNotifier",
        "objectName" : "overlord.rtgov:name=Situations",
        "descriptionScript" : "SituationDescription.mvel",
        "insertTypeScript" : "SituationType.mvel"
    } ]
  }
]

This configuration shows an example of an Active Collection Source using the org.overlord.rtgov.active.collection.epn.EPNActiveCollectionSource implementation. The additional fields associated with this implementation are:

Field	Description
subject	The EPN subject upon which the information has been published.

An example Event Processor Network configuration that will publish information on the subject (e.g. 'Situations') specified in the Active Collection Source configuration above is:

{
  "name" : "SLAMonitorEPN",
  "subscriptions" : [ {
    "nodeName" : "SLAViolations",
    "subject" : "ServiceResponseTimes"
  } ],
  "nodes" : [
    {
      "name" : "SLAViolations",
      "sourceNodes" : [ ],
      "destinationSubjects" : [ ],
      "maxRetries" : 3,
      "retryInterval" : 0,
      "eventProcessor" : {
        "@class" : "org.overlord.rtgov.ep.drools.DroolsEventProcessor",
        "ruleName" : "SLAViolation"
      },
      "predicate" : null,
      "notifications" : [ {
      	 "type" : "Processed",
      	 "subject" : "SituationsProcessed"
      },{
      	 "type" : "Results",
      	 "subject" : "Situations"
      } ]
    }
  ],
  "version" : "1"
}

Publishing Active Collection Contents as JMX Notifications

[
  .....
 {
   .....
   "activeChangeListeners" : [ {
     "@class" : "org.overlord.rtgov.active.collection.jmx.JMXNotifier",
     "objectName" : "overlord.sample.slamonitor:name=SLAViolations",
     "insertType" : "SLAViolation"
   } ],
   .....
 }
]

This configuration shows the use of the JMXNotifier active change listener implementation. This implementation has the following additional fields:

Field	Description
objectName	The MBean (JMX) object name to be used to report the notification.
descriptionScript	The MVEL script that can be used to derive the 'description' field on the notification. If not defined, then the information’s 'toString()' value will be used.
insertType	The 'type' field for the notification when performing an insert.
insertTypeScript	An optional MVEL script that can be used to derive the 'type' field for an insert.
updateType	The optional 'type' field for the notification when performing an update.
updateTypeScript	An optional MVEL script that can be used to derive the 'type' field for an update.
removeType	The optional 'type' field for the notification when performing a removal.
removeTypeScript	An optional MVEL script that can be used to derive the 'type' field for a remove.

The following JConsole snapshot shows this JMXNotifier in action, reporting SLA violations from the associated active collection:

Querying Active Collections via REST

The Active Collections configured within the runtime governance server can be accessed via a REST service, by POSTing a query specification to the URL: <host>/overlord-rtgov/acm/query

The Query Specification is comprised of the following information:

Attribute	Description
collection	The active collection name.
predicate	Optional. If defined with the parent name, then can be used to derive a child collection that filters its parent’s content (and notifications) based on the predicate.
parent	Optional. If deriving a child collection, this field defines the parent active collection from which it will be derived.
maxItems	Defines the maximum number of items that should be returned in the result, or 0 if unrestricted.
truncate	If a maximum number of items is specified, then this field can be used to indicate whether the 'Start' or 'End' of the collection should be truncated.
style	Allows control over how the results are returned. The value 'Normal' means as it appears in the collection. The value 'Reversed' means the order of the contents should be reversed.

The collection field defines the name of the collection - either an existing collection name, or if defining the 'predicate' and 'parent' fields, then this field defines the name of the derived collection to be created.

The predicate field refers to a component that implements a predicate interface - the implementation is defined based on the 'type' field. Currently only a MVEL based implementation exists, with a single field 'expression' defining the predicate as a string.

For example,

{
    "parent" : "ServiceResponseTime",
    "maxItems" : 5000,
    "collection" : "OrderService",
    "predicate" : {
        "type" : "MVEL",
        "expression" : "serviceType == \"{urn:switchyard-quickstart-demo:orders:0.1.0}OrderService\" && operation == \"submitOrder\""
    },
    "truncate" : "End",
    "style" : "Reversed"
}

If the Active Collection Manager (ACM) does not have a collection named 'OrderService', then it will use the supplied defaults to create the derived collection. If the collection already exists, then the contents will simply be returned, allowing multiple users to share the same collection.

Pre-Defined Active Collections

This section describes the list of Active Collections that are provided "out of the box".

ServiceResponseTimes

This active collection is a list of org.overlord.rtgov.analytics.service.ResponseTime objects.

The response times represent an aggregation of the metrics for a particular service, operation and response/fault, over a configured period. The ResponseTime object has the following fields:

• service type

• operation

• fault - the optional fault name

• timestamp

• average - average response time (where response time represents multiple invocations)

• min - minimum response time (where response time represents multiple invocations)

• max - maximum response time (where response time represents multiple invocations)

Situations

This active collection is a list of Situation objects.

The Situation object represents a 'situation of interest' that has been detected within the Event Processor Network, and needs to be highlighted to end users. The Situation object has the following fields:

• type - the type of situation

• subject - the identity for the entity related to the situation (e.g. service)

• description - a free format textual description of the situation

• timestamp - when the situation occurred

• severity - "Low", "Medium", "High" or "Critical"

• activity references - optional references to activity events that resulted in the situation

This active collection configuration also publishes it contents via a JMX notifier, based on the following configuration details:

[
  {
    ........
  },{
    "@class" : "org.overlord.rtgov.active.collection.epn.EPNActiveCollectionSource",
    "name" : "Situations",
    "type" : "List",
    "itemExpiration" : 40000,
    "maxItems" : 0,
    "subject" : "Situations",
    "activeChangeListeners" : [ {
        "@class" : "org.overlord.rtgov.active.collection.jmx.JMXNotifier",
    	"objectName" : "overlord.rtgov:name=Situations",
    	"descriptionScript" : "SituationDescription.mvel",
    	"insertTypeScript" : "SituationType.mvel"
	} ]
  }
]

ServiceDefinitions

This active collection is a map of Service Type name to Service Definition. The Service Definition defines:

• the name of the service type (which is also the key for the map),

• the operations it provides (including request, response and fault message types)

• the operations it consumes (including request, response and fault message types)

• the metrics concerning the operations provided and consumed

An example of a service definition, represented in JSON is:

{
    "serviceType":"{http://www.jboss.org/examples}OrderService",
    "operations":[{
        "name":"buy",
        "metrics":{
            "count":30,
            "average":1666,
            "min":500,
            "max":2500
        },
        "requestResponse":{
            "metrics":{
                "count":10,
                "average":1000,
                "min":500,
                "max":1500
            },
            "invocations":[{
                "serviceType":"{http://www.jboss.org/examples}CreditAgencyService",
                "metrics":{
                    "count":10,
                    "average":500,
                    "min":250,
                    "max":750
                },
                "operation":"checkCredit"
            }]
        },
        "requestFaults":[{
            "fault":"UnknownCustomer",
            "metrics":{
                "count":20,
                "average":2000,
                "min":1500,
                "max":2500
             }
        }]
    }],
    "metrics":{
        "count":30,
        "average":1666,
        "min":500,
        "max":2500
    }
}

The list of service definitions returned from this active collection, and the information they represent (e.g. consumed services), represents a near term view of the service activity based on the configuration details defined in the collection’s active collection source. Therefore, if (for example) a service has not invoked one of its consumed services within the time period of interest, then its details will not show in the service definition.

This information is simply intended to show the service activity that has occurred in the recent history, as a means of monitoring the real-time situation to deal with emerging problems.

The duration over which the information is retained is determined by two properties in the ServiceDefinitions active collection source configuration - the "scheduledInterval" (in milliseconds) which dictates how often a snapshot of the current service definition information is stored, and the "maxSnapshots" property which defines the maximum number of snapshots that should be used. So the duration of information retained can be calculated as the scheduled interval multiplied by the maximum number of snapshots.

Principals

This active collection is a map of Principal name to a map of named properties. This information is used to convey details captured (or derived) regarding a 'principal'. A principal can represent a user, group or organization.