A New Application Landscape: Islands, Mountains, Clouds and Streams
Event Stream Processing (ESP) is a relatively new computing paradigm, designed specifically to address the requirements of distributed and real-time, high-performance data processing. In contrast to traditional data management systems wherein data are first stored, indexed and then processed by queries, Event Processing Applications work on data in-flight, as it moves (streams) through the system. Data streams flow through applications as discrete message units called events (or event tuples) comprised from a set of structured data elements. Queries for filtering, correlation and aggregation are applied to in-flight data and results are delivered to consuming applications as discrete business events, facilitating streams of actionable information. Reacting to business events provides an organization with real-time situational awareness, allowing it to quickly adapt to the ever changing landscape (Streamscape) of enterprise data.
The StreamScape Service Application Engine™ exploits event stream processing as part of it's communication fabric and data processing facilities, allowing users to create scalable event processing applications that can be easily integrated with other event stream processing solutions. The application engine is a technology complementary to Complex Event Processing (CEP)[1], providing critical event sourcing, data mapping and event-based communication facilities. The engine's unique capabilities allow a broad range of enterprise, web application and social computing activities to be turned into actionable events, expanding the value of event processing technologies and making them part of the collaborative computing platform.
 |
Business Events vs. Messages |
Business Events are defined as any discrete state changes or occurrences within the enterprise that require a business or an organization to take action. For example, a change in the price of a stock, an increase in a patients heart rate or the failure of a data synchronization process are all considered actionable business events; and distinguished from messages by several characteristics.
Events are typically produced with the intent of being filtered, correlated or processed by event processing applications or Complex Event Processing engines. Unlike messaging systems that favor anonymous passing of opaque data, exchange of information between event processing applications is accomplished using structured data in a content-aware fashion as content is used to route and filter event flows. Certain events are meant to propagate across applications, essentially passing through system participants and preserving the chain of causality. Such events are considered idempotent and not supported by messaging technologies. Raising an event makes it's content directly available to consumers and should not require intermediaries or additional communication abstractions such as queues, topics or subjects, eliminating the most time consuming and error prone aspects of messaging application development.
Events are inherently transient. Although event data can be stored for future analysis, many event processing applications will consume a data stream and discard it's content after taking appropriate actions. Stream processing technologies leverage new query and aggregation techniques to sift through enormous amounts (mountains) of information without the need for storing and indexing such data. Alternatively event stream data may be saved, resulting in large collections of so-called big data. Such datasets may be turned back into data streams, allowing businesses to make sense out of mountains of information by analyzing their data stream content. Big data analytics in various forms is quickly becoming the next frontier for innovation, competition and productivity across a number of industries.
 |
Stream Sourcing and Data Drought |
Event Stream Processing addresses the challenge of integrating and synchronizing disparate islands of data in real-time. Stream processing technologies can treat changes at remote locations (such as meters, SOA services, database or file operations) as event streams, allowing organizations to apply the stream processing paradigm to data correlation, distribution and synchronization across disparate collections (islands) of information; thereby leveraging event stream processing to govern in-flight data, improve scalability and lower the cost of integration efforts.
One of the main challenges with stream processing is the so called stream sourcing problem. Event processing applications and Complex Event Processing engines require streams of structured data in order to function. Although enterprise systems perform many data change operations, results of such activities must first be converted into event streams so they may be recognized and handled by stream processing technologies. Lack of this critical capability results in a data drought that limits the scope and relevance of event stream processing solutions.
Currently, ESP implementations are prevalent in High Frequencey Trading applications, Capital Markets systems and certain Government applications wherein data formats are well established and rarley change. However, most innovation and arguably most qustionable data processing occurs in custom applications and the transient data layer wherin the proprietary logic of a business resides. This application layer often requires developers to bridge the integration gap by building custom stream-generating adapters using a vendor supplied SDK.
Adapters solve the critical problems of Format and Protocol by structuring source data in a way that is recognizable to stream processors; and by using a vendor-specific communication protocol. Unfortunaltey, at present there is no unified standard for describing event data and no standard protocol exists for stream processing technologies with certain vendors advocating the use of tuples (similar to records or table rows) while others allow the use of objects or other complex structures.
Lack of standardized tools and technologies for sourcing event streams and making them available across a broad range of stream procesing technologies is contributinhg to slow adoption of the event stream processing paradigm across the enterprise. To promote broader acceptance, software vendors must evolve their solutions allowing developers and architects to easily integrate CEP and ESP technologies into home-grown systems, Web Applications, Social Networks and Cloud Computing platforms.
 |
The Event Cloud |
The rising popularity of cloud computing opens stream processing technologies to possible cloud deployment using the Software as a Service (SaaS) model. Event processing applications may be offered as services on a pay-as-you-go basis or may play a more critical role in a larger cloud offer, performing tasks such as click-stream analysis, fraud detection or resource management.
At the heart of StreamScape's application engine is the Service Event Fabric™, a self organizing event cloud capable of hosting load-able application components and providing facilities for adaptive peer-to-peer communications. All application engine activity as well as a variety of external resource operations (ie. Files, Databases) may be transformed into event streams. Event Triggers and EDL allow developers to capture and route critical events to their intended recipients using the event fabric. Web Applications and SOA Services hosted by the engine can easily expose activities such as service invocation or browser clicks as event streams. Interactions between participant components in the cloud, data space operations and Instant Messenger activity (including presence information) can likewise be made available to stream processing technologies.
The engine allows developers to take advantage of application fabric capabilities typically found in larger, more expensive cloud computing platforms; including support for managed application life-cycle, peer communications, distributed deployment and un-structured data storage services, allowing architects to implement cost-effective cloud systems within the enterprise.
[1] According to The Gartner Group, a leading IT market research firm, the CEP market is growing rapidly and looks to remain that way, reaching a worldwide market size of around $580 million by 2013 with a compound annual growth rate (CAGR) from 2008 through 2013 of 31%.