With more than 12,000 API being offered by companies such as Uber, Airbnb, Alibaba, Paypal and Pandora, the API Economy is upon us. Whether they allow established firms to expand their ecosystem and reach into new markets or enable start-ups to quickly define or acquire a segment of an emerging market, one thing is certain: digital technologies and their Application Programing Interfaces are the engine behind new front-office, customer-facing, and demand-generating systems. Digital demand moves at the pace of the market, competition, and customer expectation rather than the dev cycle of traditional IT vendors; and API-centric architecture has emerged to fill this gap.
Unfortunately, some developers and architects still see the APIs as evolution and continuation of integration-based architectures that have long been in use within enterprise IT. This limited view often misses the larger fact that APIs have become a foundation technology for development of robust and scalable enterprise applications.
Modern apps use APIs on the front end to connect the user experience (UX) tier (often written in HTML5 or native mobile client languages) to back-end systems. Apps use APIs on the back-end to access process flow and data services. And apps themselves are often built out of interconnected “micro-services”, that are wired together via APIs.
The biggest challenge for many firms is a lack of vision as they struggle to grasp the benefits of API-centric architecture. Rather than see APIs as a new platform or channel for customer engagement, organizations consider APIs an after-thought that is part of infrastructure integration. As such, one of the biggest obstacles to adoption of the API model is a re-alignment of IT resources, away from in-house system support and towards a service-based model that supports any current and future application or device.
As with any technological paradigm shift, cost and time-to-market play a major role in proper adoption. Many IT organizations are simply unable to make a timely transition to a service-based infrastructure. As result, a business often fails to capitalize on the value that an API-centric architecture provides and competition gets there first.
StreamScape offers its customers a single, powerful platform for API authoring, data automation and analytics that complements an existing IT tool set. Organizations can easily expose any existing back-end system, process or data source as a secure and customizable web-based API, without writing custom code or changing existing apps. This allows companies of all size to quickly realize the potential of an API-centric approach and take advantage of new platforms or technologies without compromising security or sacrificing performance of their back-end systems. Instead of developing and supporting an API platform organizations can buy or lease one in the Cloud at a fraction of the cost.
CIO Review: What are the latest technological trends observed in the API landscape and how do they impact the solutions and services StreamScape offers?
An API strategy is not an all-or-nothing decision — to use them or not to use them. There may be public APIs open to everyone, private APIs open only to certified developers, paying customers or partners; there are free APIs offered by companies like Google and flat fee or revenue-sharing ones such as those provided by Apple.
Abundance of open and syndicated data sources now allows businesses to offer data aggregation and analysis as a service. Aggregate data are often exposed as APIs that in turn use 3rd party API to collect and process data specific to customer needs from other platforms. Aggregation as a service is an emerging trend across the API landscape with companies like Splunk leading the charge.
StreamScape’s Reactive Data Platform™ offers unique capabilities that allow users to easily build their own data aggregation services that collect and host in-flight, aggregate data in Application Dataspaces™. A dataspace is a secure, mostly-memory store that holds collections of structured or unstructured data. Content may be populated via client API calls, external data streams or proxy API calls made to other systems, allowing users to perform ‘on-demand aggregation’ of critical information across disparate sources.
Mobile applications represent a return to the more strongly delineated client-server model of development, wherein most interactions occur client-side and constant network communications between clients and a back-end system must be maintained. The APIs often include function and information services that firms offer to their ecosystem users as well as those that may be used for internal purposes. As such, an organization needs to be disciplined about API versioning, security, documentation and provide a set of governance mechanisms to enforce this.
Unlike many other technologies that were adapted to support single page applications and API authoring, the Reactive Data Platform™ was purpose- built to support API versioning, security and governance that enforces developer best practices and allows smaller teams to be very effective at developing resilient and scalable, componentized applications.
CIO Review: The Reactive Data Platform™ is a data analytics and aggregation technology for cloud and web applications and offers many features. Could you tell us more about the technology?
Sure. Apologies if the elevator pitch sounds a bit practiced. (smiles) We are an new type of infrastructure technology. The Reactive Data Platform™ is a light-weight software suite for developing and hosting Application Data Services. These specialized components (often called Micro Services) let users query and process data using some innovative techniques otherwise found in reactive programming frameworks and languages like Scala or Ehrlang. The platform includes an SQL-like data processing language (script, no compiling needed) and API authoring tools. We make it easy for the avarage user to work with data streams, events and storage systems like files or databases. We also make management and scaling across cheap hardware very simple through the use of our DevOps tools. This shields users from the technical aspects of data management, integration and aggregation. So they can solve some pretty complex data problems fairly quickly, without spending a fortune on technology or developers.
At the heart of the platform is a unique, in-memory data streaming engine that allows users to model, query and observe data, reacting to any changes. In data processing this is sometimes called the Actor Model. Actors allow the user to script business logic that reacts to data changes, streams of events or client API calls (ie. web events). So this is a stream processing engine (SPE) that also knows how to query databases, files and so forth. Users can also write their own functions to extend logic and capabilities. Functions make it easy to to access local data, generate events or access external resources like databases, files or enterprise applications. So data from any source can be turned into a stream, processed by the engine an joined to any other data anywhere. Functions, queries or streaming results can be exposed in application-specific format such as XML, JSON, Web Sockets to Web Clients, consuming applications or devices like tablets, smart phones or televisions. We support HTTP(S), (S)FTP, XMPP, JDBC, Web Sockets and the emerging OData standard. Its a very comprehensive platform. Did I mention it's very light-weight and installs in minutes? (laughs)
This platform is intended to complement the existing IT tool set and typically sits on top of an existing enterprise infrastructure, allowing each part of a system (and its APIs) to change without disturbing the peace across all components. I think data aggregation technologies are critical in a world of consumer-driven technology, multi-vendor competition and standards wars that are now fought in the marketplace rather than the lab.
Platforms fit the economics of digital business. They provide a means to gain scale efficiencies across the enterprise rather than trying to drive them across individual infrastructures. This is essential in a world where IT transaction volumes grow faster than business revenues, in small margin transactions. A simple example: consider online banking where many transactions are free of charge but have a real cost to the bank. Platforms provide a way to drive down transaction costs and preserve company margins.
A platform offers a business-oriented view of technology, modeling and organizing functions around specific business actions like one-click sales, search, order processing and pricing; using data rather than business logic as the source of specialization. This enables platform users to add new features and functions (APIs) that are available to certain ecosystem users or all customers providing the flexibility and adaptability required of a modern business.
CIO Review: StreamScape is a unique name for a company. Could you tell us the story behind the brand name?
StreamScape’s name connotes the key capabilities and philosophy of our technology platform. Classic database technologies process data by running a query on a complete information set. This approach is not really practical for today’s application data, which tends to be very noisy, voluminous and always on the move. In-memory databases are also a dead-end because you don’t have infinite memory and data volumes tend to grow very quickly, so you can’t solve modern problems with hardware alone. And when you add web APIs on top of it, returning giant result sets is also not an option. Transient data has to stick around somewhere and let users and devices consume it at their own pace.
The approach we took was to treat all data as a stream of information, regardless of where it comes from (or goes to) and we built a unified query engine that can work with this type of in-flight data. In contrast to databases, our data is pushed thru the query engine and when it sees something in the stream that matches the query it saves the result set in a data collection or allows other components to react to the relevant information, facilitating reactive data processing. As such the landscape of data islands, data lakes and data streams simply becomes the streamscape of data.
Although the engine sees everything as a stream, results or reference data can be stored in more traditional collections like tables, maps or queues in the Application Dataspace™ and queried via standard SQL. Users can blend data from streams, external systems or local data collections seamlessly and expose such data via an API.
Unlike a database StreamScape is strictly a query engine, not a storage engine. It can ingest files, unstructured data, web request or relational query results and process them as streams or save them to a dataspace. The Reactive Data Platform™ offers Service Packs for various types of content such as XML documents, file, FTP, Web Services, Hadoop and Databases as well as applications such as Salesforce, allowing users to animate data in external systems and turn it into streams that may be queried by the engine. All part of the ecosystem in the application streamscape.
CIO Review: What does it take for a company like StreamScape to stay relevant, to deliver technology and services that fulfill your client's business objectives?
As a technology-driven organization we believe innovation comes from functional convergence; and a large number of ground-breaking products have emerged in the past decade to support this idea. Smart phones, tablets and networked appliances are just a few examples of integrated devices whose value is greater than the sum of their technology parts. So, with hardware cost and processor size decreasing each year, enterprise-class software is starting to cross over into the consumer application space.
Several promising technologies have emerged in the last few years that are focused on mobile computing, wearables and the Internet of Things. We believe this trend will continue as customers extend business-level, enterprise systems with the exploding power of consumer technologies like smart- phones, tablets and watches. Some of it is hype, but there is a lot of real business value that will emerge.
A fair amount of StreamScape’s R & D is focused on researching and integrating commodity features (such as Instant Messaging, Swagger or OData support) into the platform, reducing the solution footprint with the goal of achieving a kind of hardware synchrony; allowing the same software and services to scale across enterprise-grade servers, commodity hardware as well as micro-devices such as a tablet or Raspberry Pi.
Adoption of certain features may still be a few years away, so patience is a virtue in high demand when you plan for the future. But our track record has been pretty good so far. It is equally important to know when to stop taking advice and focus on execution. As often it happens, todays mighty oak is just yesterday’s nut that held its ground.
CIO Review: Even the most competent of organizations face market competition. What are the strategies employed by StreamScape to thwart this competition? What are the company’s key differentiating factors?
We have competitors? (smiling) Sure, a fair question. The API market is a new technology space. Adoption of API-centric architecture by IT organizations and vendors alike frequently confuses prospects, creating competition in the unlikeliest of places. Working with our customers to gain a deeper understanding of their needs exposed the larger problem with modern enterprise architecture; shifting our attention to the challenges of API authoring and transient data that often needs to be kept around while servicing API requests. Although a number of would-be competitors are addressing issues of API security and governance, the real challenge that we address is data aggregation and use by API platforms.
API-centric architecture presents a broad set of challenges. We believe our competition is ‘digging in the wrong place’ and providing simple mechanisms for exposing data or business logic as API, rather than focusing on the tougher problem of how applications want to do model and use the underlying data.
At this time limited technologies are available to assist application developers with defining and versioning API, working with document and schemas or managing data flow; forcing users to rely on immature frameworks or developing proprietary solutions. We are truly fighting our competitors where they aren’t and identifying an important market gap in the process.
Our journey of discovery led to the realization that API-centric architecture requires new technologies and a new mind set. Even with available frameworks, emerging solutions and open source tools building a solution is not easy. Developers need a simpler way to define APIs in the cloud and on premise. Making API enablement easy, attainable and cost-effective is a key differentiating factor for StreamScape.
CIO Review: Can you draw an analogy between personal traits, hobbies and how you steer StreamScape?
Well, over the years I was fortunate enough to work for several great companies that rewarded creativity and motivated employees to promote their ideas to customers. This was the kind of tech-company I wanted to start someday; a place that offered certain levels of transparency, allowing team members to directly participate in the greater mission.
Today, most of the engineering team is customer-facing, allowing developers and support staff to work with the sales team, gaining a broader insight into customer needs and staying ahead of emerging technology trends. We find this type of feedback loop critical. Client interaction fosters a deeper sense of problem ownership among the staff, making it possible to develop targeted solutions that are natural extensions of the product. As a result, software quality improves and we learn valuable things from our customers in the process.
What' on the roadmap for StreamScape in the upcoming years?
DevOps (a blend of development and operations) is a software development method that leverages automation to enable an agile development and streamlined deployment processes, accelerating the release of software to production. Initially very popular within internet companies that needed to break down the traditional walls between development and IT operations in order to deliver cloud-based applications at scale, DevOps has also become a ﬁxture within enterprise IT for many of the same reasons.
Enterprise IT organizations are embracing the same collaborative development techniques as internet companies and DevOps has proven an efﬁcient way to reduce costs by sharing responsibilities across development and operations teams. As a technology purpose- built for Cloud and Web, StreamScape are committed to the DevOps model by providing tools that facilitate constant improvement and collaboration between groups. It is our belief that application development tools—as critical agents of change—are instrumental in managing technology as well as shaping culture. Our goal is to equip our customers with the technologies that address contemporary IT challenges of building and managing agile, high-velocity organizations.
We intend to continue offering the next generation of application resource tools that allow users to take advantage of the API economy and ask hard, meaningful questions of the data without spending a fortune in the process. Our focus is on web-based application development, integration with automation tools (ie. Docker, Chef and Puppet), version control services such as Git, small-footprint query engines and information networks built for the purpose of reactive data processing.
We are currently working with Microsoft Azure, Amazon and other cloud providers as well as data storage partners like MongoDB, Hadoop, Cassandra and application vendors such as Salesforce to make this happen.
CIO Review: Any other interesting insight [that] we might have missed?
Plenty (laughs), but we will keep it short. It has been observed, recently that the future of Corporate IT looks a lot like Google in terms of culture, evolution and responsiveness to customer needs. We tend to agree, with an added caveat. To build a relevant, forward-thinking and reactive IT organization requires a different way of thinking and a renewable infrastructure. In the end, “it’s the platform, stupid”.