100% open source low latency,
high-throughput streaming technology
Available in Open Source & Continuuity Reactor: Q3 2014

What is jetStream?

jetStream is a distributed framework built on Apache Hadoop™ and Apache HBase™ for real-time, high-throughput, low-latency data processing and analytics applications.
  • Supports CEP-like continuous query semantics in a SQL-like declarative language
  • Supports model-based, discrete event semantics in an imperative Java API

Challenges with existing technology

Supports only in-memory processing

  • No support for an integrated, scalable persistence layer
  • No support for model-based streaming

Semantics

  • Hard-to-reason semantics
  • No support for true only-once semantics

Operational Complexity

  • Fault Tolerance must be dealt with
  • Scalability at run-time not possible
  • Security not well integrated
  • Logging, Metrics & Alerting

Developer Experience

  • Cumbersome imperative programming paradigm
  • Declarative language capabilities are limiting
  • No easy testing capabilities
  • No easy way to debug distributed real-time applications


the jetstream solution

Features

Killer Performance

  • 10s of millions of events per second
  • Scales to 100s of nodes
  • Millions of concurrent streams
  • Millions of data operations/second

Deep Integrations

  • Runs and scales as a native Hadoop YARN Application
  • Reads, writes, and tightly integrates with HDFS and HBase

Simple Semantics

  • Exactly-once event processing using an app-level Java API with consistency, reliability, and persistence
  • Stream processing using a SQL-like language for processing data in motion

Extensive Features

  • Fault-tolerance and Horizontal Scalability without burdening the developer
  • Enterprise security features in addition to debugging, logging, and monitoring tools
  • Simpler programming model, tools and UIs

Use Cases

FLIGHT PATH TRACKING

Anomaly Detection
  • Problem
    Track multiple aircraft simultaneously to detect any deviations from the proposed flight paths
  • Attributes
    Real-time, streaming event data and models to learn flight path
  • Solution
    In-memory stream querying solution backed by stream based learning models

SMART CITIES

Advanced Monitoring and Alerting
  • Problem
    Monitor utility consumption to reduce spending & wastage; Better management of public resources: parking spots to electricity grid
  • Attributes
    Fault-tolerance, real-time alert systems, low cost commodity hardware solutions
  • Solution
    Fault-tolerance, streaming application with querying capabilities and eventual processing guarantees

FAQ

What is being announced?
Continuuity and AT&T Labs have joined forces, collaborated and will release as an open source project codenamed jetStream, a new disruptive technology that will provide an integrated, high-quality and consistent streaming analytics application framework.
What is jetStream?
jetStream is the codename for an open source project that Continuuity and AT&T Labs have collaborated on. jetStream offers a simple, efficient and cost-effective way for businesses, OEMs/ISVs, system integrators, service providers, and developers to create a diverse range of big data analytics and streaming applications that address a broad set of business use cases such as network intrusion detection and analytics, real-time analysis for spam filtering, social media market analysis, location analytics, and real-time recommendation engines that match relevant content to the right users at the right time.
Why the collaboration between AT&T Labs and Continuuity?
jetStream brings together the complementary functionality of Continuuity BigFlow and the AT&T streaming analytics tool to create a unified real-time framework that combines in-memory stream processing with model-based event processing and direct integration for a variety of data systems including Apache HBase and HDFS. By combining AT&T low-latency and declarative language support with BigFlow’s durable, high-throughput computing capabilities and procedural language support, jetStream provides developers with a disruptive new way to take in and store vast quantities of data, build massively scalable applications, and update applications in real-time as new data is ingested. Specifically, developers will be able to do the following:
  • Direct integration of real-time data ingestion and processing applications with Hadoop and HBase and utilization of YARN for deployment and resource management
  • Framework-level correctness, fault tolerance guarantees, and application logic scalability that reduces friction, errors, and bugs during development
  • A transaction engine that provides delivery, isolation and consistency guarantees that enable exactly-once processing semantics
  • Scalability without increased operational cost of building and maintaining applications
  • Develop pipelines that combine in-memory continuous query semantics with persistent, procedural event processing using simple Java APIs
When will jetStream be available?
jetStream will be made available as an Apache-licensed open source project in the third quarter of 2014.
What license will jetStream be open sourced under?
jetStream will be open sourced under the Apache 2.0 License.
What is the pricing for jetStream?
Pricing for jetStream will be available upon release of the product in the third quarter of 2014. Subscription support for jetStream as well as the related training and professional services offerings will also be available at this time.
Who are the target audiences for jetStream?
jetStream is mainly for developers looking to build streaming applications. Putting jetStream into open source makes required tools for developing data-intensive, complex applications accessible to a broader base of developers across businesses and partners of all sizes.
How long has jetStream been in production?
Individual technologies have been in production for some time. The combined capability is still under heavy development and testing in preparation for open source in Q3 2014.
Can we collaborate on jetStream when it is open sourced?
Yes, this technology will be open sourced under the Apache 2.0 license. Our goal is to create a vibrant community and encourage contributions from anyone interested.
When will jetStream be available in Continuuity Reactor?
jetStream will be available in Continuuity Reactor at around the same time it will be open sourced, which is in the third quarter of 2014.