Introduction

WSO2 Streaming Integrator(SI) is a streaming data processing server that integrates streaming data and takes action based on streaming data. The streaming integration capabilities of EI are delivered via this runtime

WSO2 SI can be effectively used for:

  • Realtime ETL: CDC for DBs, tailing files, scraping HTTP Endpoints, etc.
  • Work with streaming messaging systems: It is fully compatible with Kafka and NATS, and provides advanced stream processing capabilities required to utilize the full potential of streaming data.
  • Streaming data Integration: Allows you to treat all data sources as streams and connect them with any destination.
  • Execute complex integrations based on streaming data: SI has native support to work hand-in-hand with WSO2 Micro integrator to trigger complex integration flows based on decisions derived via stateful stream processing logic.

Try it out!

To try out each of the above use cases with SI, see Tutorials

Key Features

WSO2 SI is powered by Siddhi.io, a well known cloud native open source stream processing engine. Siddhi allows you to write complex stream processing logic using an intuitive SQL-like language known as SiddhiQL. You can perform the following actions on the fly using Siddhi queries and constructs.

  • Transforming your data from one format to another (e.g., to/from XML, JSON, AVRO, etc.).
  • Enriching data received from a specific source by combining it with databases, services, etc., via inline calculations and custom functions.
  • Correlating data streams by joining multiple streams to create an aggregate stream.
  • Cleaning data by filtering it and by modifying the content (e.g., obfuscating) in messages.
  • Deriving insights by identifying interesting patterns and sequences of events in data streams.
  • Summarizing data as and when it is generated using temporal windows and incremental time series aggregations.

Streaming Integrator/ Workflow

With 60+ prebuilt and well tested collection of connectors, WSO2 SI allows you to connect any data source with any destination regardless of the different protocols and data formats used by the different endpoints.

The SI Store API exposes aggregated and collected data streams to in-memory and persistence storages via a REST API, allowing you to execute queries and generate summarized information on demand.

Synapse integration flows deployed in WSO2 Micro Integration(MI) can be executed directly by SI. This allows you to build robust data processing and integration pipelines by combining powerful stream processing and integration capabilities.

Tooling

WSO2 SI is coupled with the Streaming Integrator Tooling; a comprehensive streaming integration flow designer for developing Siddhi applications by writing queries or via the drag-and-drop functionality, testing them thoroughly before using them in production, and then deploying them in the SI server or exporting them to be deployed as a K8 or a Docker image.

Centralized and Decentralized Deployment

Being container-friendly by design with a small image size, low resource footprint, a startup time less than two seconds, etc., WSO2 SI can be easily deployed in VMs, Docker or K8s.

Its native support for Kubernetes with a K8s Operator provides a convenient way to deploy SI on a K8s cluster with a single command, thereby eliminating the need for manual configurations.

Deploying SI as a highly available minimum HA cluster allows you to achieve zero data loss with just two SI nodes.

What's Next

Top