WSO2 Streaming Integrator(SI) is a streaming data processing server that integrates streaming data and takes action based on streaming data. The streaming integration capabilities of EI are delivered via this runtime
WSO2 SI can be effectively used for:
- Realtime ETL: CDC for DBs, tailing files, scraping HTTP Endpoints, etc.
- Work with streaming messaging systems: It is fully compatible with Kafka and NATS, and provides advanced stream processing capabilities required to utilize the full potential of streaming data.
- Streaming data Integration: Allows you to treat all data sources as streams and connect them with any destination.
- Execute complex integrations based on streaming data: SI has native support to work hand-in-hand with WSO2 Micro integrator to trigger complex integration flows based on decisions derived via stateful stream processing logic.
Try it out!
To try out each of the above use cases with SI, see Tutorials
WSO2 SI is powered by Siddhi.io, a well known cloud native open source stream processing engine. Siddhi allows you to write complex stream processing logic using an intuitive SQL-like language known as SiddhiQL. You can perform the following actions on the fly using Siddhi queries and constructs.
- Extracting data from static sources in real time.
- Loading and writing data to databases, files and cloud based storages.
- Receiving data in transit from message brokers and data publishers.
- Publishing Data to destinations.
- Stream processing activities including cleansing, transforming, enriching correlating, and summarizing data.
- Performing real time ETL.
With 60+ prebuilt and well tested collection of connectors, WSO2 SI allows you to connect any data source with any destination regardless of the different protocols and data formats used by the different endpoints.
The SI Store API exposes aggregated and collected data streams to in-memory and persistence storages via a REST API, allowing you to execute queries and generate summarized information on demand.
Synapse integration flows deployed in WSO2 Micro Integration(MI) can be executed directly by SI. This allows you to build robust data processing and integration pipelines by combining powerful stream processing and integration capabilities.
WSO2 SI is coupled with the Streaming Integrator Tooling; a comprehensive streaming integration flow designer for developing) Siddhi applications by writing queries or via the drag-and-drop functionality, testing them thoroughly before using them in production, and then deploying them in the SI server or exporting them to be deployed as a K8 or a Docker image.
Centralized and Decentralized Deployment¶
Being container-friendly by design with a small image size, low resource footprint, a startup time less than two seconds, etc., WSO2 SI can be easily deployed in VMs, Docker or K8s.
Its native support for Kubernetes with a K8s Operator provides a convenient way to deploy SI on a K8s cluster with a single command, thereby eliminating the need for manual configurations.
Deploying SI as a highly available minimum HA cluster allows you to achieve zero data loss with just two SI nodes.Top