Skip to content

Setting up Kafka

The Kafka connector allows you to access the Kafka Producer API through WSO2 EI and acts as a message producer that facilitates message publishing. The Kafka connector sends messages to the Kafka brokers.

Kafka is a distributed publish-subscribe messaging system that maintains feeds of messages in topics. Producers write data to topics and consumers read from topics. For more information on Apache Kafka, see Apache Kafka documentation.

Kafka mainly operates based on a topic model. A topic is a category or feed name to which records get published. Topics in Kafka are always multi-subscriber.

Setting up the environment

To use the Kafka connector, download and install Apache Kafka. Before you start configuring the Kafka you also need WSO2 MI and we refer to that location as .

Note: The recommended version is kafka. For all available versions of Kafka that you can download, see https://kafka.apache.org/downloads. The recommended Java version is 1.8.

To configure the Kafka connector, copy the following client libraries from the <KAFKA_HOME>/lib directory to the <PRODUCT_HOME>/repository/components/lib directory.

Run the following command to start the ZooKeeper server:

bin/zookeeper-server-start.sh config/zookeeper.properties

Run the following command to start the Kafka server:

bin/kafka-server-start.sh config/server.properties

Now that you have connected to Kafka, you can start publishing and consuming messages using the Kafka brokers. For more information, see Publishing Messages using Kafka and Consuming Messages using Kafka.

Top