

Kafka Connector Example

Given below is a sample scenario that demonstrates how to send messages to a Kafka broker via Kafka topics. The publishMessages operation allows you to publish messages to the Kafka brokers via Kafka topics.

Info

You can try out this example using both 3.0.0 and 3.1.0 versions of the Kafka connector.

What you'll build

Given below is a sample API that illustrates how you can connect to a Kakfa broker with the init operation and then use the publishMessages operation to publish messages via the topic. It exposes Kakfa functionalities as a RESTful service. Users can invoke the API using HTTP/HTTPs with the required information.

API has the /publishMessages context. It publishes messages via the topic to the Kafka server.

The following diagram illustrates all the required functionality of the Kafka service that you are going to build.

If you do not want to configure this yourself, you can simply get the project and run it.

Set up Kafka

Before you begin, set up Kafka by following the instructions in Setting up Kafka.

Configure the connector in WSO2 Integration Studio

Follow these steps to set up the Integration Project and the Connector Exporter Project.

1. Open WSO2 Integration Studio and create an Integration Project.

2. Right click on the project that you created and click on Add or Remove Connector -> Add Connector. You will get directed to the WSO2 Connector Store.

3. Search for the specific connector required for your integration scenario and download it to the workspace.
   

4. Click Finish, and your Integration Project is ready. The downloaded connector is displayed on the side palette with its operations.

5. You can drag and drop the operations to the design canvas and build your integration logic.

6. Right-click the created Integration Project and select New -> Rest API to create the REST API.

7. Specify the API name as KafkaTransport and API context as /publishMessages. You can go to the source view of the XML configuration file of the API and copy the following configuration (source view).

   <?xml version="1.0" encoding="UTF-8"?>
   <api context="/publishMessages" name="KafkaTransport" xmlns="http://ws.apache.org/ns/synapse">
       <resource methods="POST">
           <inSequence>
               <kafkaTransport.init>
                   <name>Sample_Kafka</name>
                   <bootstrapServers>localhost:9092</bootstrapServers>
                   <keySerializerClass>org.apache.kafka.common.serialization.StringSerializer</keySerializerClass>
                   <valueSerializerClass>org.apache.kafka.common.serialization.StringSerializer</valueSerializerClass>
                   <maxPoolSize>100</maxPoolSize>
               </kafkaTransport.init>
               <kafkaTransport.publishMessages>
                   <topic>test</topic>
               </kafkaTransport.publishMessages>
           </inSequence>
           <outSequence/>
           <faultSequence/>
       </resource>
   </api>

   Now we can export the imported connector and the API into a single CAR application. The CAR application needs to be deployed during server runtime.

Exporting Integration Logic as a CApp

CApp (Carbon Application) is the deployable artifact on the Enterprise Integrator runtime. Let us see how we can export integration logic we developed into a CApp along with the connector.

Creating Connector Exporter Project

In order to bundle Connector into a CApp a Connector Exporter Project is needed.

1. Navigate to File -> New -> Other -> WSO2 -> Extensions -> Project Types -> Connector Exporter Project.
   

2. Enter a name for the Connector Exporter Project.

3. In the next screen select, Specify the parent from workspace and select the specific Integration Project you created from the dropdown.

4. Now you need to add the Connector to Connector Exporter Project that you just created. Right click on the Connector Exporter Project and select, New -> Add Remove Connectors -> Add Connector -> Add from Workspace -> Connector

5. Once you are directed to the workspace, it displays all the connectors that exist in the workspace. You can select the relevant connector and click Ok.

Creating a Composite Application Project

To export the Integration Project as a CApp, a Composite Application Project needs to be created. Usually, when an Integration project is created, this project can be created as part of that project by Integration Studio. If not, you can specifically create it by navigating to File -> New -> Other -> WSO2 -> Distribution -> Composite Application Project.

Exporting the Composite Application Project

1. Right click on Composite Application Project and click on Export Composite Application Project.
   

2. Select an Export Destination where you want to save the .car file.

3. In the next Create a deployable CAR file screen, select both the created Integration Project and the Connector Exporter Project to save and click Finish. The CApp will get created at the specified location provided at the previous step.

Get the project

You can download the ZIP file and extract the contents to get the project code.

Deployment

Follow these steps to deploy the exported CApp in the Enterprise Integrator Runtime.

Testing

Create a topic:

Let’s create a topic named “test” with a single partition and only one replica. Navigate to the and run the following command:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test     

Note

If your Kafka version is older than 2.2, then the --bootstrap-server option will not be supported. Use the --zookeeper options as shown below.

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Sample Request:

Send a message to the Kafka broker using a CURL command or sample client.

curl -X POST -d '{"name":"sample"}' "http://localhost:8290/publishMessages" -H "Content-Type:application/json" -v

Expected Response:

Navigate to the and run the following command to verify the messages:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

See the following message content:

{"name":"sample"}

This demonstrates how the Kafka connector publishes messages to the Kafka brokers.

What's next

• You can deploy and run your project on Docker or Kubernetes. See the instructions in Running the Micro Integrator on Containers.
• To customize this example for your own scenario, see kafka Connector Configuration documentation.

Top