Kafka Connector Example¶
Given below is a sample scenario that demonstrates how to send messages to a Kafka broker via Kafka topics. The publishMessages operation allows you to publish messages to the Kafka brokers via Kafka topics.
What you'll build¶
Given below is a sample API that illustrates how you can connect to a Kakfa broker with the init
operation and then use the publishMessages
operation to publish messages via the topic. It exposes Kakfa functionalities as a RESTful service. Users can invoke the API using HTTP/HTTPs with the required information.
API has the context /publishMessages
. It will publish messages via the topic to the Kafka server.
The following diagram illustrates all the required functionality of the Kafka service that you are going to build.
If you do not want to configure this yourself, you can simply get the project and run it.
Configure the connector in WSO2 Integration Studio¶
Follow these steps to set up the Integration Project and the Connector Exporter Project.
-
Open WSO2 Integration Studio and create an Integration Project.
-
Right click on the project that you created and click on Add or Remove Connector -> Add Connector. You will get directed to the WSO2 Connector Store.
-
Search for the specific connector required for your integration scenario and download it to the workspace.
-
Click Finish, and your Integration Project is ready. The downloaded connector is displayed on the side palette with its operations.
-
You can drag and drop the operations to the design canvas and build your integration logic.
-
Right click on the created Integration Project and select New -> Rest API to create the REST API.
-
Specify the API name as
KafkaTransport
and API context as/publishMessages
. You can go to the source view of the XML configuration file of the API and copy the following configuration (source view).
Now we can export the imported connector and the API into a single CAR application. The CAR application needs to be deployed during server runtime.<?xml version="1.0" encoding="UTF-8"?> <api context="/publishMessages" name="KafkaTransport" xmlns="http://ws.apache.org/ns/synapse"> <resource methods="POST"> <inSequence> <kafkaTransport.init> <bootstrapServers>localhost:9092</bootstrapServers> <keySerializerClass>org.apache.kafka.common.serialization.StringSerializer</keySerializerClass> <valueSerializerClass>org.apache.kafka.common.serialization.StringSerializer</valueSerializerClass> <maxPoolSize>100</maxPoolSize> </kafkaTransport.init> <kafkaTransport.publishMessages> <topic>test</topic> </kafkaTransport.publishMessages> </inSequence> <outSequence/> <faultSequence/> </resource> </api>
Get the project¶
You can download the ZIP file and extract the contents to get the project code.
Deployment¶
Follow these steps to deploy the exported CApp in the Enterprise Integrator Runtime.
Deploying on WSO2 Enterprise Integrator 7
You can copy the composite application to the Other Resources
section. Make sure you first export the PATH as below.
$ export PATH=/path/to/mi/cli/directory/bin:$PATH
-
Log in to Micro Integrator using the following command.
./mi remote login
-
Provide default credentials admin for both username and password.
-
In order to view the APIs deployed, execute the following command.
./mi api show
Click here for instructions on deploying on WSO2 Enterprise Integrator 6
-
You can copy the composite application to the
/repository/deployment/server/carbonapps folder and start the server. -
WSO2 EI server starts and you can login to the Management Console https://localhost:9443/carbon/ URL. Provide login credentials. The default credentials will be admin/admin.
-
You can see that the API is deployed under the API section.
Testing¶
Create a topic:
Let’s create a topic named “test” with a single partition and only one replica.
Navigate to the
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
Sample Request:
Send a message to the Kafka broker using a CURL command or sample client.
curl -v POST -d '{"name":"sample"}' "http://localhost:8290/services/KafkaTransport" -H "Content-Type:application/json"
Expected Response:
Navigate to the
See the following message content:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This demonstrates how the Kafka connector publishes messages to the Kafka brokers.{"name":"sample"}
What's next¶
- You can deploy and run your project on Docker or Kubernetes.
- To customize this example for your own scenario, see kafka Connector Configuration documentation.