File Connector Example¶
File Connector can be used to perform operations in the local file system as well as in a remote server such as FTP and SFTP.
What you'll build¶
This example describes how to use the File Connector to write messages to local files and then read the files. Similarly, the same example can be configured to communicate with a remote file system (i.e FTP server) easily. The example also uses some other mediators coming with WSO2 EI to manipulate messages.
An API is exposed to accept XML messages (employee information). When a message is received, it is converted to a CSV message and then stored to a property. A check is done to see if the CSV file (with the same information) exists in the file system. If it does not exist, the connector creates a CSV file with CSV headers included. Then, the connector appends the new CSV entries in the current message to the CSV file. The connector then reads the same CSV file and converts the information back to XML and responds to the client.
Setting up the environment¶
Create a folder in your local file system with read and write access. This will be your working directory. In this example, it is /Users/hasitha/temp/file-connector-test/dataCollection
.
Note
If you set up a FTP server, SFTP server, or a Samba server, do the required configurations and select a working directory. Save the host, port, and security related information for future use.
Configure the connector in WSO2 Integration Studio¶
Follow these steps to set up the Integration Project and the Connector Exporter.
{!references/connectors/importing-connector-to-integration-studio.md!}
Creating the Integration Logic¶
-
Create a new integration project. Be sure to enable a Connector Exporter.
-
Create an API named
TestAPI
with the/fileTest
context. This API will accept employee information. -
In the default resource of the API, enable POST requests.
-
Add the Log mediator to the design canvas and configure a custom log that indicates when the API receives a message.
- Add the DataMapper mediator and configure it to transform the incoming XML message to a CSV message.
-
Double-click the Datamapper mediator and add a new transform configuration called 'xmlToCsv'.
-
Save the following content as an XML file. This will be the data input file.
<test> <information> <people> <person> <name>Hasitha</name> <age>34</age> <company>wso2</company> </person> <person> <name>Johan</name> <age>32</age> <company>IBM</company> </person> </people> </information> </test>
-
Load the input file into the Datamapper config view.
-
Save the following content as a CSV file. This will be the data output file.
Name,Age,Company Hasitha,34,wso2 Johan,32,IBM
-
Load the output CSV file into the datamapper config view.
-
Configure the mapping as shown below. Each element in the input should be mapped to the respective element in the output.
-
Specify the input as XML and output as CSV in the datamapper as shown below.
-
Add the Enrich mediator and configure it to save the output generated by the datamapper to a property named
CONTENT
. -
Now, let's use the File connector to check if the file containing employee information already exists in the file system.
- Add the checkExist operation of the File connector to the canvas.
-
Create a new file connection pointing to the working directory we already set up. Keep this as the File connection for the operation.
-
Configure the file path as
/dataCollection/employees/employees.csv
. This file will store the employee information.
-
Add the Filter mediator to branch out the logic based on the result from the File connector’s
checkExist
operation.Note
If the file does not exist, the File connector’s write operation (which we configure later) will create the file.
-
Click the Filter mediator and define the filter condition as shown below.
-
Inside the “else” block, add the File Connector's write operation and configure it to write the static content of CSV file headers: “Name,Age,Company”.
Note
Be sure to append a new line at the end of the file. The Write Mode needs to be
Create New
.
-
-
After the Filter mediator, use the Enrich mediator again to put back the saved payload into the message payload.
-
Add the File connector’s write operation again and configure it to append the CSV message to the existing file. The Write Mode needs to be 'Append'.
Note
As we need the newest message on the top, always append to line number 2.
-
Add the File connector’s read operation and configure it to read the same CSV file.
Note
The file reading will start from line number 2. The content is read as a text message.
-
Add the Datamapper mediator again and configure it to convert the CSV message (after reading) back to XML.
-
Double-click the data mapper and add a new configuration called 'csvToXml'.
-
This time, the mapping should be from CSV to XML.
-
-
Finally, use the Respond mediator to send the transformed message to the API caller.
-
Now, let's configure a fault sequence to generate an error message when an error occurs in the message flow.
-
Creat a fault sequence with a Log mediator and Respond mediator.
-
Configure the Log mediator generate a custom error.
-
Add the fault sequence to the API resource as its fault sequence.
-
{!references/connectors/exporting-artifacts.md!}
Deployment¶
Follow these steps to deploy the exported CApp in the Enterprise Integrator runtime.
{!references/connectors/deploy-capp.md!}
Testing¶
-
Create a file called data.xml with the following payload.
Note
When you configuring this
source
parameter in the Windows operating system, set this property as<source>C:\\Users\Kasun\Desktop\Salesforcebulk-connector\create.txt</source>
.<test> <information> <people> <person> <name>Hasitha</name> <age>34</age> <company>wso2</company> </person> <person> <name>Johan</name> <age>32</age> <company>IBM</company> </person> <person> <name>Bob</name> <age>30</age> <company>Oracle</company> </person> <person> <name>Alice</name> <age>28</age> <company>Microsoft</company> </person> <person> <name>Anne</name> <age>30</age> <company>Google</company> </person> </people> </information> </test>
-
Invoke the API as shown below using the curl command.
Info
Curl Application can be downloaded from here.
curl -H "Content-Type: application/xml" --request POST --data @data.xml http://10.100.5.136:8290/fileconnector/create
-
Check the file system to verify that the CSV file has been created.
-
If you invoke the API again with a different set of employees, the new employees will get appended to the same file. The response you receive will include all the employees that were added from both messages.
In this example, the File connector was used to create a file, write to a file, and to read a file. By blending these capabilities together with other powerful message manipulation features of WSO2 EI, it is possible to define a working scenario in minutes. The File connector has many more functionalities. Refer the File Connector reference guide for more information.
What's Next¶
- You can deploy and run your project on Docker or Kubernetes. See the instructions in Running the Micro Integrator on Containers.
- To customize this example for your own scenario, see File Connector Configuration documentation for all operation details of the connector.