File Connector Example

File Connector can be used to perform operations in the local file system as well as in a remote server such as FTP and SFTP.

What you'll build

This example describes how to use the File Connector to write messages to local files and then read the files. Similarly, the same example can be configured to communicate with a remote file system (i.e FTP server) easily. The example also uses some other mediators coming with WSO2 EI to manipulate messages.

An API is exposed to accept XML messages (employee information). When a message is received, it is converted to a CSV message and then stored to a property. A check is done to see if the CSV file (with the same information) exists in the file system. If it does not exist, the connector creates a CSV file with CSV headers included. Then, the connector appends the new CSV entries in the current message to the CSV file. The connector then reads the same CSV file and converts the information back to XML and responds to the client.

Setting up the environment

Create a folder in your local file system with read and write access. This will be your working directory. In this example, it is /Users/hasitha/temp/file-connector-test/dataCollection.

Note

If you set up a FTP server, SFTP server, or a Samba server, do the required configurations and select a working directory. Save the host, port, and security related information for future use.

Configure the connector in WSO2 Integration Studio

Follow these steps to set up the Integration Project and the Connector Exporter.

{!references/connectors/importing-connector-to-integration-studio.md!}

Creating the Integration Logic

  1. Create a new integration project. Be sure to enable a Connector Exporter.

    create project

  2. Create an API named TestAPI with the /fileTest context. This API will accept employee information.

    create API

  3. In the default resource of the API, enable POST requests.

    select post method

  4. Add the Log mediator to the design canvas and configure a custom log that indicates when the API receives a message.

  5. Add the DataMapper mediator and configure it to transform the incoming XML message to a CSV message.
  6. Double-click the Datamapper mediator and add a new transform configuration called 'xmlToCsv'.

    new datamapper config

  7. Save the following content as an XML file. This will be the data input file.

    <test>
    <information>
        <people>
            <person>
                <name>Hasitha</name>
                <age>34</age>
                <company>wso2</company>
            </person>
            <person>
                <name>Johan</name>
                <age>32</age>
                <company>IBM</company>
            </person>
        </people>
    </information>
    </test>
  8. Load the input file into the Datamapper config view.

    load input

  9. Save the following content as a CSV file. This will be the data output file.

    Name,Age,Company
    Hasitha,34,wso2
    Johan,32,IBM
  10. Load the output CSV file into the datamapper config view.

    load output csv

  11. Configure the mapping as shown below. Each element in the input should be mapped to the respective element in the output.

    data mapping

  12. Specify the input as XML and output as CSV in the datamapper as shown below.

    datamapper input output config

  13. Add the Enrich mediator and configure it to save the output generated by the datamapper to a property named CONTENT.

    enrich - save payload

  14. Now, let's use the File connector to check if the file containing employee information already exists in the file system.

    1. Add the checkExist operation of the File connector to the canvas.
    2. Create a new file connection pointing to the working directory we already set up. Keep this as the File connection for the operation.

      working directory

    3. Configure the file path as /dataCollection/employees/employees.csv. This file will store the employee information.

      checkExist operation

  15. Add the Filter mediator to branch out the logic based on the result from the File connector’s checkExist operation.

    Note

    If the file does not exist, the File connector’s write operation (which we configure later) will create the file.

    1. Click the Filter mediator and define the filter condition as shown below.

      filter mediator

    2. Inside the “else” block, add the File Connector's write operation and configure it to write the static content of CSV file headers: “Name,Age,Company”.

      Note

      Be sure to append a new line at the end of the file. The Write Mode needs to be Create New.

      create new

  16. After the Filter mediator, use the Enrich mediator again to put back the saved payload into the message payload.

    enrich - put back payload

  17. Add the File connector’s write operation again and configure it to append the CSV message to the existing file. The Write Mode needs to be 'Append'.

    Note

    As we need the newest message on the top, always append to line number 2.

    append to file

  18. Add the File connector’s read operation and configure it to read the same CSV file.

    Note

    The file reading will start from line number 2. The content is read as a text message.

    read csv file

  19. Add the Datamapper mediator again and configure it to convert the CSV message (after reading) back to XML.

    1. Double-click the data mapper and add a new configuration called 'csvToXml'.

      output datamapper config

    2. This time, the mapping should be from CSV to XML.

      output datamapper dialog

  20. Finally, use the Respond mediator to send the transformed message to the API caller.

  21. Now, let's configure a fault sequence to generate an error message when an error occurs in the message flow.

    1. Creat a fault sequence with a Log mediator and Respond mediator.

      fault sequence

    2. Configure the Log mediator generate a custom error.

      error log

    3. Add the fault sequence to the API resource as its fault sequence.

{!references/connectors/exporting-artifacts.md!}

Deployment

Follow these steps to deploy the exported CApp in the Enterprise Integrator runtime.

{!references/connectors/deploy-capp.md!}

Testing

  1. Create a file called data.xml with the following payload.

    Note

    When you configuring this source parameter in the Windows operating system, set this property as <source>C:\\Users\Kasun\Desktop\Salesforcebulk-connector\create.txt</source>.

    <test>
    <information>
        <people>
            <person>
                <name>Hasitha</name>
                <age>34</age>
                <company>wso2</company>
            </person>
            <person>
                <name>Johan</name>
                <age>32</age>
                <company>IBM</company>
            </person>
            <person>
                <name>Bob</name>
                <age>30</age>
                <company>Oracle</company>
            </person>
            <person>
                <name>Alice</name>
                <age>28</age>
                <company>Microsoft</company>
            </person>
            <person>
                <name>Anne</name>
                <age>30</age>
                <company>Google</company>
            </person>
        </people>
    </information>
    </test>
  2. Invoke the API as shown below using the curl command.

    Info

    Curl Application can be downloaded from here.

    curl -H "Content-Type: application/xml" --request POST --data @data.xml http://10.100.5.136:8290/fileconnector/create
  3. Check the file system to verify that the CSV file has been created.

    file creation result

  4. If you invoke the API again with a different set of employees, the new employees will get appended to the same file. The response you receive will include all the employees that were added from both messages.

In this example, the File connector was used to create a file, write to a file, and to read a file. By blending these capabilities together with other powerful message manipulation features of WSO2 EI, it is possible to define a working scenario in minutes. The File connector has many more functionalities. Refer the File Connector reference guide for more information.

What's Next

Top