Skip to content

Working with Amazon S3 Object Service

Ballerina is an open-source programming language that empowers developers to integrate their system easily with the support of connectors. In this guide, we are mainly focusing on connecting to the Amazon Simple Storage Service API to create, store, download, and use data with other services.

The wso2/amazons3 module allows you to perform the following operations.

  • Create Bucket
  • List Buckets
  • Delete Bucket
  • Create Object
  • List Objects
  • Get Object
  • Delete Object

This example explains how to use the S3 client to connect with the Amazon S3 instance and to create a Amazon S3 object, get objects in a bucket, get object data and delete an object.

You can find other integrations modules from wso2-ballerina GitHub organization.

What you'll build

This application connects with the Amazon S3 API and creates a new object in an Amazon S3 bucket, list the available objects in the bucket, display the object content and delete a specified object in the bucket.

working with Amazon S3 Object service

Before you begin

Get the code

Pull the module from Ballerina Central using the following command.

ballerina pull wso2/integration_with_amazon_s3_object

Alternately, you can download the ZIP file and extract the contents to get the code.

Download ZIP

Implementation

A Ballerina project is created for the integration use case explained above. Please follow the steps given below to create the project and modules. You can learn about the Ballerina project and modules in this guide.

1. Create a new project.

$ ballerina new working-with-object-service

2. Create a module.

$ ballerina add integration_with_amazon_s3_object

The project structure is created as indicated below.

working-with-object-service
    ├── Ballerina.toml
    └── src
        └── integration_with_amazon_s3_object
            ├── Module.md
            ├── main.bal
            ├── resources
            └── tests
                └── resources

3. Set up credentials for accessing Amazon S3

  • Visit Amazon S3 and create an Amazon S3 account.

  • Create a new access key, which includes a new secret access key. - To create a new secret access key for your root account, use the security credentials page. Expand the Access Keys section, and then click Create New Root Key.

  • To create a new secret access key for an IAM user, open the IAM console. Click Users in the Details pane, click the appropriate IAM user, and then click Create Access Key on the Security Credentials tab.

  • Download the newly created credentials, when prompted to do so in the key creation wizard.

4. Add project configurations file

Add the project configuration file by creating a ballerina.conf file under the root path of the project structure. This file should have following configurations. Add the obtained SAmazon S3 configurations to the file.

ACCESS_KEY_ID="<Amazon S3 key ID>"
SECRET_ACCESS_KEY="<Amazon S3 secret key>"
REGION="<Amazon S3 region>"
BUCKET_NAME="<Amazon S3 bucket name>"
TRUST_STORE_PATH="<Truststore file location>"
TRUST_STORE_PASSWORD="<Truststore password>"

5. Write the integration

Open the project with VS Code. The integration implementation is written in the src/integration_with_amazon_s3_object/main.bal file.

import ballerina/config;
import ballerina/http;
import ballerina/log;
import ballerina/stringutils;

import wso2/amazons3;

// Constants for error codes and messages.
const string ERROR_CODE = "Sample Error";
const string RESPOND_ERROR_MSG = "Error in responding to client.";
const string CLIENT_CREATION_ERROR_MSG = "Error while creating the AmazonS3 client.";
const string PAYLOAD_EXTRACTION_ERROR_MSG = "Error while extracting the payload from request.";
const string PAYLOAD_CONVERTION_ERROR_MSG = "Error occured while converting bucket list to json";
const string OBJECT_CREATION_ERROR_MSG = "Error while creating object on Amazon S3.";
const string OBJECTS_RETRIEVING_ERROR_MSG = "Error while listing objects on bucket : ";
const string OBJECT_DELETION_ERROR_MSG = "Error while deleting object from Amazon S3.";
const string INVALID_PAYLOAD_MSG = "Invalid request payload";

// Create Amazons3 client configuration with the above accesskey and secretKey values.
amazons3:ClientConfiguration amazonS3Config = {
    accessKeyId: config:getAsString("ACCESS_KEY_ID"),
    secretAccessKey: config:getAsString("SECRET_ACCESS_KEY"),
    region: config:getAsString("REGION"),
    clientConfig: {
        http1Settings: {chunking: http:CHUNKING_NEVER}
    }
};

// Create AmazonS3 client with the above amazonS3Config.
amazons3:AmazonS3Client|amazons3:ConnectorError amazonS3Client = new(amazonS3Config);

@http:ServiceConfig {
    basePath: "/amazons3"
}

service amazonS3Service on new http:Listener(9090) {
    @http:ResourceConfig {
        methods: ["POST"],
        path: "/imageStore/{bucketName}/{objectName}"
    }
    // Function to create a new object into an existing bucket.
    resource function createObject(http:Caller caller, http:Request request, string bucketName, string objectName) {
        // Assign amazonS3Client global variable to a local variable
        amazons3:AmazonS3Client|amazons3:ConnectorError s3Client = amazonS3Client;
        if (s3Client is amazons3:AmazonS3Client) {
            // Define new response.
            http:Response backendResponse = new();
            // Extract the object content from request payload.
            string|xml|json|byte[]|error objectContent = extractRequestContent(request);
            if objectContent is error {
                // Send the error response.
                createAndSendErrorResponse(caller, <@untainted> <string>objectContent.detail()?.message,
                                PAYLOAD_EXTRACTION_ERROR_MSG);
            } else {
                // Invoke createObject remote function from amazonS3Client.
                error? response = s3Client->createObject(<@untainted> bucketName, <@untainted> objectName,
                                                    <@untainted> objectContent);
                if (response is amazons3:ConnectorError) {
                    // Send the error response.
                    createAndSendErrorResponse(caller, <@untainted> <string>response.detail()?.message,
                                    OBJECT_CREATION_ERROR_MSG);
                } else {
                    // If there is no error, then object created successfully. Send the success response.
                    backendResponse.setTextPayload(<@untainted> string `${objectName} created on Amazon S3 bucket : 
                            ${bucketName}.`, contentType = "text/plain");
                    respondAndHandleError(caller, backendResponse, RESPOND_ERROR_MSG);
                }
            }
        } else {
            // Send the error response.
            createAndSendErrorResponse(caller, <string>s3Client.detail()?.message, CLIENT_CREATION_ERROR_MSG);
        }
    }

    @http:ResourceConfig {
        methods: ["GET"],
        path: "/imageStore/{bucketName}/{objectName}"
    }
    // Function to get object.
    resource function getObject(http:Caller caller, http:Request request, string bucketName, string objectName) {
        // Assign amazonS3Client global variable to a local variable
        amazons3:AmazonS3Client|amazons3:ConnectorError s3Client = amazonS3Client;
        if (s3Client is amazons3:AmazonS3Client) {
            // Define new response.
            http:Response backendResponse = new();
            //Get the response content type from query params.
            string? params = request.getQueryParamValue("responseContentType");
            string responseContentType = <string> params;

            // Invoke getObject remote function from amazonS3Client.
            var response = s3Client->getObject(<@untainted> bucketName, <@untainted> objectName);
            if (response is amazons3:S3Object) {
                // S3Object will be returned on success.
                // Set the object content to the payload with the expected content type.
                backendResponse.setBinaryPayload(<@untainted> <byte[]>response["content"], contentType = <@untainted> responseContentType);
                respondAndHandleError(caller, backendResponse, RESPOND_ERROR_MSG);
            } else {
                // Send the error response.
                createAndSendErrorResponse(caller, <@untainted> <string>response.detail()?.message,
                                 "Error while creating object on Amazon S3.");
            }
        } else {
            // Send the error response.
            createAndSendErrorResponse(caller, <string>s3Client.detail()?.message, CLIENT_CREATION_ERROR_MSG);
        }
    }

    @http:ResourceConfig {
        methods: ["GET"],
        path: "/imageStore/{bucketName}"
    }
    // Function to list objects.
    resource function listObjects(http:Caller caller, http:Request request, string bucketName) {
        // Assign amazonS3Client global variable to a local variable
        amazons3:AmazonS3Client|amazons3:ConnectorError s3Client = amazonS3Client;
        if (s3Client is amazons3:AmazonS3Client) {
            // Define new response.
            http:Response backendResponse = new();
            // Invoke listObjects remote function from amazonS3Client.
            var response = s3Client->listObjects(<@untainted> bucketName);
            if (response is amazons3:ConnectorError) {
                // Send the error response.
                createAndSendErrorResponse(caller, <@untainted> <string>response.detail()?.message,
                                OBJECTS_RETRIEVING_ERROR_MSG + "${bucketName}.");
            } else {
                // If there is no error, then object list retrieved successfully. Send the object list.
                var list = json.constructFrom(response);
                if (list is json) {
                    backendResponse.setJsonPayload(<@untainted> list, contentType = "application/json");
                    respondAndHandleError(caller, backendResponse, RESPOND_ERROR_MSG);
                } else {
                    createAndSendErrorResponse(caller, <@untainted> <string>list.detail()?.message,
                                PAYLOAD_CONVERTION_ERROR_MSG);
                }
            }
        } else {
            // Send the error response.
            createAndSendErrorResponse(caller, <string>s3Client.detail()?.message, CLIENT_CREATION_ERROR_MSG);
        }
    }

    @http:ResourceConfig {
        methods: ["DELETE"],
        path: "/imageStore/{bucketName}/{objectName}"
    }
    // Function to delete object.
    resource function deleteObject(http:Caller caller, http:Request request, string bucketName, string objectName) {
        // Assign amazonS3Client global variable to a local variable
        amazons3:AmazonS3Client|amazons3:ConnectorError s3Client = amazonS3Client;
        if (s3Client is amazons3:AmazonS3Client) {
            // Define new response.
            http:Response backendResponse = new();
            amazons3:ConnectorError? response = s3Client->deleteObject(<@untainted> bucketName, <@untainted> objectName);
            if (response is amazons3:ConnectorError) {
                // Send the error response.
                createAndSendErrorResponse(caller, <@untainted> <string>response.detail()?.message,
                                        OBJECT_DELETION_ERROR_MSG);
            } else {
                // If there is no error, then object deleted successfully. Send the success response.
                backendResponse.setTextPayload(<@untainted> string `${objectName} deleted from Amazon S3 bucket : ${bucketName}.`,
                                        contentType = "text/plain");
                respondAndHandleError(caller, backendResponse, RESPOND_ERROR_MSG);
            }
        } else {
            // Send the error response.
            createAndSendErrorResponse(caller, <string>s3Client.detail()?.message, CLIENT_CREATION_ERROR_MSG);
        }
    }
}

// Function to extract the object content from request payload
function extractRequestContent(http:Request request) returns @tainted string|xml|json|byte[]|error {
    string contentTypeStr = request.getContentType();
    if (stringutils:equalsIgnoreCase(contentTypeStr, "application/json")) {
        var jsonObjectContent = request.getJsonPayload();
        if (jsonObjectContent is json) {
            return jsonObjectContent;
        }
    }
    if (stringutils:equalsIgnoreCase(contentTypeStr, "application/xml")) {
        var xmlObjectContent = request.getXmlPayload();
        if (xmlObjectContent is xml) {
            return xmlObjectContent;
        }
    }
    if (stringutils:equalsIgnoreCase(contentTypeStr, "text/plain")) {
        var textObjectContent = request.getTextPayload();
        if (textObjectContent is string) {
            return textObjectContent;
        }
    }
    var binaryObjectContent = request.getBinaryPayload();
    if (binaryObjectContent is byte[]) {
        return binaryObjectContent;
    } else {
        error err = error(ERROR_CODE, message = INVALID_PAYLOAD_MSG);
        return err;
    }
}

// Function to create the error response.
function createAndSendErrorResponse(http:Caller caller, string errorMessage, string respondErrorMsg) {
    http:Response response = new;
    //Set 500 status code.
    response.statusCode = 500;
    //Set the error message to the error response payload.
    response.setPayload(<string> errorMessage);
    respondAndHandleError(caller, response, respondErrorMsg);
}

// Function to send the response back to the client and handle the error.
function respondAndHandleError(http:Caller caller, http:Response response, string respondErrorMsg) {
    // Send response to the caller.
    var respond = caller->respond(response);
    if (respond is error) {
        log:printError(respondErrorMsg, err = respond);
    }
}

Testing

First let’s build the module. Navigate to the project root directory and execute the following command.

$ ballerina build integration_with_amazon_s3_object

This creates the executables. Now run the ntegration_with_amazon_s3_object.jar file created in the above step.

$ java -jar target/bin/integration_with_amazon_s3_object.jar
You will see the following log upon the successful invocation of the service.

[ballerina/http] started HTTP/WS listener 0.0.0.0:9090

1. Testing the create Object service

(I) JSON Content

Create a file called content.json with following JSON content:

{
    "name": "John Doe",
    "dob": "1940-03-19",
    "ssn": "234-23-525",
    "address": "California",
    "phone": "8770586755",
    "email": "johndoe@gmail.com",
    "doctor": "thomas collins",
    "hospital": "grand oak community hospital",
    "cardNo": "7844481124110331",
    "appointment_date": "2025-04-02"
}
- Invoke the following curl request to create a new object in the newly created bucket.
curl -v -X POST --data @content.json http://localhost:9090/amazons3/imageStore/firstbalbucket/firstObject.json --header "Content-Type:application/json"
You see the response as follows:
firstObject.json created on Amazon S3 bucket : firstbalbucket.

(II) Binary Content

Let's upload an image (sample.jpg) to the s3 bucket we created above. - Invoke the following curl request to create a new object in the newly created bucket.

curl -v -X POST http://localhost:9090/amazons3/imageStore/firstbalbucket/image.jpg -H 'Content-Type: image/jpg' -T sample.jpg -H "Expect:"
You see the response as follows:
image.jpg created on Amazon S3 bucket : firstbalbucket.

2. Testing the list Object service

  • Invoke the following curl request to list objects in a bucket.
curl -X GET http://localhost:9090/amazons3/imageStore/firstbalbucket

3. Testing get Object service

(I) JSON Content

  • Set the responseContentType as application/json to retrieve a JSON object and invoke the following curl request to get the newly created object.
    curl -v -X GET http://localhost:9090/amazons3/imageStore/firstbalbucket/firstObject.json?responseContentType=application/json
    You see the response as follows:
    {
        "name": "John Doe",
        "dob": "1940-03-19",
        "ssn": "234-23-525",
        "address": "California",
        "phone": "8770586755",
        "email": "johndoe@gmail.com",
        "doctor": "thomas collins",
        "hospital": "grand oak community hospital",
        "cardNo": "7844481124110331",
        "appointment_date": "2025-04-02"
    }

(II) Binary Content

  • Set the responseContentType as image/jpg and use following URL to open newly created image on browser.

    http://localhost:9090/amazons3/imageStore/firstbalbucket/image.jpg?responseContentType=image/jpg

  • You can use the following URL to download the newly created image.

    http://localhost:9090/amazons3/firstbalbucket/image.jpg?responseContentType=application/octet-stream

4. Testing delete Object service

  • Invoke the following curl request to delete the above object.
    curl -v -X DELETE http://localhost:9090/amazons3/imageStore/firstbalbucket/firstObject.json
    You see the response as follows:
    firstObject.json deleted from Amazon S3 bucket : firstbalbucket.
Top