Chaitanya Baranwal
Exploring and learning.

My Experience with using DigitalOcean Spaces in Javascript

I write this article mainly because even though there are guides out there which show how to use DigitalOcean Spaces with Node.js, I could not find any which clearly laid out what I needed to do for the project I had in mind. However, at least in my case, software development was not rocket science, since I was able to finish my project with the help of StackOverflow and this Medium article.

A little bit of context

There exists an app (created using p5.js) which allows children with limited ability to draw lines and shapes using their eyes. For the purposes of the exhibition, design a service where people can draw images using their eyes on a laptop and submit the image, which is then pushed on to a central screen where all the submitted images are being displayed simultaneously.

Owing to my limited knowledge in hardware and file transfer technologies, connecting the two computers and transferring files offline between them was not the first thing that came to my mind. What immediately struck me was that I could download the drawn image, transfer it to a central server, and the second website fetches images from the central server to display them on the central screen. Some of the options for the central server were DigitalOcean, Amazon S3 or Google Cloud, but since I had worked with DO droplets before (and also because Github Education Pack gives free DO credits), I decided to check out DigitalOcean Spaces. Disclaimer that this was my first experience using a lot of technologies, ranging from DO Spaces to ExpressJS, so I hacked together a lot of things. Apologies if it frustrates all the software engineers out there. :)

About DigitalOcean spaces

DigitalOcean Spaces is an S3-compatible object storage platform, which means one can programmatically upload, retrieve and delete images from the storage platform using a familiar JavaScript library called aws-sdk. Given you have created a DigitalOcean Space already with all the necessary configurations, you can place the config parameters in a file called config.js as a JSON object:

const config = {
    region: 'sgp',
    accessKeyId: xxx,
    secretAccessKey: xxx,
    bucketName: 'test-bucket'
    baseUrl: xxx,
    maxImages: 20
}

For the purposes of my project, I mostly needed to upload, fetch and get the total number of images (to name the next image accordingly, more on this later), but you can refer to the previously mentioned Medium article for a more comprehensive list of stuff you can do using Spaces.

Importing the aws-sdk library is very simple. In Node.js, one can simply do:

const AWS = require('aws-sdk');
const config = require('./config');

If one is using JavaScript embedded in HTML, one can import the aws-sdk library using:

<!-- Remember to put the appropriate version for aws-sdk -->

<script src="https://sdk.amazonaws.com/js/aws-sdk-<version>.min.js"></script>
<script type="text/javascript" src="config.js"></script>

Setting up Spaces in JavaScript:

// declare API object
var spacesEndpoint = new AWS.Endpoint(config.region + '.digitaloceanspaces.com');
var s3 = new AWS.S3({
    endpoint: spacesEndpoint,
    accessKeyId: config.accessKeyId,
    secretAccessKey: config.secretAccessKey
});

// define bucket name
var params = { Bucket: config.bucketName }

Performing standard file-storage operations in Spaces

Since I needed to perform multiple Spaces functions for my project, I am going to divide the following sections by each function I needed to perform, and the problems I faced in each.

Another disclaimer that the stuff mentioned below is very hacky, and might not abide by good software engineering principles.

Getting the number of images in a bucket

Since the central screen could not display all the images, I instead chose to display the 20 latest images on the screen. Because of this, I needed to the fetch the last 20 images from the bucket, and I named the image files in a way which enabled me to do that. The image name would basically be something like image (<number>).png, and because of this I needed to get the number of images in the bucket to select the latest images by number.

Initially I relied on the autonaming function, where a browser adds a number to the file on finding that another file of the same name exists in the directory. However, I soon realised that after 100 copies, browsers start storing files using timestamp, which is why I needed to provide the image number while downloading the files. To do so, I had to get the number of files in the bucket. Doing the following would not work (I think), because Spaces only lists 1000 objects at once:

s3.listObjectsV2(params, function(err, data) {
    return data.Contents.length;
});

Moreover, maintaining a counter variable would not work because aws-sdk uses async callback functions to upload files, which ensured that I could not update a counter directly inside the callback function. This is how I ended up getting the length of a bucket, owing to a StackOverflow answer:

// function to get total number of images
const getBucketLength = (params, count = 0) => new Promise((resolve, reject) => {
    s3.listObjectsV2(params).promise()
      .then(({Contents, IsTruncated, NextContinuationToken}) => {
        count += Contents.length;
        !IsTruncated ? resolve(count) : resolve(getBucketLength(Object.assign(params, {ContinuationToken: NextContinuationToken}), count));
      })
      .catch(reject);
});

What this does is use a recursive async function, and update the count parameter till the file list is truncated. As soon as the file list is not truncated (meaning we have reached the end of the file list), the final count is returned. Promises are used because of async functions, which will not give a value unless it is needed.

Uploading Images

The idea was that images will download to a specific folder in the project directory, following which the aws-sdk library will read these images and upload them to the object storage.

The problem which I faced now was AWS tried to fetch the image before it was even downloaded, which led to an error. As a very hacky solution, what I instead did was get the bucket length and and download the image with the appropriate image number, get bucket length again (in another file which handles upload) and look in the directory for the image with the corresponding number to upload. Here is the code I ended up with:

// 'Submit image' button pressed
app.post('/upload', function(request, response) {

    getBucketLength(params).then(function(length) {

        // timeout present to account for the lag between downloading image and attempting upload
        setTimeout(function() {
            let filePath = 'images/image' + (length == 0 ? '' : ` (${length})`) + '.png';
            let params = {
                Bucket: config.bucketName,
                Key: path.basename(filePath),
                Body: fs.createReadStream(filePath),
                ACL: 'public-read'
            };
            // upload the image onto bucket
            s3.upload(params, function(err, data) {
                if (err) {
                    console.log('Error: ' + err);
                    return response.redirect('/error');
                }
                console.log('Success!');
                return response.redirect('/');
            });
        }, 3500);

    });

});

Now that I think of it, I could have updated a counter variable in a global file whenever the Submit Image button was pressed.

Fetching images from Spaces

Since the images I were fetching were to be displayed on an HTML page, I did not need the download the images, I could simply use the file link as the src attribute for an img tag. However, I needed to modify my Space’s CORS policy (and make my website ridiculously vulnerable to CSRF attacks) for this to work out. This is what my loadImages() function did:

// function to load images
function loadImages() {
  getBucketLength(params).then(function(length) {
    // number of images to be shown in page
    const img_count = Math.min(length, config.maxImages);

    for (let i = 1; i <= img_count; i++) {
      let img = document.createElement('img');
      const id = `${img_count - i}`;
      img.id = id;
      img.src = config.baseUrl + (length - i == 0 ? '' : ` (${length - i})`) + '.png';
      document.getElementById('photos').appendChild(img);
    }

  });
}

Final project

And that was it! Apart from the eye-tracking code, most of the substantial parts of my project have been highlighted above. Since it was my first freelance project, I actually went to the exhibition and checkout out what I did in real life. It was an immensely gratifying feeling to see that a lot of children had fun with my project and found it cool. :)

Here are some pictures of the final project (added the googly eyes for an extra bit of fun):

App screenshot App screenshot

The source code for the project can be found here: https://github.com/chaitanyabaranwal/Rainbow_Superhero. Hope you liked the read!