Uploading large files to AWS S3 over API Gateway

Serverless is amazing when you don’t know what your server load will be and you pay only when your function runs.

More details on serverless here:


The Problem

Although it’s popular and used widely there are always some pros and cons.

One such con is API Gateway has a hard timeout of 30seconds. So, your API must return a response within 30 seconds, or else it’s going to fail.

What If you are creating an API to upload a file of maybe 30MB to AWS S3 using API Gateway?

There’s a high chance it will take a bit more than 30seconds and your API will eventually fail.

So, what’s the solution then? 🤔

We can use AWS S3 presigned URL to upload files to S3 without the hard time limit of 30seconds.


The Solution – Creating a simple AWS Lambda API to create presigned URLs for uploading.

Architecture

There’s already a blog post to show how to create a serverless API using AWS console which can be found below:

The serverless JavaScript code to create presigned URLs:

const AWS = require('aws-sdk');
const s3 = new AWS.S3({
  signatureVersion: 'v4'
});
const bucketName = process.env.S3_BUCKET_NAME; 
exports.handler = async (event, context, callback) => {
    const expiryInSeconds= 60 * 5;
    const body = JSON.parse(event.body);
    const key = 'files' + '/' + body['file_name'];
    
    const url = s3.getSignedUrl('putObject', {
        Bucket: bucketName,
        Key: key,
        Expires: expiryInSeconds,
    });
  
    const response = {
        statusCode: 200,
        headers: {
          'Access-Control-Allow-Origin': '*',
          'Access-Control-Allow-Credentials': true,
        },
        body: JSON.stringify({ filename: body['file_name'],  presigned_url: url}),
    };

    callback(null, response);
};

Here, we create an instance of AWS S3 and use a simple method called getSignedUrl() to generate resigned URLs.

On connecting it to API Gateway, we send a POST API call with the body having file_name as key.

Let’s see the deployed API in action

We pass file_name as car.jpg which creates a presigned_url for us to use.

Next, we copy the resigned URL and perform a PUT call as below:

We select the file(car.jpg) as binary and send a PUT request which response with 200 on success.

Now let us check s3 if the file is created:


So, we were successfully able to create a photo upload API with serverless 👌

Hope you liked it. Cheers 🍻

 

Leave a Reply

Your email address will not be published. Required fields are marked *