Image Compression in S3 Bucket using Lambda, Node.js, and CloudFront

In today's digital world, images play a crucial role in enhancing user experience on websites and applications. Take the example of Instagram where millions of images are shared by users, it would be impossible to deliver the users a smooth experience if the images shared are not processed and optimised by Instagram. Different users can upload images in a variety of formats and sizes based on their smartphone/camera devices. If Instagram starts taking these images as it is on its server and starts serving users these same images then it would be a huge problem in user experience and put a heavy toll on its system because these high-resolution images can significantly impact page load times, leading to slower performance and not to miss the storage cost of all these high-resolution images. To tackle this challenge, image compression techniques come to the rescue. Instagram has put in place a well-defined policy of how the images are handled to make sure the app has a smooth user experience for all. In this article, we will explore how to implement image compression in an S3 bucket using AWS Lambda and Node.js, enabling faster loading times and optimizing storage space.

Prerequisites: Before we dive into the implementation, make sure you have the following prerequisites in place to implement this solution:

1. An AWS account with appropriate permissions to create and configure Lambda functions and S3 buckets.

2. Node.js and npm (Node Package Manager) are installed on your local machine.

The Solution

Our solution will allow a client-side app to upload images to an S3 Bucket, once the image is uploaded to S3 Bucket, it will trigger a Lambda function which will compress and resize the image using a Node.js library and store it in a separate S3 Bucket from where the final image will be served to end users using CloudFront.

Let's break down the different components used in this solution -

1. The client app - It can be a web app or mobile app that will upload an image to S3 Bucket.

2. S3 Buckets - We will have two separate S3 Buckets, one for uploading images from the client app (we will refer to this bucket as - Upload Bucket) and another one for storing compressed images (we will refer to this bucket as - Output Bucket) and serving these images to end users. I have explained the reason for using two buckets further in this article.

3. Lambda Function - The Lambda function is responsible for handling image compression. It uses the AWS SDK and the sharp library in Node.js to retrieve the original image from Upload Bucket, compress it, and upload the compressed image to the Output Bucket. The function is triggered automatically upon the upload event in Upload Bucket.

4. CloudFront CDN - To efficiently serve the compressed images to users, CloudFront, the AWS content delivery network, is utilized. CloudFront caches the compressed images in edge locations located worldwide, bringing the images closer to end-users for faster delivery and reducing the load on the storage bucket. CloudFront also provides additional benefits such as DDoS protection, SSL/TLS termination, and content caching optimizations.

Why Two Separate S3 Buckets?

Implementation

Step 1: Set up the Upload and Output Buckets: The first step is to create two S3 buckets: one for uploading the original images and another for storing the compressed images.

Step 2: Set up a Lambda Function: Next, we need to create an AWS Lambda function that will handle the image compression.

Next, we will utilise the Event Triggers available for Lambda. One thing I love about Lambda is that it allows a plethra of Event Triggers not only from AWS services but also some other third party services which we can use to invoke the Lambda function and build amazing solutions. We will add an S3 event trigger for this Lambda and select the Upload Bucket as the event source we created earlier and choose the appropriate event type as - All object create events. Now, whenever an image gets uploaded to the Upload Bucket, this Lambda will be invoked.

Step 3: Implement Image Compression Logic:

We will use Node.js and Sharp Library to compress the image. Sharp is a very popular library in Node.js for server-side image processing and provides a range of options to resize and convert images to different formats. We will upload this code for the handler function to the created Lambda function and this will do all the job for us.

const AWS = require('aws-sdk');
const sharp = require('sharp')

exports.handler = async (event) => {
  const s3 = new AWS.S3();

  // Retrieve the original image from the S3 bucket (Upload)
  // The event received by this Lambda from S3 Bucket contains the Object Key
  const params = {
    Bucket: event.Records[0].s3.bucket.name,
    Key: event.Records[0].s3.object.key
  };
  const originalImage = await s3.getObject(params).promise();

  // Compress the image using sharp
  // You can resize and convert to the desired format as per use case
  const compressedImage = await sharp(originalImage.Body)
    .resize(800) // Set the desired dimensions
    .toBuffer();

  // Upload the compressed image to the Output Bucket
  // You can change the Key name as per your use case
  const uploadParams = {
    Bucket: 'your-output-bucket-name',
    Key: `compressed/${event.Records[0].s3.object.key}`,
    Body: compressedImage
  };
  await s3.putObject(uploadParams).promise();

  return {
        success: true,
        message: 'Image compression complete!'
  };
};

Step 4: Set up CloudFront Distribution: To leverage CloudFront for fast and efficient image delivery, we will set up a CloudFront Distribution and add our Output Bucket as the origin for this distribution. Instead of directly serving our compressed images from S3 we will be serving these through CloudFront. This makes sure our users get optimized images super fast and their experience is smooth.

Conclusion

In this article, we explored how to implement image compression in an S3 bucket using AWS Lambda and Node.js. By reducing the file size of images, we can improve page load times, enhance user experience, and optimize storage space. Image compression and resizing factors can be decided based on specific use cases. Overall, it's all about smooth user experience, storage costs and optimizing the overall system architecture.