Dalmeng's Dev.

mefriend

AWS Cost Optimization by Reducing Image Cost

—

by

Dalmeng
in

– Overview

Hello, I’m a lead developer at MeFriend.ai. Our service supports +5,000 characters, which means we need to deliver a high volume of images quickly and efficiently.

MeFriend.ai – Click Here!

While analyzing AWS costs, we found that CloudFront was using more budget than we had anticipated.

In our focus on rapid development, we hadn’t optimized AWS costs in several areas, leading to significant daily expenses of $15 in CloudFront.

Our service currently stored about 3GB of character images, and with over 20 images needing to load quickly even on the main page, we decided to reduce the size of images stored in S3.

However, simply reducing image sizes would indeed lower costs but at the expense of image quality, which would affect user experience—a trade-off we wanted to avoid.

To address this, we implemented an image tiering approach, where images are served in different quality tiers based on the context in which they’re needed. This is often called Responsive Image Serving or Multi-resolution Image Serving, but we decided to name our approach “Image Tier-based Serving.”

We divided the images into three main tiers based on the common sizes used in our view:

  • Small (90×90)
  • Middle (252×252)
  • Original (560×560)

Now let’s dive in how we can we reduce and optimize AWS CloudFront cost.

This article follows the previous article (Building an Image Server).

Easily Building a Image Server with S3 and CloudFront

– Steps

  • How are we going to make it?
  • Convert images to fit tier
  • Upload images to S3
  • Update image urls in Database
  • Serve images based on tier
  • Result

– How are we going to make it?

WebP Format

All images stored in S3 were JPG, PNG format. We know that these formats have lower image compression efficiency than WebP format.

So, first task was to convert these images’ format to WebP format. By doing that, we can serve same quality images with more lower cost.

Separating Image Tier

Although Basic image sizes are reduced by converting WebP, We realized it was wasteful to display both very small icons and large images at the same resolution. Dividing images into tiers based on size could be a bit tricky, but we believed that it would significantly help reduce costs.

To handle the image transformations, I used the open-source software ImageMagick, enabling fast and straightforward conversion across all images.

– Convert image to fit tier

1. Download all image files from S3

First, we needed to download all image files from the bucket. You can use the AWS CLI to download files quickly and easy.

aws s3 cp s3://mai-image/profile ./profiles –recursive

Notes

  • Before running the code, you need to grant permission to access S3.
  • You can set up AWS credentials using the aws configure command.

2. Convert images using ImageMagick

To simplify the conversion process, I organized the directory structure as follows:

  • The folders small, middle, and original are designated for the converted images. (These three folders are currently empty.)
  • All files downloaded from AWS are in the profiles folder.
.
├── profiles/
├── small/
├── middle/
└── original/

In profiles folder, we were able to convert all images easily and quickly using below shell script.

for file in *.png; do
  filename=$(basename "$file" .png)
  
  magick "$file" -resize 90x90 -quality 85 "../small/${filename}.webp"
  
  magick "$file" -resize 252x252 -quality 85 "../middle/${filename}.webp"
  
  magick "$file" -resize 560x560 -quality 85 "../original/${filename}.webp"
done

This script adjusts the resolution and converts the format to WebP.

To further reduce file size, I set the quality to 85% of the original image. I found that 85% provides a significant reduction in file size without noticeably impacting visual quality.

As s result, from a total image size of 3GB,

  • Small: 6MB
  • Middle: 35MB
  • Original: 160MB

we were able to reduce the overall size to about 6% of the original.

– Upload Images to S3

aws s3 cp ../small s3://mai-image/profile/small --recursive

aws s3 cp ../middle s3://mai-image/profile/middle --recursive

aws s3 cp ../original s3://mai-image/profile/original --recursive

Image uploading is similar to image downloading using AWS CLI.

– Update Image URLs in DB

The image conversion is complete, but we need to update character profile images to point to the converted images. Since we use MongoDB as our main database, modifying the data structure was simple, and we were able to handle the update with a some mongosh script.

Previously, the profile_image field was a string pointing to a single image URL like this:

profile_image: “https://cdn.mefriend.ai/mai-image/profile/1.png”

With the introduction of three tiers, we decided to change the structure of profile_image to the following like this:

profile_image: {
    small: "https://cdn.mefriend.ai/mai-image/profile/small/1.png",
    middle: "https://cdn.mefriend.ai/mai-image/profile/middle/1.png",
    original: "https://cdn.mefriend.ai/mai-image/profile/original/1.png",
}

1. Define Utility Function

function transformProfileImage(profileImage) {
    if (!profileImage || typeof profileImage !== "string") {
        return {
            small: `/static/default.png`,
            middle: `/static/default.png`,
            original: `/static/default.png`,
        };
    }

    const matchTest = profileImage.match(/https:\/\/cdn\.mefriend\.ai\/mai-image\/([^\/]+)\.(jpg|png|jpeg)/);
    if (matchTest) {
        const id = matchTest[1];
        return {
            small: `https://cdn.mefriend.ai/mai-image/profile/small/${id}.webp`,
            middle: `https://cdn.mefriend.ai/mai-image/profile/middle/${id}.webp`,
            original: `https://cdn.mefriend.ai/mai-image/profile/original/${id}.webp`,
        };
    }

    return {
        small: `/static/default.png`,
        middle: `/static/default.png`,
        original: `/static/default.png`,
    };
}

Because name of image file has not changed, image name converting was very simple. Mongosh supports custom function definition, so we were able to define utility function that converts profile image url to the format that we intend.

Logic is very simple: extracting image name using regex, and make a new image URL following our image tier policy that we had defined before.

2. Update Image URLs using Utility Function

db.character.updateMany(
    { profile_image: { $type: "string" } },
    [
        {
            $set: {
                profile_image: {
                    $function: {
                        body: transformProfileImage.toString(),
                        args: ["$profile_image"],
                        lang: "js"
                    }
                }
            }
        }
    ]
);

If the number of data and the number of updated data are the same, it indicates that the image URL conversion was successful.

– Serve images based on tier

function ResponsiveProfileImage(profileImage, tier) {
    const ret = profileImage?.[tier] || "/static/pic/default.png";
    return ret;
}

export default ResponsiveProfileImage;

We created a function to return images according to their tier, allowing us to display images on the frontend as shown below, based on the appropriate tier.

<img src={ResponsiveProfileImage(character?.profile_image , "small")}
/>

– Result

Each image was originally around 1–2MB, but we were able to dramatically reduce the size to 3KB, 13KB, and 40KB for each tier.

small tier image

middle tier image

original tier image

In terms of cost, we were spending around $15 on CloudFront, but after applying tier-based image serving, we reduced it to $1–2, achieving around 90% in savings.

Developing a service and running it are two entirely different things. Even if a feature works well, it won’t be sustainable if it’s too expensive or inefficient to operate in the long term.

AWS cost optimization is a challenge for many developers. So it was a so valuable experience, as it’s a crucial aspect of long-term service management. This process has motivated us to not only optimize CloudFront but also identify and reduce other unnecessary costs across the board.

Thank you for reading this detailed post!

(The bucket resources and database data mentioned are examples created for illustrative purposes and do not reflect our actual environment.)