For a while, I've used Serverless functions to deploy my backend APIs, which has been going alright for me. What appealed to me with Serverless is that I didn't have (nor wanted) to manage my infrastructure. It was very easy to write a Serverless function and deploy it globally.

My main issue with Serverless is that the cold starts can be very slow. For example, I have a backend API that connects to a database, performs some basic logic, and returns a response. When the function starts for the first time (i.e. cold start), the request takes ~3-4 seconds to complete (subsequent requests take 100-400ms). My app doesn't have a large number of users, so this slow cold start time is very noticeable to them.

The second issue with serverless is how bad it scales, cost-wise. A popular article recently released by Amazon Prime showed how they saved 90% on server costs by moving parts of their infrastructure from serverless to a monolith.

Because of this, I decided to ditch the whole serverless platform and make my backend run on AWS Lightsail with a fully set up CI/CD Pipeline. Here's how you can do the same:

#1 - Creating an S3 Bucket

First of all, we need to create an S3 bucket, which is where our source code, built code, and build logs will be stored. This bucket will be used by both our AWS Lightsail servers and our CodePipeline to store/download our code.

Go to the AWS console (or use the CLI), and create an S3 bucket named myapp-codedeploy-bucket. It's fine to leave the default settings as they are (blocking all public access, ACLs disabled, etc.)

Since this bucket is private, we need to create an IAM user so that other services can safely access resources within the bucket.

Later, we will set up an AWS CodeDeploy agent on each of our Lightsail servers that automatically downloads the built source code from that S3 bucket and then runs our application.

#2 - Creating An IAM User

Before creating the user, we need to create a policy that allows access to our s3 bucket. Go to the IAM Policies tab in the AWS console and create a new policy. Paste the following in the JSON editor:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*"
            ],
            "Resource": [
                "arn:aws:s3:::myapp-codedeploy-bucket/*" //replace with your bucket's ARN
            ]
        }
    ]}

I named this policy myapp-s3bucket-policy. Now, we will create an IAM user and attach this policy to that user. Go to the IAM Users tab and create a new user.

In the "Set Permissions" tab, attach the previous policy we created (myapp-s3bucket-policy) to that user:

After creating the IAM User (which I named myapp-deploy-user), I lastly need to create an access key for that user. You can do that in the AWS console, or using the CLI as such:

$ aws iam create-access-key --user-name "myapp-deploy-user"

Save the output of this command, mainly the AccessKeyId and SecretAccessKey fields as we will need them later.

#3 - Creating The Lightsail Instance

Creating a Lightsail instance is simple, just go to the Lightsail dashboard and create an instance directly, then SSH into it. I created a $5 instance that runs on Amazon Linux 2023:

Before we install the codedeploy agent, create a YAML file that stores the credentials of the IAM user we previously created in the following directory: /etc/codedeploy-agent/conf/codedeploy.onpremises.yml.

$ sudo mkdir /etc/codedeploy-agent/
$ sudo mkdir /etc/codedeploy-agent/conf
$ sudo vim /etc/codedeploy-agent/conf/codedeploy.onpremises.yml

Paste the following contents into the file:

aws_access_key_id: <ACCESS_KEY>
aws_secret_access_key: <SECRET_ACCESS_KEY>
iam_user_arn: <IAM_USER_ARN>
region: <AWS_REGION>

Replace the field values with the equivalent fields relating to your region and the IAM User you previously created.

Now, we can install the codedeploy agent by running the following scripts:

$ sudo yum -y install ruby
$ wget https://aws-codedeploy-us-east-2.s3.amazonaws.com/latest/install
$ chmod +x ./install
$ sudo ./install auto

To see if the codedeploy agent is running on your Lightsail instance, run the following:

$ sudo service codedeploy-agent status
The AWS CodeDeploy agent is running as PID 25579

#4 - Registering CodeDeploy Instance

After we've successfully installed the codedeploy agent on our server, we need to register our instance with CodeDeploy. This action can only be done through the AWS CLI. Run the following command in the AWS CLI and enter your Lightsail instance name, the ARN of the IAM user we created, and the region of your instance:

$ aws deploy register-on-premises-instance --instance-name MyApp --iam-user-arn arn:aws:iam::641512571549:user/myapp-deploy-user --region us-east-2

Then, the instance needs to be tagged in CodeDeploy. The tags are important because it lets CodeDeploy know where to deploy your code.

$ aws deploy add-tags-to-on-premises-instances --instance-names MyApp --tags Key=Name,Value=MyAppServer

To make sure you set everything up correctly, run the following command in the AWS CLI and make sure you see your instance name in the output JSON:

$ aws deploy list-on-premises-instances --region us-east-2
{
    "instanceNames": [
        "MyApp"
    ]
}

#5 - Setting up AWS CodePipeline & CodeBuild

Now, we set up our CI/CD Pipeline. First, we start with the build stage.

In the AWS console, go to the CodePipeline service and create a new pipeline. In the advanced settings section, store the artifacts in the s3 bucket we previously created.

Next, connect your GitHub account and choose the repository where your code will be stored.

In the "Add build stage" section, you can optionally add a build stage in your pipeline. Note, depending on your project, you might not need to setup this stage. For my project, this is going to be a simple Typescript Node.js server. I will add a build stage, using AWS CodeBuild:

Click the Create Project button to create a new CodeBuild configuration. There are a few options to configure that relate to the build server, but the main thing that concerns us is the Buildspec. The Buildspec file is where you control how your project is built. Here's what I put for my Express server that's written in Typescript:

After adjusting the Buildspec, go down to the "Service role permissions" section and add a service role that has access to the S3 bucket we previously created. This makes sure that our CodeBuild stage can download our code from S3 and build it.

After we finish setting up AWS CodeBuild, go back to the "Create pipeline" form and set up the deploy stage. Here, choose "AWS CodeDeploy" as your deploy provider. Before continuing, we need to go to the AWS CodeDeploy page and set up a new application.

In the AWS Console, go to the CodeDeploy service and create a new application under the "applications" section. Make sure to add the tag name we previously attached to our CodeDeploy instance in the AWS CLI:

After creating the application, click "Create Deployment Group". In this form, there are two main things we need to adjust, the "Service Role" and the "Environment configuration".

For the service role, you can create a new CodeDeploy service role easily by following this AWS User Guide.

In the "Environment configuration", add the server tags attached to your code deploy instance that we previously set.

Now, go back to the CodePipeline form and adjust your deployment stage to be set to the deployment app and group that we just created:

Click next and create your code pipeline.

#6 - Managing Deployment Steps

Within your application source code, you must include a file named appspec.yml. This file tells the codedeploy agent on your servers how to deploy and start your app. Here's the one I have for my Node.js server:

version: 0.0
os: linux
files:
   # copy the built code to /home/ec2-user/app directory
  - source: /
    destination: /home/ec2-user/app
hooks:
  ApplicationStart:
    # run this script to start the server
    - location: scripts/start.sh
      timeout: 300
      runas: ec2-user

I also have a scripts folder within my code source that instructs the codedeploy agent on how to start the app. There are many other hooks you can specify within the appspec.yml file, which you can learn more about here.

I use the pm2 NPM package package to start/restart my Node.js server inside the scripts/start.sh script:

#!/bin/bash
cd /home/ec2-user/app/dist
# Get the PID of the PM2 process
pm2_pid=$(sudo pm2 pid index)

# Check if the PID is empty or null
if [ -z "$pm2_pid" ]; then
    # If the PID is empty or null, start the process
    sudo pm2 start index.js
else
    # If the PID is not empty or null, reload the process
    sudo pm2 reload index
fi

If you set up everything correctly, your CodePipeline should run successfully.

#7 - Optional: Managing Your Server/Handling Scalability

One of the main benefits of Serverless is that it is easily scalable, so how can we do the same for our Lightsail setup. If you use AWS EC2, you have access to AWS Auto-Scaling, which does this for you, but sadly this is not available for AWS Lightsail.

However, there is an alternative way to do this easily, using Caddy. With Caddy, you can serve your site using HTTPS automatically and set a reverse proxy to load balance between your different Lightsail instances. A simple Caddyfile setup that generates a TLS certification for a domain named "example.com" and load-balances between a few servers looks like this:

example.com

reverse_proxy node1:80 node2:80 node3:80

As for autoscaling, we can use AWS's Lightsail API to pragmatically create more instances. The API allows you to include a launch script so you can set up your server automatically.