Things Ian Says

A CD Pipeline in Amazon Web Services

In a previous article, I described serving a website from an S3 bucket, with CloudFront allowing us to apply SSL. This article looks at the other side of the process — how we populate the S3 bucket in the first place. I would typically use Jenkins to set up a build and deploy pipeline for this, but here I look at using the build tools AWS provides.

The Aim

Here is what I wanted from my pipeline:

• A git repository for my code
• Build automatically triggered by code check-in
• Deploy triggered by completed build
• E-mail notifications for:
  • Build
  • Deploy

I also wanted to make this as cheap as possible, and to use the AWS free tier wherever possible. So, with this in mind I started looking at the AWS developer tools.

AWS Developer tools

Amazon has a set of developer tools which cover these basic stages, so this was my first port of call:

• CodeCommit — a git-based code repository
• CodePipeline — a pipeline tool which allows us to string together tasks (like Jenkins)
• CodeBuild — the tool which does the build

You can think of these tools as working like this:

However, when I looked at the AWS free tier, it turned out that CodePipeline was not in the free tier. So, I decided to think creatively about what else I could use. I started thinking about using an AWS Lambda to trigger the build job — I knew that I could use the AWS SDK from within a Lambda, so that seemed like it would work. But then how to trigger the Lambda? It turns out that CodeCommit can trigger a Lambda itself, but for flexibility and ease of monitoring, I decided to get it to send a notification via SNS instead. That notification could then both trigger my Lambda and also send me an e-mail.

To give me visibility as the job progressed, I’d also get the Lambda to send an SNS notification when it started the build, and then get CodeBuild to send an SNS notification when the deployment was built. These would all trigger e-mails to me.

Putting it all together, it looks like this:

Sending an SNS Notification on Commit

The first thing we need to do is set up our SNS topic. This is easily done from the SNS page:

The resulting dialogue is very simple, just requiring a name for the topic and a display name (needed for SMS, if you want to use that):

Having created our topic, we can now set a trigger to be fired when code is committed. We start by going in to the CodeCommit settings, and then clicking on the trigger tab:

From there, we can use the create trigger button:

In the dialogue, I named the trigger and set the conditions for it to get triggered, which were on a push to existing branch for the master branch. I also selected the action to take (send to SNS) and the name of the SNS topic.

You can see a Test Trigger button at the bottom of dialogue, which we can use to check everything is okay. If we click this button, we get a dialogue indicating that everything has gone as expected:

Since we have nothing watching the SNS topic, we don’t have a simple way of checking that it’s actually working (beyond AWS telling us it is working). So let’s get it to send an e-mail.

Sending an E-mail

We start off by navigating to the create subscription dialogue, which we can do via the topic we created earlier (as shown here) or via the subscriptions option:

In the create subscription dialogue, we have a dropdown of all the possible items which can interact with SNS topics. We will select e-mail:

This only needs us to enter our e-mail address to proceed:

We now get a message saying that the e-mail address needs to be confirmed before it can receive any SNS notifications:

AWS obviously doesn’t want SNS to enable spamming, so requires a confirmation step to happen before an e-mail address is enabled. If you are trying this out, you will need to wait briefly until an e-mail is delivered to your inbox. The e-mail contains a link for you to follow, after which you will receive e-mail notifications when something is published to the SNS topic.

You can now check that everything is working okay, by using the Test Trigger button again. This time, you should receive an e-mail notification at the e-mail address you entered earlier. Here is an example of what you can expect:

Creating our Build Task

The AWS security model is fairly fine-grained, so the first thing we need to do is create a policy which will allow our build task to do everything it needs to. If we think through what we need to do, we get a list like this:

• Access our CodeCommit (git) repo
• Copy files to our S3 bucket
• Push a message to an SNS topic to indicate the deployment is complete
• Write logging output

So, we open up the IAM page, navigate to Policies, and then select Create Policy, then Create Your Own Policy. In the policy document, we start by entering the preamble:

{
    "Version": "2012-10-17",
    "Statement": [

We want to be able to pull from our CodeCommit repo:

        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:codecommit:us-east-1:6XXXXXXXXXX1:example"
            ],
            "Action": [
                "codecommit:GitPull"
            ]
        },

We want to be able to copy files to S3:

        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::example",
                "arn:aws:s3:::example/*"
            ],
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:List*"
            ]
        },

We want to write a notification to SNS:

        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:sns:us-east-1:6XXXXXXXXXX1:exampleDeployedTopic"
            ],
            "Action": [
                "sns:Publish"
            ]
        },

And we want to allow logging:

        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:logs:us-east-1:6XXXXXXXXXX1:log-group:/aws/codebuild/exampleBuild",
                "arn:aws:logs:us-east-1:6XXXXXXXXXX1:log-group:/aws/codebuild/exampleBuild:*"
            ],
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ]
        }

Finally, we finish off the policy JSON:

    ]
}

We now need to build a Role based on this policy. So, in IAM, we select Roles, then Create New Role, which gives us the following screen, where we scroll down to select AWS CodeBuild:

Once we click on select, we get a list of policies we can attach. This is a long list, so type into the search box the name of the policy you’ve just created and narrow the list down:

Now click through the next two screens, adding in a name for the role in the appropriate input box. This will create the role — remember the name you called it, we will need it later.

Next we will create our build task, using CodeBuild. Start by selecting CodeBuild from the services menu, and click on the Create Project button. This will give you the following dialogue:

We start by giving our build task a name:

Now we can choose where we get the code to build from. The option we want is AWS CodeCommit, but there are also other choices. If we want to stay in AWS, we can select an S3 bucket. We can also use GitHub or Atlassian Bitbucket:

Because we have selected AWS CodeBuild, we get a list of repositories available to us in a dropdown. We just select the one we want to use for our build:

Next we need to define the environment we are going to do our build in. We don’t actually need anything specific (since we are just going to copy some files), but if we wanted to do something more complicated, here is an example of how we would set up a NodeJS build environment:

Note also that in the above dialogue we have selected Use the buildspec.yml option. We will come back to that shortly.

The next thing we define is where to put the artefacts which result from our build task. Normally we would define something here, but for our purposes, this is not ideal for us. CodeBuild puts the resulting artefacts within two levels of directories reflecting the build task structure. However, we want our files to be placed at the top level of the S3 bucket (so they can be served as a website). We therefore set no artefacts here, and will add the sync of files to our S3 bucket from within the build task itself.

Finally, we choose our role from earlier (the one I told you to remember) to run the build task:

Save the CodeBuild configuration, and we need to create the buildspec.yml file we mentioned earlier. This is a file we place at the root level of our codebase. It is a YAML file, which contains Linux commands, grouped by build phase. The four build phases are install, pre_build, build, and post_build.

We don’t actually need a build step at the moment, so we will put a placeholder echo command in there. We will use post_build to do the two steps we’ve discussed earlier —- sync our HTML to our S3 bucket, and send a notification to SNS. The build environment has the AWS CLI tool available in it, so we can use that in our buildspec.yml:

version: 0.2

phases:
  build:
    commands:
      - echo "Building example"
  post_build:
    commands:
      - aws s3 sync ./html s3://example
      - >
        aws sns publish
        --topic-arn "arn:aws:sns:us-east-1:6XXXXXXXXXX1:exampleDeployedTopic"
        --subject "[example/aws] CodeBuild deploy completed"
        --message "Example deployed `date`"        

If we go back to the CodeBuild screen, we can test our build job by clicking on the Start Build button:

If we click on this button (assuming we have set up everything correctly), our S3 bucket will get updated from our CodeCommit repo.

Pause and Reflect

We’ve created much of our pipeline, so it’s worth a quick pause to see where we are. We’ve set up our trigger from CodeCommit, our SNS topic to handle it, and an e-mail when the commit happens. We’ve also set up a build task which pushes to our S3 bucket. If we refer to our initial diagram of what we wanted, this is where we are:

Setting up a couple more SNS topics is very minor (in fact I’m not going to cover it — it’s simply a repeat of the earlier SNS setup steps), so all we are missing is our Lambda — which is also the thing which holds our whole pipeline together.

So let’s finish this off by adding in the Lambda.

Trigger CodeBuild from a Lambda

The first thing we need to do is create an IAM role for our Lambda, which allows it to do what it needs to. In order to do this, we also need a custom policy we can apply to the role. I’ve already shown how we create a role and apply a policy earlier (so I won’t cover it here), but I will talk through the policy we need to create.

As before, we start with a preamble:

{
    "Version": "2012-10-17",
    "Statement": [

The whole reason we are creating this Lambda is to trigger a build, so the policy needs to allow that to happen:

        {
            "Effect": "Allow",
            "Action": [
                "codebuild:StartBuild"
            ],
            "Resource": [
                "arn:aws:codebuild:us-east-1:6XXXXXXXXXX1:project/exampleBuild"
            ]
        },

One of our original requirements was to get an alert when the build was started, so we also need to give our Lambda permission to publish a Notification:

        {
            "Effect": "Allow",
            "Action": [
                "SNS:Publish"
            ],
            "Resource": [
                "arn:aws:sns:us-east-1:6XXXXXXXXXX1:exampleBuildStartedTopic"
            ]
        },

Finally, we want to be able to log what’s going on for troubleshooting:

        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-east-1:6XXXXXXXXXX1:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:us-east-1:6XXXXXXXXXX1:log-group:/aws/lambda/exampleCommitLambda:*"
            ]
        }
    ]
}

We then save this policy as before, and create a role which has this policy.

We are now ready to create our Lambda. First we go the the Lambda page and click on Create Function. We are then offered a screen which allows us to select a blueprint:

We don’t want to use a pre-defined blueprint, so we click on Author from Scratch. Then we are asked how we want to trigger our Lambda:

We select SNS, and click on activate:

Next we configure our Lambda to use NodeJS:

Note the field called Handler. This is the name of the function which is called when our Lambda is triggered. We will export it from the code we are about to write.

We then get to choose how to enter the definition of our Lambda (i.e. the NodeJS code). We have a few options, like uploading a file, but we will enter inline:

Here is the code we enter for our Lambda. We start by importing the AWS SDK and defining CodeBuild and SNS instances:

var AWS = require("aws-sdk");
var codebuild = new AWS.CodeBuild();
var sns = new AWS.SNS();

We export a function called handler. This needs to match the name of the function we defined in our configuration earlier. It takes three parameters — an object for the event, an object for the context the Lambda is running in, and an optional callback function we can use to pass information back out of the Lambda:

exports.handler = (event, context, callback) => {

Now I write a callback function for success/error handling when I attempt to publish to an SNS topic. This simply writes a message to the console indicating success or error, and also uses the Lambda’s callback to pass back the same indication. In the case of an error, it also dumps the stack to the console to help with debugging:

    var handleSnsResult = function (err, data) {
        if (err) {
            console.log("Error sending SNS:", err, err.stack);
            callback(null, "Error sending SNS")
        } else {
            console.log("SNS has been published:", data);
            callback(null, "SNS has been published");
        }
    };

The purpose of the Lambda is to start the build, so we call the startBuild method on our CodeBuild instance to do this. We use the callback function for startBuild to publish a success or failure message to our SNS topic. That publish, in turn calls the handleSnsResult function we defined above to handle the result:

    codebuild.startBuild({
        projectName: "exampleBuild"
    }, function (err, data) {
        if (err) {
            console.log("Error triggering Codebuild:", err, err.stack);
            sns.publish({
                Message: "CodeBuild trigger failed for example: " + err,
                Subject: "[example/aws] CodeBuild trigger failed",
                TopicArn: "arn:aws:sns:us-east-1:6XXXXXXXXXX1:exampleBuildStartedTopic"
            }, handleSnsResult);
        } else {
            console.log("Codebuild has been triggered:", data);
            sns.publish({
                Message: "CodeBuild has been triggered for example: " + JSON.stringify(data),
                Subject: "[example/aws] CodeBuild has been triggered",
                TopicArn: "arn:aws:sns:us-east-1:6XXXXXXXXXX1:exampleBuildStartedTopic"
            }, handleSnsResult);
        }
    });
};

The Lambda page has a test button which we can use to verify that our Lambda function is working correctly. Here is what we see if we test now:

You can see the message “SNS has been published” in the result returned box. This is from where we used the callback in the handleSnsResult function. You can also see the “Codebuild has been triggered” message in the log output box. This is from our console.log call in the callback to our startBuild method. This is also available via CloudWatch logs. I also received two e-mail messages, one from this Lambda and the other from the CodeBuild job we created earlier.

The final check is to push to our CodeCommit repository, and check that the pipeline ran correctly.

Summary

This article has shown how we can create a build/deploy pipeline in AWS. It has demonstrated the use of the following AWS services:

• CodeCommit for software version control
• SNS for task synchronisation and e-mail alerts
• Lambda for triggered tasks
• CodeBuild for defining and running a build process
• S3 Buckets for hosting a static website

As mentioned at the top of this article, a previous article, describes how the S3 Bucket is used to serve up our website. Putting the two pieces together, we have the following as a diagram of both the deploy and serve elements working together: