We live in an era where everyone wants everything in no time. Well, in some cases, it can happen, but for some, it requires a set of principles to streamline the whole process to make it possible, not exactly, in no time but faster than the one traditional process. And when it comes to development, DevOps makes this thing possible for developers to meet the app development requirements of clients and prospective app users in a calculative time
DevOps is a set of principles/rules that streamline processes for developers and operations teams to break the silos and work asynchronously to achieve the outcome with less time to market. But for it to be achieved, they require some tools, which we call CI/CD tools, to create the whole development to deployment pipeline.
As you can see, the market nowadays is flooded with all kinds of DevOps CI/CD tools, but choosing the best one to achieve your deployment needs is necessary. And as your requirement is to continuously deploy your mission-critical build into the Amazon ECS, we recommend to go for the deadly DevOps tools combination of AWS CodePipeline and Terraform to get the efficiency you need.
Do you like to know how this great combination of AWS CodePipeline and Terraform works together to create a CI/CD pipeline that boosts the app development speed to continuously deploy builds into the Amazon ECS ecosystem.? Don’t you worry! This whole blog is created for that purpose only to guide you in the right direction!
So, shall we delve in? Let’s go:
AWS CodePipeline is the continuous delivery service enabling DevOps engineers to model, visualize, and streamline the software build release cycle through the quick CI/CD pipeline creation. So, that means easy development and deployment configuration processes, resulting in a helping hand for releasing software built with less time-to-market.
Now, let’s have a look at what else you can do with the AWS CodePipeline:
Check a detailed history of your pipeline, like pipeline start and end time, how long it ran, and execution details (IDs).
Terraform is an Infrastructure-as-Code (IaC) tool that enables developers to configure cloud-based and on-premises resources in an easily understandable file, which then can be versioned, reused, and shared across systems. You can even utilize the continuous cycle to manage your entire infrastructure through a well-configured lifecycle.
With Terraform, you can manage low-level infrastructure components, like storage, computing, and network resources, along with high-level components, including DNS data and SaaS features.
How does Terraform work?
Terraform mainly works in three critical stages: Write, Plan, and Apply. Let’s know them in a bit more depth:
Now, you do want to know the benefits to reap out by using AWS CodePipeline and Terraform together! It’s the next!
Using the AWS CodePipeline and Terraform together, you can unlock several benefits, including:
Easy tracking of code changes and management of infrastructure through a single source of truth.
Now, the moment we all were waiting for! So, let’s dive into this technical guide to continuously deploy the app into Amazon ECS using AWS CodePipeline and Terraform.
Let’s go through some prerequisites first before starting with the main process.
A Python Flask web application with Dockerfile
Now, let’s take a look at the architectural process we’ll follow throughout the project:
Following image seems more appealing to me.
Here, the major components for implementation are:
ALB (AWS Application Load Balancer) to pass user requests to running ECS task and balance load.
Let’s make the destination ready before delving into the pipeline to streamline the continuous deployment of build releases. And for that, let’s create start with creating three files:
1. variables.tf to store reference variables
This showcases the necessary dependencies.
Now, let’s associate VPC and subnets with Amazon ECS resources to seamlessly provision the ECS cluster.
Well, it can be done in two distinct ways as we are using Terraform, such as:
As you have gone through the prerequisites already, which you definitely have 😜, we’ll pick the second option.
As we are using containers for deploying the application into ECS, we have to push the docker images to a content registry system, like Docker Hub, Elastic Container Register (ECR), Google Container Registry (GCR), etc. But we are using ECR so we’ll push all those images into the ECR.
To do so, we’ll create an ECR repository. Well, whether to keep the repository private or public is totally up to your requirements. Here, we’re going to keep the ECR repository, Private. See below:
Here, we’ve named the url as “image_repo_url” above as shown in the image.
A point to remember! From now on, when we build and push a new image into the ECR, ensure to tag the image. Also, we’ll name every new version tag as the latest with the intention to overwrite the previous image versions.
Now, all required variables for the ECS cluster creation are ready to use!
2. provider.tf to set up providers
When planning to run CI/CD terraform effectively, it’s important to log into the AWS account locally. Confirm that, we’ run the following command:
Also, refer to this guide to use the AWS CLI effortlessly
2. ecs.tf to set up resources
This block of code contains three types of configurations.
Here, Amazon ECS service can be used to process and manage specific task definition instances simultaneously inside the cluster.
In case of one task failure, the Amazon ECS service scheduler instantly launches another instance for replacing the same.
📝Note: the launch_type should be FARGATE to ensure that the build is deployed into the AWS Fargate service. Amongst various reasons, Farget has been chosen for easy management soon as it eliminates much of the overhead for the same.
II. Network-based configuration
After those lines, L59-125 focuses on the Network configuration. If being too specific, it shows the configuration of the AWS security group using Terraform, and it enables us to limit network boundaries for incoming and outgoing traffic.
In addition to that, we’ll also set up load-balancing resources to have a sight on traffic and create a load balancer for the application. For more information, do refer to the Terraform guide for the same.
III. IAM configuration
Now, further, the remaining code lines are for configuring the ECS permission resources that can engage with ECR for logging.
Let’s do the AWS configuration:
Initializing the directory: $ terraform init
Now, you can see the Terraform generated files:
Now, it’s time to verify the implementation.
For that, go to the Amazon Elastic Container Services page to start with the verification process:
Can you see the task definition linked to the appropriate image in the ECR cluster? Yes! Great Job!
Level 1 achieved! 🥳
But note that 📝, this is still not in its working phase as it is still pending to push the flask application in.
🚨Be careful of the information you make available.
Worry not! Let’s move to the next step, which is the deployment stage!
It’s time to prepare the Flask application and push it into the docket.
Let’s start with running your mission-critical build locally: $ pip install flask.
And then create a basic Flask app: flask_app/app.py
Write a command to run the app: $ python3 flask_app/app.py
Now, let’s dockerize the app with the command to create a Dockerfile :flask_app/Dockerfile
Caution: Here, we avoided using FROM python: 3.8 as the Docker hub has some download restrictions, resulting in an app building up the block.
Now, let’s build the docker image with the command: docker build -t flask-application-demo:latest ./flask_app/
This will create the flash-application-demo image and tag it with the label latest, as we’ve discussed in the beginning to overlap the older image.
Want to know whether the image has been built or not? Let’s check it: $ docker image ls
It’s confirmed now that the Flask app image has been created! Hurray!! So, it’s time to run the Docker container: $ docker run -d -p 5000:5000 flask-application-demo
At last visit the local URL you’ve been using to check the ready build of the Flask application.
As the local setup of the app is ready, it’s time to move ahead toward the CodePipeline configuration to deploy the build as we push it to GitHub.
Check-in time to see how the directory tree looks like:
The BuildSpec Activities in AWS CodeBuild
It stands for build specification made of a set of build commands and relatable YAML formatted settings used by CodeBuild to successfully run the build.
Apart from the locals, you can even use BuildSpec to build images that we’ve just done locally and push them to ECS task definitions to deploy the basic Flask application
Let’s have a look at the buildspec.yaml code fil
Here, the buildspec is parted into four stages:
It does require some variables to build and deploy the application, which will be covered in the section.
Again check the directory tree for more confirmation:
Let’s Finally Set Up the AWS CodePipeline
The pipeline we are going to configure will be solely responsible for linking AWS to a GitHub repository, finding the mission-critical app, and implementing and deploying it as per the given buildspec setup.
As discussed above, let’s first create some variables in the file variables.tf:
Now, the CodePipeline has been loaded with some required variables, especially for the GitHub repository. Moving on to the next step to create the main.tf file:
Wow! Such a big block of code. 😲 Let’s do the postmortem of it to understand it step-by-step.
If checking the 8th line, you’ll find referencing the encryption key for the data linked in the L248. This is necessary to detect as it will become our key to securing the GitHub action artifacts located in the Amazon S3 buckets. Make a deep dive into the KMS through the official source!
Follow the below steps to create a KMS key.
So many processes blowing off 🤯our minds, aren’t they? Let’s take a detour of the code to understand it nicely.
The L1-43 are used to create CodePipeline resources, each linked in different stages.
Refer to this link to find Terraform resources.
Lines 12-41 are used to define two stages in the CodePipeline.
Apart from that, the deploy stage plays a pretty important role in the CodePipeline configuration. Therefore, it is greyed out in the Terraform architecture
The source stage is simply used to connect the pipeline with the GitHub repository with the command CodeStarSourceConnection.
📝Note: Remember, this command: CodeStarSourceConnection is only used when using the source code from GitHub, Bitbucket, and GitHub Enterprise Server.
See the L126, where the build stage is mentioned with CodeBuild to build and deploy the app located in the aws_codebuild_project resource configuration file. You can even use a close alternative to this, which is – Custom Jenkins.
This source phase also represents the output artifact in the form of an input artifact in the build stage.
See lines 55-125 to check how CodePipeline roles and policies are created, which is important for its associated S3 bucket, CodeStar, IAM, KMS, and CodeBuild activities.
See Lines 126-172 to create a CodeBuild resource asking to use a Linux container to pass the environmental variables into the build, alongside the use of a buildspec file in the source block.
See lines 173-247 where CodeBuild’s roles and policies are declared, which plays an important role in accessing the IAM, S3 bucket, ECR, and ECS.
Uff! 😮💨Finally the relief of explaining a bunch of configurations. Let’s move on to the application of our setup.
It’s directory tree checking time! ⌚
Now, let’s apply the configuration with the use of Terraform:
Here’s the command for it: $ terraform apply
Implementation Verification Time ⌚
Let’s begin with checking the CodePipeline!
There are times when the first attempt does fail, as we require to authenticate the GitHub connection. Hence, you’ll detect an error like this as shown below:
Now, let’s Authorize the GitHub connection by following this path: Navigate to Settings>Connections;
Once we’ll update the GitHub connection request, we’re all good to delve in:
As we update the connection, the status will change into Available.
Now, push the local Flask app source code to the GitHub repository and observe the pipeline stages and status;
If triggering the pipeline at this moment, it will throw an error like this:
[GitHub] Upload to S3 failed with the following error: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
The error indicates a missing link to the KMS key with the CodePipeline and CodeBuild roles. But worry not! It can be done right now as well 😌:
Now, open flask_app_s3kmskey
Scroll down a bit and now make the link as discussed above:
To see the results, it’ll ask for rerunning the Pipeline, which is not an issue 🤓
Open the AWS CodeBuild to see some more information:
Now, click on the latest build run from the history to check some information regarding the same:
Now, you know how to explore more information! The genius you! 😎
Shall we now move forward to the ECS verification? Let’s go!
Search for the ECS cluster, we just created to check the cluster information:
Now, follow this path to check the networking configuration: Clusters > Services > flask-app-demo > Networking
When you see the load balancer URL, click on it, and voila, you can see your first Flask Application ready – AKA the deployed version of the app.
Now, moving on to the Continuous Deployment Verification process:
At first, you’ll need to update the flask application: flask_app/app.py
Then push the application code to the GitHub, like this:
There’ll also a requirement to check the Pipeline – done like this:
Once the build is done, you can check the information reflected on the app like this:
Woohoo! 🥳 You’ve now excelled at deploying the application into the AWS ECS using AWS CodePipeline and Terraform!
You’ve done a great job so far! However, you can optimize this process a bit by taking some suggestions, like this:
Finally, make the most out of AWS CodePipeline to deploy other AWS services, like Amazon EBS.
Keep trying, keep experimenting, and keep striving for excellence! That’s what we promote at Tuvoc Technologies.
You can visit our news and projects on a weekly basis. Or you can subscribe to email for regular email updates.