Quickly set up dev/QA/pre-prod k8s cluster for teams
The Challenge
In my current project, we run our infrastructure on ERL, AWS, and Azure. We have multiple microservices that make up our application and we deploy each of them to different Kubernetes clusters/environments(dev/stage/QA/prod) on ERL, AWS, and Azure.
Given the rapid development cycle and PoCs that our team does, we need to quickly create environments to test new services without affecting our existing clusters/environments.
Earlier it was manageable to create these clusters by using ERL console, but lately, we realized that it took a long time to get the VMs, configure them and set up the Kubernetes clusters which our Dev team could use. We explored AWS to quickly provision resources(EC2, S3, etc.) but we were not satisfied by the time spent on navigating the AWS console setting up VPCs, subnets, security groups, keys, etc.
The DevOps team was not able to focus on other initiatives like evaluating better CD tools etc. as most of the time was spent on setting up multiple environments for dev/QA team.
Once we decided that we need to address this pain point, we started exploring Infrastructure and configuration management tools to automate the creation of our dev/QA environments. That is when we discovered Terraform and Ansible.
Terraform and Ansible
Terraform is a tool for developing, changing and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. It supports major cloud vendors, so it’s agnostic to which cloud you are running, and it can help with automation.
Ansible is an open-source automation platform. It is very, very simple to set up and yet powerful. Ansible can help you with configuration management, application deployment, task automation. It can also do IT orchestration, where you have to run tasks in sequence and create a chain of events that must happen on several different servers or devices.
So without going into the theory, let's see what we did to make our lives easier.
The tools used:
- Terraform
- Ansible
- Jenkins
- Bash Scripting/Python
- Bitbucket
We integrated Terraform and Ansible with Jenkins so that we are able to fill our requirements (such as Instance_Name, Instance_type, Subnet, VPC, Key_pair, etc.) in Jenkins field, and allow Jenkins variables to do the work for us.
We use a bash script in the backend which takes the input from Jenkins and writes it to .vars file. The Jenkins input variables would get appended to the Terraform-user-input variable file(.vars file). Check out the image below for better understanding.
This is just a one-time effort, and then we can create unlimited Kubernetes cluster in any region within minutes.
Make sure to take the variables as input in Jenkins job, and run a background script(be it in Bash, Python, etc.), that keeps appending the values in these variables to .vars file, and then performing Terraform plan and Terraform apply. Make sure to delete the terraform.tfstate in every build, else you will end up modifying your existing infrastructure and land yourself in trouble.
Now all you need to is click "Build Now" in the Jenkins job and you will have your Kubernetes cluster up and running in no time.
Feel free to use, modify and hack around the code to suit your needs.