This content originally appeared on Level Up Coding - Medium and was authored by payal moondra
Many organizations have been switching to DevOps culture in recent years. Continuous integration and continuous delivery are the most important aspects of CI/CD. Today, a lot of engineering teams aim for the fastest deployment of bug fixes and features. To achieve this, having a robust CI/CD pipeline is a must. However, we need to consider the cost of implementing these solutions as well.
I would say containerizing your process is the first step towards an effective pipeline. If you are working on CI/CD pipelines and have already containerized them, then below are the things that I learned that you can implement to reduce the time and cost of your architecture.
- Use shallow clones.
- Different .dockerignore files.
- Use DOCKER_BUILDKIT.
- Use remote cache for docker builds.
- Run commands in parallel wherever possible.
- Disable npm progress and use parallel-worker.
Use shallow clones :
The standard clone command generally pulls out all the files and the entire history of the revision of all the files. This takes a lot of time(depending upon the size of the repository and the history). Instead, using shallow clones can save you a significant amount of time because it will only pull the latest revision based on the depth given.
How to use it?
Jenkins: You can set shallow clone and depth of cloning in Jenkins advanced configuration.
AWS CodeBuild: You can set the clone depth in the AWS CodeBuild project configuration as below. (Refer: https://aws.amazon.com/codebuild/)
Shell: Use depth with your git clone command.
git clone --depth
CircleCi: You can use below CircleCi orbs below to set a shallow clone. (Refer: https://circleci.com/docs/2.0/orb-intro/)
git-shallow-clone/checkout
Different .dockerignore files:
If you are using docker for your pipelines, then this can be another way to save image building time. Docker build will ignore all the files and folders listed in dockerignore and hence you can put all the files which are not required as part of your CI or CD in .dockerignore. You can also have more than one .dockerignore file if you want to share the same docker image for your CI as well as CD. This way, you can have a list of files and folders required in the test but not in development into .dockerignore.prod and make the deployment faster.
How to use it?
For CI, create a file called .dockerignore.ci and put files and folders in it which are not needed to run CI. For deployment, create a file called .dockerignore.prod and put files and folders in it which are not needed. In the CI pipeline, run the below command before the docker builds step.
ln -fs .dockerignore.ci .dockerignore
Similarly, for deployments, run the below command before the build step.
ln -fs .dockerignore.prod .dockerignore
Using DOCKER_BUILDKIT:
Docker buildkit is one of the latest additions of features which is available in the latest Docker release 19.03. Buildkit provides fast docker builds by caching and running possible layers in parallel. With buildkit enabled, your docker image building time will reduce than otherwise.
How to use it?
Run the below command before starting docker build:
export DOCKER_BUILDKIT=1
It has various options that can be added to the docker file. I am showing one example of how to enable cache for pip install.
COPY requirements.txt $PROJECT_DIR/
RUN — mount=type=cache,target=/root/.cache/pip pip install -r $PROJECT_DIR/requirements.txt
Refer: https://docs.docker.com/develop/develop-images/build_enhancements/
Remote cache for docker builds:
If you are using CI tools or services which do not provide docker layer caching, then using remote cache could be a very useful option to save a tremendous amount of time for builds. Without docker layer caching, each docker layer will get built from scratch every time. This will take more time on all builds. Instead, if you manage your pipelines in a way that pushes the standard base image on the docker hub or AWS ECR and docker build uses the cache from that image, you will save a lot of time for builds.
How to use it?
- Create a docker image from Dockerfile
- Push the image to AWS ECR (<base_image_uri>)
- Include — cache-from parameter in your docker builds command in the build pipeline.
Ex:
Docker build -f Dockerfile — cache-from <base_image_uri> -t <image_name>
Run commands in parallel wherever possible:
If you have multiple tasks to be executed in one job or stage of the pipeline, you can consider running them in parallel if they are independent.
How to use it?
Command 1 &
Command 2 &
Command 3 &
wait
If you also want to gather exit codes of all processes and continue or terminate the pipeline based on that, you can refer to some of the suggestions provided here: https://stackoverflow.com/questions/1570262/get-exit-code-of-a-background-process
Set npm progress to false or use yarn:
If you are building npm modules as part of the pipeline, you can either use yarn install/build or you can disable npm progress bar. Both of these will save significant time in building web packs.
How to use it?
npm set progress false
npm install
npm build
You can also consider running web packs in parallel. Choose the number of workers to be the same as the number of cores available in the system where the web packs are built.
(Details here: https://www.npmjs.com/package/parallel-worker)
We have successfully achieved an automated faster CI as well as deployment pipeline using the above solutions. We have successfully reduced more than 50% of execution time and 99% manual intervention in these pipelines. Also, we have been able to reduce the cost of the entire infrastructure because most of our tools/services are of the pay-as-you-use model. If you are interested in knowing more about them, you can take a look at this article I have written on AWS blogs.
HackerEarth Scales Up Continuous Integration for Future Needs with AWS | Amazon Web Services
Runbook to optimise your CI/CD pipelines was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by payal moondra
payal moondra | Sciencx (2021-10-14T12:22:51+00:00) Runbook to optimise your CI/CD pipelines. Retrieved from https://www.scien.cx/2021/10/14/runbook-to-optimise-your-ci-cd-pipelines/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.