This content originally appeared on DEV Community and was authored by nofarb
In this post, we’ll introduce Cache Intelligence , an innovative solution to simplify caching in continuous integration (CI) pipelines by automatically caching and restoring known dependencies. With Cache Intelligence, offered in Harness Continuous Integration module, developers can optimize CI build times and improve the efficiency of software delivery.
Why Cache Intelligence?
CI builds often require downloading and installing large dependent software components, such as libraries, which can be very time-consuming. When building locally or reusing build machines for multiple builds, a cache is usually available to speed up the process. However, this is often not the case when using modern continuous integration solutions because they execute pipelines in ephemeral environments, such as containers or virtual machines, which are terminated after execution. This approach ensures that pipelines run in isolated environments, free from interference by other build instances, but the lack of a local cache can result in longer build times. As such, configuring a cache is recommended when using ephemeral build environments, but this task can be tedious.
The responsibility of incorporating this logic into pipelines, and managing the cache typically falls on the user. Even though caching can already be achieved in various ways — such as save and restore cache steps or mounting volumes — engineers are still responsible for the bulk of configuration and maintenance, taking time away from development. Many developers don’t even realize that caching is a solution to expedite their build times.
Harness Cache Intelligence simplifies the build process in CI pipelines by streamlining cache management and automatically caching and restoring known dependencies. We store the cache securely in Harness cloud, our fully hosted builds environment, freeing you from the hassle of providing your own storage. By removing the complexity of cache management, developers benefit from more streamlined workflows, allowing them to increase efficiency and productivity and focus on writing code.
Let’s demonstrate how users can get started with Cache Intelligence to see how this new feature significantly reduces build time.
Demonstrating the Impact of Cache Intelligence on Build Speed
In order to show how effective Cache Intelligence is in reducing build time, we’ve used the Harness Developer Hub (a web application built with Yarn) to conduct two consecutive builds.
In the first execution, the cache was unavailable, and the build step took 9m 5s, with the entire build process totaling 10m 56s.
In the second execution, the cache was restored and utilized, reducing the build step to 3m 10s and the overall build time to just 6m 1s.
The result is a 45% reduction in the total build duration, demonstrating the significant impact of Cache Intelligence on optimizing build performance.
In the images below, you can see the difference in performance between individual steps when the cache is not present :
Cache is not available:
Cache is available :
This comparison showcases the significant improvements in build speed and efficiency from Cache Intelligence.
Try it yourself — Setting Up and Using Cache Intelligence
When using Harness CI, you first need to add the “caching” property in your CI stage. We store the cache securely in Harness cloud, our hosted environment, freeing you from the hassle of providing your own storage.
- stage:
name: Build Jhttp
identifier: Build_Jhttp
type: CI
spec:
caching: # --------------- ADD LINE
enabled: true # ----------------ADD LINE
cloneCodebase: true
Currently, automatic dependency detection and caching supports Bazel, Maven, Gradle, Node, Yarn, and Go build tools. If you’re using a different build tool or have dependencies stored in non-standard locations, Cache Intelligence can still be used by specifying the paths you’d like to cache. For instance:
- stage:
name: Build Jhttp
identifier: Build_Jhttp
type: CI
spec:
caching:
enabled: true
Paths: # ------- Add 'Paths' under 'caching'
- tmp/cache # ------- Specify one or more paths to cache
cloneCodebase: true
Here’s a step-by-step example demonstrating how can you use Cache Intelligence:
- Fork the Harness Developer Hub GitHub repository https://github.com/harness/developer-hub into your account.
- Sign up for a free account (or login to an existing account)
- Follow the Get Started wizard in Harness CI.
- Configure connection to your Source Code Manager
- When you are prompted to select a Source Control Manager, select ‘GitHub’ and use either OAuth or Access Token as the authentication method that Harness CI will use to connect to your GitHub account.
- When you are prompted to select a repository, select the repository that you forked in the earlier step, and then select Configure Pipeline.
- Select ‘Create empty pipeline configuration’ and then select Create Pipeline.
After following these steps, you will have a basic ‘hello world’ pipeline that clones your forked repository and prints a welcome message. Modify the pipeline as following
Add the ‘caching’ property under the stage’s spec property
caching:
enabled: true
Replace the ‘Echo Welcome Message’ step with the following step that builds the application
type: Run
name: Build
identifier: Build
spec:
connectorRef: account.harnessImage
image: node:18
shell: Sh
command: |-
yarn
yarn build
At the end you will have a pipeline similar to the following:
pipeline:
name: Harness Developer Hub
identifier: Harness_Developer_Hub
projectIdentifier: playground
orgIdentifier: default
tags: {}
properties:
ci:
codebase:
connectorRef: Nofar_githubcom
repoName: harness/developer-hub
build: <+input>
stages:
- stage:
name: Build
identifier: Build
type: CI
spec:
caching:
enabled: true
cloneCodebase: true
platform:
os: Linux
arch: Amd64
runtime:
type: Cloud
spec: {}
execution:
steps:
- step:
type: Run
name: Build
identifier: Build
spec:
connectorRef: account.nofar_dockerhub
image: node:18
shell: Sh
command: |-
yarn
yarn build`
Get Started with Harness Cache Intelligence
Harness’ Cache Intelligence presents an advanced approach to enhancing build performance in CI pipelines. Ready to start streamlining your cache management? Sign up for a free today, or request a demo!
This content originally appeared on DEV Community and was authored by nofarb
nofarb | Sciencx (2023-05-01T22:43:03+00:00) A new approach to caching in continuous integration (CI). Retrieved from https://www.scien.cx/2023/05/01/a-new-approach-to-caching-in-continuous-integration-ci/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.