This content originally appeared on DEV Community and was authored by se-piyush
Working with repositories that house multiple subdirectories, each representing an independent service deployed on AWS Lambda, presents unique challenges. One of the primary issues is determining which services need deployment based on changes within their respective directories. In this blog, I will share how I tackled this challenge by creating an efficient CI/CD pipeline using GitHub Actions, divided into two key jobs: detecting changes and building and deploying the necessary services.
Challenges with Multi-Directory Repositories
When dealing with a repository structured with multiple subdirectories, each corresponding to an independent service, it becomes crucial to identify changes at a granular level. Each service must be individually assessed for modifications to ensure that only the necessary components are built and deployed. This selective deployment helps optimize the CI/CD process, saving time and resources.
Solution: Dividing the Pipeline into Two Jobs
To address this issue, I designed the CI/CD pipeline with two primary jobs:
detect-changes: This job identifies which folders (services) have changes using git diff and stores them in an array.
build-and-deploy: This job takes the array generated in the previous step, creates zip packages, and deploys them as Lambda functions.
Workflow Configuration
Here is a detailed breakdown of the GitHub Actions workflow configuration:
name: lambda package update
on:
push:
branches:
- dev
paths:
- "**"
jobs:
detect-changes:
runs-on: ubuntu-latest
outputs:
services: ${{ steps.set-services.outputs.services }}
steps:
- name: Checkout repository
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Set services variable
id: set-services
run: |
echo "Checking for changes..."
services="[]"
current_commit=${{ github.sha }}
previous_commit=${{ github.event.before }}
for service in serviceA serviceB serviceC; do
if git diff --name-only ${{github.event.before}} ${{github.sha}} | grep "^${service}/"; then
services=$(jq -c --arg service "$service" '. + [$service]' <<<"$services")
fi
done
echo "::set-output name=services::$services"
- name: Debug services variable
run: echo ${{ steps.set-services.outputs.services }}
build-and-deploy:
needs: detect-changes
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
if: ${{ needs.detect-changes.outputs.services != '[]' }}
strategy:
matrix:
service: ${{ fromJson(needs.detect-changes.outputs.services) }}
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Set up Node.js
uses: actions/setup-node@v2
with:
node-version: "20"
- name: Install dependencies
run: |
cd ${{ matrix.service }}
npm install --omit=dev
- name: Create zip package
run: |
cd ${{ matrix.service }}
zip -r ../${{ matrix.service }}.zip node_modules src index.js
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: role-arn
aws-region: region
- name: Deploy to AWS Lambda
run: |
aws lambda update-function-code --function-name ${{ matrix.service }}-${{ github.ref_name }}-api --zip-file fileb://${{ matrix.service }}.zip
Key Components Explained
Detect Changes Job
- Checkout Repository: Checks out the repository to access the code.
- Set Services Variable: Uses git diff to identify which service directories have changes and sets the services variable accordingly. The services array is then output for use in the next job.
Build and Deploy Job
Conditional Execution: This job only runs if there are changes detected in the previous step.
Matrix Strategy: Iterates over each service that has changes.
Install Dependencies and Create Zip Package: Installs the necessary dependencies and creates a zip package for each modified service.
AWS Deployment: Deploys the zip package to AWS Lambda, updating the function code.
Benefits and Optimization
By implementing this two-step CI/CD pipeline, I achieved a more efficient and optimized deployment process. This setup ensures that only the services with changes are built and deployed, significantly reducing unnecessary builds and deployments. Additionally, using GitHub Actions' matrix strategy allows for parallel processing of multiple services, further speeding up the deployment process.
Conclusion
Managing multi-directory repositories with independent services can be complex, but with a well-structured CI/CD pipeline, you can streamline the process. By detecting changes and conditionally deploying only the necessary services, you can save time and resources, making your development workflow more efficient. This approach can be adapted to various use cases, providing a scalable solution for complex repository structures.
This content originally appeared on DEV Community and was authored by se-piyush
se-piyush | Sciencx (2024-06-24T16:15:38+00:00) Efficiently Managing Multi-Directory Repositories with GitHub Actions CI/CD Pipeline. Retrieved from https://www.scien.cx/2024/06/24/efficiently-managing-multi-directory-repositories-with-github-actions-ci-cd-pipeline-2/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.