Keda ScaledObject As Code Using CDK8S

Abstract

keda is Kubernetes Event-driven Autoscaling and it’s been used wisely now. In this blog, it provides the way to create Keda scaledobject CRD as code using CDK8S typescript.
With importing Keda CRDs and using CDK8S you can create K…


This content originally appeared on DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’» and was authored by πŸš€ Vu Dao πŸš€

Abstract

  • keda is Kubernetes Event-driven Autoscaling and it's been used wisely now. In this blog, it provides the way to create Keda scaledobject CRD as code using CDK8S typescript.
  • With importing Keda CRDs and using CDK8S you can create Keda scaledobjects using your familiar programming languages such as typescript as scale.

Table Of Contents

  • Pre-requisite
  • Overview of Keda
  • Import Keda CRDs
  • Write code
  • Build keda scaledobjects from code
  • Apply and test
  • Conclusion

πŸš€ Pre-requisite

  • Install typescript, node, and cdk8s as well as projen (optional) which is a tool of managing project configuration as code.
  • Getting started with cdk8s

πŸš€ Overview of Keda

  • KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication.
  • KEDA supports multiple triggers within an Scaledobject. Each trigger is exposed split as a metric and the HPA Controller does a MAX between all the metrics.

πŸš€ Import Keda CRDs

  • Keda does not provide its CRDs separately so we can found the manifest in GH release section. Here I import the current latest version of keda v2.8.0 and out the imports folder in src/imports
  ⚑ $ cdk8s import https://github.com/kedacore/keda/releases/download/v2.8.0/keda-2.8.0.yaml --output src/imports/
  ------------------------------------------------------------------------------------------------
  A new version 2.0.88 of cdk8s-cli is available (current 2.0.13).
  Run "npm install -g cdk8s-cli" to install the latest version on your system.
  For additional installation methods, see https://cdk8s.io/docs/latest/getting-started
  ------------------------------------------------------------------------------------------------
  Importing resources, this may take a few moments...
  keda.sh
    keda.sh/clustertriggerauthentication
    keda.sh/scaledjob
    keda.sh/scaledobject
    keda.sh/triggerauthentication
  • Import result
  ⚑ $ tree src/imports/
  src/imports/
  └── keda.sh.ts

  0 directories, 1 file

πŸš€ Write code

  • Overview of keda scaledObjects in this post

  • It's much more convinient to use visual code writing KEDA scaledobject in typescript language. We can read the document and find all references of construct, objects and properties of KEDA CRDs
  • This blog provides the usecase of creating scaledObject (SO) for Apache airflow worker component. It contains 3 triggers (Scalers) in the SO

1. Cron - Scale applications based on a cron schedule.

  • Airflow server scheduled pipelines and worker components are scaled out at that time, but it takes time to start nodes, join node to cluster and Pod ready (about 2-3mins) so we use cron to pre-scale workers

2. PostgreSQL - Scale applications based on a PostgreSQL query.

  • This scaler bases on the output of query command to scaleout works, here we count the number of running/queued airflow task instances belongs to scheduled pipeline (eg. running reports). Divide the count number to Airflow worker concurrency (eg. 16 as default)

  • targetQueryValue: '1.5' - The result of above calculation divide to this target value to decide how many pods will be scaled out

3. CPU - Scale applications based on cpu metrics

  • This is optional as it ensure provisioning workers when CPU Utilization is higher than 80%

    • PostgreSQL Scaler requires TriggerAuthentication to provide password of airflow user in order to query the database. The credential is get from K8S secret airflow-secret within the airflow namespace
    const pgAuth = new TriggerAuthentication(this, 'KedaPostgresAuthentication', {
      metadata: {
        name: 'keda-airflow-postgresql-auth',
        namespace: 'airflow',
      },
      spec: {
        secretTargetRef: [{
          parameter: 'password',
          name: 'airflow-secret',
          key: 'postgresql-password',
        }],
      },
    });
  • Some SO specs need to know
    • pollingInterval: This is the interval to check each trigger on. By default KEDA will check each trigger source on every ScaledObject every 30 seconds. So to reduce the query connections/workload to airflow database we need to care this vaule.
    • cooldownPeriod: The period to wait after the last trigger reported active before scaling the resource back to 0.

πŸš€ Build keda scaledobjects from code

  • Source code:
  ⚑ $ tree src/
  src/
  β”œβ”€β”€ imports
  β”‚   └── keda.sh.ts
  β”œβ”€β”€ keda-airflow.ts
  └── main.ts

  1 directory, 3 files
  • Build resource
  ⚑ $ npx projen build
  πŸ‘Ύ build Β» default | ts-node --project tsconfig.dev.json .projenrc.ts
  πŸ‘Ύ build Β» compile | tsc --build
  πŸ‘Ύ build Β» post-compile Β» synth | cdk8s synth
  No manifests synthesized
  πŸ‘Ύ build Β» test | jest --passWithNoTests --all --updateSnapshot
  No tests found, exiting with code 0
  ----------|---------|----------|---------|---------|-------------------
  File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
  ----------|---------|----------|---------|---------|-------------------
  All files |       0 |        0 |       0 |       0 |
  ----------|---------|----------|---------|---------|-------------------
  πŸ‘Ύ build Β» test Β» eslint | eslint --ext .ts,.tsx --fix --no-error-on-unmatched-pattern src test build-tools projenrc .projenrc.ts
  • Manifest yaml file
  ⚑ $ tree dist/
  dist/
  └── keda
      └── airflow-keda-so.yaml

  1 directory, 1 file

πŸš€ Apply and test

  • Apply manifest and check result
  # k apply -f

  # k get so -n airflow
  NAME               SCALETARGETKIND       SCALETARGETNAME   MIN   MAX   TRIGGERS   AUTHENTICATION                 READY   ACTIVE   FALLBACK   AGE
  airflow-worker-1   apps/v1.StatefulSet   airflow-worker    2     12    cron       keda-airflow-postgresql-auth   True    True     False      2d21h

  # k get hpa -n airflow
  NAME                        REFERENCE                    TARGETS                                  MINPODS   MAXPODS   REPLICAS   AGE
  keda-hpa-airflow-worker-1   StatefulSet/airflow-worker   500m/1 (avg), 500m/1 (avg) + 2 more...   2         12        2          2d21h

πŸš€ Conclusion

  • Within the Scaledobject class, you can jump to definition to understand the meaning of each properties and also know which required/optional attributes.
  • We can create a custom construct and base on that to provision multiple KEDA scaledobjects with customize specs/meta such as min/max/desired replicas, triggers, trigger authentication, etc.


This content originally appeared on DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’» and was authored by πŸš€ Vu Dao πŸš€


Print Share Comment Cite Upload Translate Updates
APA

πŸš€ Vu Dao πŸš€ | Sciencx (2022-09-25T14:32:38+00:00) Keda ScaledObject As Code Using CDK8S. Retrieved from https://www.scien.cx/2022/09/25/keda-scaledobject-as-code-using-cdk8s/

MLA
" » Keda ScaledObject As Code Using CDK8S." πŸš€ Vu Dao πŸš€ | Sciencx - Sunday September 25, 2022, https://www.scien.cx/2022/09/25/keda-scaledobject-as-code-using-cdk8s/
HARVARD
πŸš€ Vu Dao πŸš€ | Sciencx Sunday September 25, 2022 » Keda ScaledObject As Code Using CDK8S., viewed ,<https://www.scien.cx/2022/09/25/keda-scaledobject-as-code-using-cdk8s/>
VANCOUVER
πŸš€ Vu Dao πŸš€ | Sciencx - » Keda ScaledObject As Code Using CDK8S. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2022/09/25/keda-scaledobject-as-code-using-cdk8s/
CHICAGO
" » Keda ScaledObject As Code Using CDK8S." πŸš€ Vu Dao πŸš€ | Sciencx - Accessed . https://www.scien.cx/2022/09/25/keda-scaledobject-as-code-using-cdk8s/
IEEE
" » Keda ScaledObject As Code Using CDK8S." πŸš€ Vu Dao πŸš€ | Sciencx [Online]. Available: https://www.scien.cx/2022/09/25/keda-scaledobject-as-code-using-cdk8s/. [Accessed: ]
rf:citation
» Keda ScaledObject As Code Using CDK8S | πŸš€ Vu Dao πŸš€ | Sciencx | https://www.scien.cx/2022/09/25/keda-scaledobject-as-code-using-cdk8s/ |

Please log in to upload a file.




There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.