This content originally appeared on Level Up Coding - Medium and was authored by Lee James Gilmore
Load, smoke, functional and fuzz testing serverless applications using the Serverless Framework and Artillery, with supporting code samples!
Introduction
The following article describes how you can load test your serverless applications using Artillery and the Serverless Framework, with the supporting code repo being available here. There are examples of load, smoke, fuzz and functional tests, with generated reports.
Why load test?
“Why would you load test your fully scalable serverless applications? Surely the reason for moving to serverless was too negate the need to load test in the first place??” ?
When working with enterprise level serverless architectures you still need to load test due to other key factors in my opinion; for example:
❌ less scalable downstream services and how they react to a sudden scale up of lambda (perhaps integrated legacy systems or database technologies with max connections)
❌ asynchronous eventually consistent processes falling within certain acceptable thresholds for the end user.
❌ hard and soft AWS regional limits.
❌ how reserved and provisioned concurrency configuration has affected the endpoints at scale.
❌ memory configuration on the lambdas.
❌ and even poorly written code within the lambdas itself (memory leaks, poor batching configuration etc etc).
This is shown below visually:
What is Artillery?
Artillery is an open source load and functional/smoke testing solution which can be installed as a dependency of your serverless solution using NPM, configured using a yml file and accompanied CSV file for load test data, and ran within your pipelines for regular load/smoke testing. There are also additional community plugins to use, one of which allows fuzz testing.
The example architecture
The basic example serverless code repo which accompanies this blog article has the following architecture:
Please note this is the minimal code and architecture to demonstrate the use of artillery, so this is not production ready and does not adhere to coding and architecture best practices
- Artillery uses the employees in the CSV file to create new employees through the POST employees endpoint (AWS API Gateway > AWS Lambda).
- The employee records are persisted to the employees AWS DynamoDB table.
- Artillery iterates the CSV file, performing a GET employee on the individual employees.
What benefits would the tests in this scenario give?
In the simple architecture scenario above, using Artillery you would be able to assert:
- Do we have the provisioned capacity set correctly on DynamoDB or will we get throttling due to lack of available resources? Should we increase it, or potentially change too ‘on demand’ with our expected scale? (what would a Black Friday equivalent load look like for example?)
- Do we have the reserved/provisioned concurrency set correctly on the lambdas? How do the cold starts look, and are we getting 429 style responses due to load?
- Do the response times (latency) fall within our NFRs for our end users? i.e. are the average response times acceptable? An example could be an employee downloading their generated payslip, a service generating bespoke PNG images of memes, a service which checks your credit check against an external service etc
- Are the asserted API responses correct functionally? i.e are we getting back the properties that we expect following a new release with the correct values?
- What responses do we get if we over/under post, send bad/malicious payloads etc? Is our API validation working correctly?
As you can see, even in this very trivial example above there are many benefits to the artillery testing. Extrapolate this out with potential async processing and more AWS services/configurations involved, integrated downstream legacy/slower systems, along with functional changes made to the API; shows how useful this approach can be from a joint load and functional/smoke test suite perspective running in a pipeline.
How to deploy the example code?
? Note: Running the following commands will incur charges on your AWS account so change the config accordingly.
In the root of the folder run npm i and then npm run deploy:develop which will install all of the dependencies and then deploy to AWS.
This will generate the resources for you using the Serverless Framework i.e. the API, compute layer and database. You will see similar output to this:
Then, in the tests/load.yml file amend the target property in the environments > develop section to be the base url generated when you ran the deploy (I have put in entries for staging and production just to show how this would work). An example:
environments:
develop:
target: "https://something.execute-api.eu-west-1.amazonaws.com/develop"
? In reality this would be your well known domain endpoints using AWS Route 53 and CloudFront for example, which I won’t do for this simple example obviously — but you wouldn’t typically be amending this file like we have above.
You can then run the load test using npm run test:load:develop, which on success will generate a develop.json file in the results folder, and a timestamped report in the reports folder.
You can also run npm run test:smoke:develop which will run a set of smoke tests rather than a load test, and generate a report as above.
Finally, for using artillery for running functional tests (including fuzzing) run npm run test:func:develop which will produce a text file report of the tests ran (including any failed tests and the setup data/scenario which caused the failure etc)
? The mock employee data in the CSV file was generated using https://www.mockaroo.com/ which is a free excellent resource for developers
How is this configured?
The main artillery configuration is split out into two files (in practice you would split this further):
load.yml for the load and smoke tests.
functional.yml for the functional tests (inc fuzzing).
The yml files are invoked through NPM scripts, each of which runs the tests generating an output JSON file, and a post script which takes the JSON file and generates a report:
"test:smoke:develop": "DEBUG=plugin:expect $(npm bin)/artillery run --output ./tests/data/results/smoke-develop.json -e smoke-develop ./tests/load.yml",
"posttest:smoke:develop": "$(npm bin)/artillery report --output ./tests/data/reports/smoke/report-smoke-develop-\"$(date \"+%Y-%m-%d_%H-%M-%S\")\".html ./tests/data/results/smoke-develop.json",
load.yml
The load testing file load.yml is shown below with verbose comments:
? As you can see, the only difference between the smoke tests and the load tests is that the smoke tests are ran through one at a time sequentially as one virtual user i.e. no load.
The data for the tests are being pulled from a data.csv file with the following line in the config:
payload:
path: "./data/data.csv" # pull in the employee data csv
You can also see that the expect plugin is being used to ensure that the:
✔️ correct status codes are being returned.
✔️ the correct content type is returned.
✔️ the response has the correct properties.
✔️ some testing of the returned values being correct, for example fullName which is a concatenation of the firstName and surname properties within the code.
functional.yml
The functional and fuzzing tests can be seen in the functional.yml file extract below with verbose comments:
You can see from above that the fuzzer plugin has been introduced, which will generate bad input to be used within out tests i.e. shown here:
firstName: "{{ naughtyString }}"We are explicitly testing:
✔️ ️that our payload validation within our lambda functions are working correctly.
✔️ checking for the correct expected response codes.
✔️ checking the correct responses are being returned.
? The functional tests are ran with one virtual user similar to the smoke tests.
We have added a basic validate function in our code which takes JSON schema using AJV to check our incoming payloads, and if there are any failures it throws a custom error type called AppError:
The custom AppError error type allows us to distinguish between known errors we have thrown in code (for example bad requests we have validated through AJV), and uncaught errors. We do this as we don’t want to return stack traces or secrets in the response if there is an unknown error!
An example of a basic generic error handler to be used in the lambda handlers is shown here (default being 500 error with a standard error message):
We can then test the payload input using JSON schema as shown below for the create employee endpoint:
And if the payload fails validation we can return the correct response which can be validated using our functional tests above!
Reporting! ?
Once the tests have ran, an NPM post script generates a local report in the reports folder for the given environment and test type; for example: report-load-develop-2021–07–28_20–27–07.html
You can then open up the basic generated report and view it which shows the following example output:
If you want to see a fancier style of report which you get from the generated JSON output, you can go to https://reportviewer.artillery.io/ and upload the generated JSON file to get the following results:
In most browsers you can then select > edit > print > print to pdf to get a downloadable report in PDF format.
The following configuration in both yml files ensures that the pipeline will fail if we have any errors, or if the latency is over a certain threshold:
ensure:
p95: 1000 # ensure latency is equal to or under 1000ms
maxErrorRate: 0 # no percentage of error rate i.e. no errors or pipeline fails
Plugins ?
Artillery has a host of plugins which can be used in conjunction with it to extend its functionality, for example:
artillery-plugin-cloudwatch — A plugin for artillery.io that records response data into AWS CloudWatch.
artillery-plugin-fuzzer — This plugin makes it easy to run some fuzz testing on your HTTP API. Use it to send unexpected payloads to your endpoints, see what breaks and fix it to make your application more secure and more resilient. The payloads generated by this plugin are based on the awesome Big List Of Naughty Strings.
artillery-plugin-sns — A plugin for artillery.io that publishes response data to an AWS SNS topic.
artillery-plugin-dynamodb — A plugin for artillery.io that publishes stats to AWS DynamoDB.
See more plugins here, and you can obviously create your own too.
Scaling up to enterprise level load testing! ??
With the community addition of Artillery, the number of requests you can make is constrained by the resources of the machine the tests are running from (i.e. performance and network).
To mitigate this and go beyond average scale, you can use serverless-artillery, which allows you to utilise the benefits of AWS Lambda to scale out your Artillery tests to enterprise levels (rather than running from one machine).
Use serverless-artillery if:
- You want to know if your high traffic services (either internal or public) can handle a huge amount of traffic load (i.e. performance or load testing).
- You want to test if your services behave as you expect after you deploy new changes (i.e. acceptance testing).
- You want to constantly monitor your enterprise services over time to make sure the latency of your services is under control (i.e. monitoring mode).
Alternatives
One thing to mention here is that there are other more established tools on the market, such as Apache JMeter; however artillery may suite your current needs and it is very lightweight since it is installed using NPM, can run in a CI pipeline, and there is also a Pro version of Artillery which has enhanced features and more established integrations with AWS etc
In my opinion it is the right tool for the job, and another tool to have in your mental toolkit when architecting and developing new serverless solutions! ?
Wrapping up
I would love to connect with you on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you found the articles inspiring or useful please feel free to support me with a virtual coffee https://www.buymeacoffee.com/leegilmore and either way lets connect and chat! ☕️
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi ?
Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)
About me
“Hi, I’m Lee, an AWS certified technical architect and Lead Software Engineer based in the UK, currently working as a Technical Cloud Architect and Principal Serverless Developer, having worked primarily in full-stack JavaScript on AWS for the past 5 years.
I consider myself a serverless evangelist with a love of all things AWS, innovation, software architecture and technology.”
** The information provided are my own personal views and I accept no responsibility on the use of the information. ***
Serverless load testing at scale with Artillery ? was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Lee James Gilmore
Lee James Gilmore | Sciencx (2021-08-03T13:58:45+00:00) Serverless load testing at scale with Artillery. Retrieved from https://www.scien.cx/2021/08/03/serverless-load-testing-at-scale-with-artillery/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.