This content originally appeared on Level Up Coding - Medium and was authored by Lee James Gilmore
How to integration test DynamoDB service layers in your Serverless solutions using Jest, with visuals and code examples written in the AWS CDK and TypeScript.
Introduction
I was chatting to a colleague last week about integration testing DynamoDB calls in Serverless solutions, and how it can speed up development; and realised that the approach was a little difficult to describe out loud.
“I was chatting to a colleague last week about integration testing DynamoDB calls in Serverless solutions, and how it can speed up development; and realised that the approach was a little difficult to describe out loud”
For that reason I put together a relatively short article to discuss the approach with a quick code example showing how you could achieve it.
💡 Note: There is also a code repo associated which shows one way to integration test your DynamoDB service layers whilst in development.
How does this speed up development and give max confidence? ✔️
As most of us know, local development with Serverless is still both in its infancy, and still not fully effective. There are various tools we can use such as Serverless Offline and LocalStack, but the disadvantages far outweigh the advantages with local development in my opinion:
❌ Configuration between domain services can become a nightmare.
❌ Local development never gives the same level of confidence as the cloud.
❌ Local emulators are almost always out of date with new services and buggy (344 known issues currently with LocalStack for example)
The move to working on the cloud! ☁️
This means that most development teams then start with ephemeral environments, deploying and testing directly on the cloud, which can be very slow and tedious, waiting for CloudFormation to complete each time a change is deployed. Some tools are starting to enhance this process, such as SAM Accelerate and Serverless Cloud, but again it is in its infancy in my opinion. (Discussion around ephemeral environments below)
Enterprise Serverless 🚀 Architecture
Another approach ✔️
I personally don’t deploy my code that often when developing my Serverless solutions, and would rather unit test the code and its boundaries/interfaces fully which make up the solution using TDD, and integration test any areas where I am likely to see issues when deploying and testing in the cloud (this is where I want to speed up the process), giving me full confidence before actually deploying and performing full e2e tests.
A typical example of an area which is error prone and slows down the cycle of development and deployment is using the AWS DynamoDB SDK within a service layer and getting the parameters (inc queries etc) correct first time.
We can use this same integration testing approach for DocumentDB, SNS, S3 etc, and fully maximise this using service layer packages or reusing them in monorepos across many serverless solutions (think Lerna or Nx).
This is fairly common practice, and a recent pole showed that the majority of serverless teams are either working fully in the cloud or a hybrid of local and cloud where it makes sense to do so:
Let’s see this approach visually 👀
OK, so lets work through how teams tend to work over time:
New Serverless teams try local development.
Serverless teams can get excited when starting to setup local development environments, and developer testing using services such as serverless-offline, localstack, serverless-dynamodb-local, serverless appsync plugin etc..
Teams realise it is a nightmare over time, and use ephemeral environments instead (or a hybrid of the two approaches).
Over time, as the Serverless team needs to integrate with more and more micro and domain services as the organisation grows (as well as additional teams); and increase their use of other AWS services; they soon realise that overall local development becomes slow and cumbersome due to the additional config between local and cloud development, and the amount of mocking required. A move to ephemeral cloud development is a must!
Teams have less cognitive load now, but the deployment loop to the cloud can be slow.
One of the tradeoffs now with ephemeral environments when working directly on the cloud is that the deploying of changes can be slow (especially with CloudFormation), and this feedback loop can really slow teams down. (time to grab a coffee….)
Teams utilise local integration tests to speed up Serverless development as a hybrid of the two approaches.
Utilising local development where it makes sense, for example integration tests of service layers before deploying to the cloud using Docker containers, can really speed up development in my experience. Let’s look at the approach and code examples below.
From a process flow point of view we can see the difference made when adding the confidence of integration testing to the mix (less chance of issues once deployed and the constant cycle of fix + deploy + test):
Getting Started! ✔️
To get started, clone the following repo with the following git command:
git clone https://github.com/leegilmorecode/serverless-dynamodb-integration-tests
This will pull down the example code to your local machine.
Testing the solution 🎯
Unit and Integration testing setup
Lets firstly look at the unit test setup with Jest. We create a jest config specifically for our unit tests looking for *.spec.ts files:
and we add a separate one for our integration tests looking for *.int.ts files:
both of which are pulled in via npm scripts as shown below (passing through the different config paths):
"test": "jest --config jest.unit.config.js",
"test:int": "LOCAL=true jest --config jest.int.config.js",
This means we can differentiate between the two sets of tests.
💡 Note: for the integration tests we also pass through an environment variable called LOCAL which is set to true, which allows us to configure DynamoDB for local or cloud.
Integration tests against a local DynamoDB
For the integration testing of DynamoDB locally, we use a Docker Container which uses DynamoDB Local through a docker-compose.yml file:
and in our pre npm scripts we start the container with the following:
"pretest:int": "docker-compose up -d dynamodb-local"
Which means when our integration tests start using Jest we have the container running to work against i.e. a local DynamoDB instance.
💡 Note: We stop the container with a post npm script once the tests have ran too.
Now when the integration tests run we ensure that we unmock the aws-sdk (as we want to utilise the actual DynamoDB calls to the local instance), create the orders table in the beforeAll Jest hook to run the tests against, and delete the local table in the afterAll Jest hook after the tests have ran:
This means that when we run our integration tests during development we have maximum confidence that it will work before pushing to the cloud!
Unit testing with mocks
When we run our unit tests we want to ensure that we mock the DynamoDB calls as we don’t want to integrate against any actual infra or database for unit testing. We do this by mocking the aws-sdk using the __mocks__ folder to mock node modules/packages as shown below using Jest mocks:
Now when we run our unit tests we can mock out the actual return values instead of hitting an actual database (example shown below)
We can manipulate the response values (whether that be rejected or resolved) so we can test our services fully.
Conclusion
We can use this approach for all of integration tests, for example a MongoDB container for a DocumentDB service layer, or even integration test fully against the cloud when testing S3 or SNS. If we publish these services as packages, or utilise and share them using monorepos, we get the full confidence that they work as expected before deploying changes for developer testing and e2e tests.
This article is sponsored by Sedai.io
Summary
I hope you found that useful!
Go and subscribe to my Enterprise Serverless Newsletter here for more of the same content:
Enterprise Serverless 🚀 | LinkedIn
Wrapping up 👋
Please go and subscribe on my YouTube channel for similar content!
I would love to connect with you also on any of the following:
https://www.linkedin.com/in/lee-james-gilmore/
https://twitter.com/LeeJamesGilmore
If you found the articles inspiring or useful please feel free to support me with a virtual coffee https://www.buymeacoffee.com/leegilmore and either way lets connect and chat! ☕️
If you enjoyed the posts please follow my profile Lee James Gilmore for further posts/series, and don’t forget to connect and say Hi 👋
Please also use the ‘clap’ feature at the bottom of the post if you enjoyed it! (You can clap more than once!!)
About me
“Hi, I’m Lee, an AWS Community Builder, Blogger, AWS certified cloud architect and Principal Software Engineer based in the UK; currently working as a Global Serverless Architect, having worked primarily in full-stack JavaScript on AWS for the past 6 years.
I consider myself a serverless advocate with a love of all things AWS, innovation, software architecture and technology.”
*** The information provided are my own personal views and I accept no responsibility on the use of the information. ***
You may also be interested in the following:
Serverless DynamoDB Integration Tests with Jest 🚀 was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Lee James Gilmore
Lee James Gilmore | Sciencx (2022-03-10T15:11:13+00:00) Serverless DynamoDB Integration Tests with Jest. Retrieved from https://www.scien.cx/2022/03/10/serverless-dynamodb-integration-tests-with-jest/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.