This content originally appeared on Level Up Coding - Medium and was authored by Madhusudhanan
In this blog, we’ll learn how to implement pagination with API Gateway, Lambda, and DynamoDB. I assume you already know at least the basics of these services and this blog is mainly focused on DynamoDB. Simply put pagination is the process of dividing the entire data into smaller chunks. This chunk of data can be fetched gradually. Pagination usually involves the following terms.
- Page — a pointer that represents the current chunk of data
- Count — total no of items to fetch on a page
- Total — count of all items present in the database (the data which we want to paginate)
Additionally, there can be filters applied to those data to refine the search results. This is a typical design of a pagination API. The client (website) will send the current page number and count properties as the query string parameters of the endpoint. But here it will be a bit different.
Let’s start by developing a REST endpoint that will be deployed on the API Gateway. The API will use the Lambda function as the backend to implement pagination in DynamoDB. You can use whatever runtime you’d like. But, I’m gonna use Typescript. Assume we are developing a discussion forum website and we want to list all the posts on a topic. We’ll have the following models/ entities.
- Topics — Root-level grouping of all posts
- Posts — There will be zero or more posts in a channel
Our endpoint would look something like this.
GET /posts?topicId=1&count=10
A short intro about supported read operations in DynamoDB for the collection of items. There are two types: Scan and Query. I’m not gonna go into this in detail. But, Scan should be avoided as it involves a “full-table scan”. This is a costly operation. Query fetches records only on a specific partition. Generally, you should always favor Query over Scan. More info here. So, we are gonna use the Query operation. Please note that for Query we need to specify the primary key (partition key and optional sort key).
As I said earlier, in DynamoDB things are a bit different for pagination. Although it divides the data into pages, there’s no Page involved; well, not directly. There are two different ways we can implement it. The process is quite similar in both ways but the place where it is implemented will differ. The process is explained in AWS’s official documentation. Here’s the link. In short, the process starts by executing a Query operation with the given input. If there are records more than 1 MB in size or if we have applied the limit, then the DynamoDB returns a property called LastEvaluationKey. We’d repeat the Query operation until this property becomes null or undefined.
async function getAllPosts(topicId) {
let queryResult;
let items = [];
let params = {...}; // For brevity this is not written
do {
queryResult = await ddbClient.query(params).promise();
items = items.concat(queryResult.Items);
params.ExclusiveStartKey = queryResult.LastEvaluationKey;
} while (typeof queryResult.LastEvaluationKey !== "undefined");
return items;
}Simply speaking, this property is like a cursor in the database. This will contain the primary key of the item where the operation was stopped (same as in the documentation). Here we don’t iterate over the records. Instead, we’d stop and send the response. Our response body comprises the result set and the LastEvaluationKey. Here’s where the difference comes in.
The check for the existence of the LastEvaluationKey property is done on the client side. If it is not null, we need to pass this property in the subsequent API calls. Because this is going to be the start of fetching the next page. For subsequent Query operations, we’d pass this value to the ExclusiveStartKey property — a property that takes LastEvaluationKey and continue the read operation; it’s more like OFFSET in SQL. If LastEvaluationKey is null or undefined, then we have reached the “last page”. Note that there may be a case where LastEvaluationKey is not null, but the result set is empty. This is because you may have applied any filters.
In the above diagram, after the third API call, the LastEvaluationKey property is not sent, which means we have reached the last page. Once fetching all paged records is done, it’s up to the client how to maintain them on the browser. Notice I have used nextPageKey. This is just a meaningful name I wanted to use in the endpoint. You can name it anything you like. The main difference with this approach is the iteration of all pages is done on the client side rather than on the server side (given in the documentation). This way we are achieving the typical pagination concept with the LastEvaluationKey property. The TypeScript code is given below. For brevity, I added only a few lines of code. You can extend this by adding filters like getting all posts between the range of date & time, by author, etc.
Let me walk you through the code. We start by getting the input from API query string parameters. The nextPageKey property might be null on the first call. So we construct Query operation params accordingly. Note that I have encoded & decoded the value of LastEvaluationKey as a Base64 string. This is just to have cleaner look. Also if you have a sort key or index defined, this can be in JSON. I designed the Topics table to have an attribute (column) called totalPosts. Whenever a new post is created under a topic, I increment its value by 1 and whenever the post is deleted, I decrement its value by 1. So, it is easy for me to get the total number of posts for a given topicId. And finally, we send the response. Note that the response body has additional fields. Because this lambda function handler is defined as “Lambda proxy integration” in API Gateway. But our final response body from API Gateway would look like this.
{
"data": [{}, {}, {}...],
"pagination": {
"count": 10,
"total": 100,
"nextPageKey": "776f726c64"
}
}That’s it. Thank you! Have a nice day :)
Learn how to implement pagination with API Gateway, Lambda, and DynamoDB was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Madhusudhanan
Madhusudhanan | Sciencx (2023-01-19T14:50:03+00:00) Learn how to implement pagination with API Gateway, Lambda, and DynamoDB. Retrieved from https://www.scien.cx/2023/01/19/learn-how-to-implement-pagination-with-api-gateway-lambda-and-dynamodb/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.