This content originally appeared on Bits and Pieces - Medium and was authored by Fernando Doglio
Learn how easy it is to create clickmaps with this technique
Redis is an in-memory database that supports a wide range of data structures, including strings, lists, sets, hashes, and sorted sets.
It also supports streams and pubsub types of messages. This is my absolute favorite database!
That said, one of the lesser-known data structures in Redis is the HyperLogLog (HLL), which is used for efficiently counting unique items in a set. I know, it sounds strange and complicated, but it’s not (I promise!).
In this article, we will explore how HLL can be used to create a clickmap for a Next.js app.
What are HyperLogLogs?
HyperLogLogs are a data structure used for estimating the cardinality of a set. Cardinality is the number of distinct elements in a set. HyperLogLogs use probabilistic algorithms to estimate cardinality with a high degree of accuracy, while using only a small amount of memory.
The reason why we’d want to use an HLL instead of directly using a Redis Set and count the items in it, is that when the number of elements grows enough, then the Set starts using too much memory. Keep in mind that we only want to know the number of elements, we don’t care about the actual elements inside it. This is why the HLL is such a great alternative.
HyperLogLogs support several operations, including adding elements to a set, merging sets, and estimating the cardinality of a set. The accuracy of the estimate depends on the number of registers used in the table and the length of the digest. Redis provides a default length of 12 bits for the digest, which corresponds to an error rate of 0.81% (which is going to be more than enough for us).
How can HyperLogLogs be used to create a clickmap?
A clickmap is a graphical representation of user clicks on a web page. Clickmaps are useful for analyzing user behavior, identifying popular content, and optimizing user experience. Clickmaps are typically created using JavaScript libraries that capture user clicks and send them to a server for storage and analysis.
HyperLogLogs can be used to create a clickmap by storing the number of clicks for each element on a page in a HyperLogLog. Each element on a page, such as a button, link, or image, is assigned a unique identifier, such as a CSS class or ID. When a user clicks on an element, the identifier is added to the HyperLogLog. The number of clicks for each element can then be estimated by querying the HyperLogLog.
Let’s take a look at how we could implement this inside a NextJS application.
Implementing our own ClickMap
The final result of what we’re going to implement will look like this:
What you’re seeing here is:
- The main page of a fake blog built in NextJS.
- Added borders on all elements that have ever been clicked.
- The colors of the borders go from light green (fewer clicks) to bright red (most clicks).
Also, what you’re not seeing:
- Every time we click anywhere, our code will save the click on an HLL.
- Every two seconds, we’ll query the HLL to update the clickmap.
To implement this clickmap using HyperLogLogs in Redis, follow these steps:
Step 1: Define the elements to track
Identify the elements on the page that you want to track clicks for. These could be buttons, links, images, or any other interactive elements. Assign a unique identifier to each element. These identifiers will be used to add clicks to the HyperLogLog.
We’re going to create a component to do that for us with the following code:
import { useEffect } from "react";
import { useRouter } from 'next/dist/client/router';
async function getUniqueIDForElem(e) {
const encoder = new TextEncoder();
const data = encoder.encode(e.innerHTML)
const hashBuffer = (await crypto.subtle.digest('SHA-256', data))
const hashArray = Array.from(new Uint8Array(hashBuffer));
const hashHex = hashArray.map((b) => b.toString(16).padStart(2, '0')).join('');
return hashHex;
}
async function trackClick(event) {
event.stopPropagation()
const target = event.target
const userId = Math.round(Math.random() * 200)
const key = target.getAttribute("id")
const url = "/blog/api/clicks"
const resp = await fetch(url, {
method: "POST",
body: JSON.stringify({
userId,
cssId: key,
timestamp: (new Date()).getTime()
})
})
}
export default function Comp() {
const router = useRouter();
const { id } = router.query;
useEffect(() => {
async function run() {
let all = document.querySelectorAll('*')
all.forEach( async (elem) => {
if(elem.getAttribute("id") == null) {
elem.setAttribute("id", await getUniqueIDForElem(elem))
}
elem.removeEventListener("click", trackClick)
elem.addEventListener("click", trackClick)
})
}
run()
}, [id])
return <div></div>
}This component is doing a couple of things:
- It’s querying the DOM and getting ALL elements.
- For each element without an ID, it’ll create one with the getUniqueIDForElem function. This function returns a hashed string using the content of the element, this way we ensure that the ID is deterministic and that it’ll always return the same value. For each element it’ll also set an “onClick” event to track the actual click.
- The trackClick function will calculate a random user id (this here is optional, I’m just adding the option to make sure users clicking at the same time don’t register as the same click), and then it’ll send a POST request to an internal API endpoint with the relevant information to track (i.e the ID of the element, the user clicking and the timestamp of the click).
We’re now ready to move on to the back-end.
Step 2: Initialize a HyperLogLog for each element
In this step, we’ll create a new HyperLogLog for each element using the Redis command PFADD. We do this because we can’t query the HLL and ask for the cardinality of one particular element, so we’ll keep track of clicks in separate keys.
The HyperLogLog should be named after the identifier for the element. In our case, we’ll use the string “myHyperLogLog_” and then we’ll add the actual CSS ID.
Inside these HLLs, we’ll add a string concatenating the user id and the timestamp, this will make sure that we don’t track potential phantom clicks or fake clicks coming at the exact same time from the same user.
The code to do that will reside inside the api/clicks.js file:
import {createClient } from 'redis';
export default async function handler(req, res) {
if (req.method !== 'POST') {
res.status(405).json({ error: 'Method not allowed' });
return;
}
const { userId, cssId, timestamp } = JSON.parse(req.body);
if (!userId || !cssId || !timestamp) {
res.status(400).json({ error: 'Missing required fields' });
return;
}
const redisClient = createClient();
await redisClient.connect()
try {
const result = await redisClient.pfAdd('myHyperLogLog_' + cssId, `${userId}-${timestamp}`)
} catch(err) {
console.error(err);
res.status(500).json({ error: 'Internal server error' });
return;
}
console.log(`Added ${userId}-${cssId}-${timestamp} to HyperLogLog`);
redisClient.quit();
res.status(200).json({ message: 'Data saved successfully' });
}As you can see, the code is actually quite simple, most of it is just boilerplate to connect to Redis and check that we’re getting all the data we need.
After that, we’re simply executing the pfAdd command.
As a note, I’m using the Redis npm module, feel free to use whatever you want in your case.
At this point, we’ve done what we wanted, we’re already saving the data into Redis. Now it’s time to display the actual clickmap.
Step 3: Estimate the number of clicks for each element
To estimate the number of clicks for each element, use the Redis command PFCOUNT to retrieve the cardinality of each HyperLogLog. The cardinality corresponds to the number of unique elements that have been added to the HyperLogLog.
We’ll calculate the total number of clicks inside a page using all the cardinalities and then, we’ll assign a rough percentage to each one when displaying the borders.
To do that, we’ll add a component that will take care of polling the values from a new API endpoint and adding the corresponding classes:
import { useEffect, useState } from "react"
import { useRouter } from 'next/dist/client/router';
import styles from '@/styles/Home.module.css'
export default function Loader() {
const router = useRouter();
const { id } = router.query;
const [started, setStarted] = useState(false)
async function updateMap() {
const data = await fetch("/blog/api/click_maps")
const jsonData = await data.json()
let totalClicks = jsonData.reduce((total, elem) => total + elem.total, 0)
if(totalClicks == 0) return
//let's calculate the percentage of each element's clicks
jsonData.forEach(elem => {
let p = ((elem.total * 100) / totalClicks)
p = Math.ceil(p/10) * 10
console.log(p)
const domElem = document.getElementById(elem.cssId)
if(domElem) {
domElem.setAttribute("className",styles["hm-" + p] )
}
})
}
useEffect( () => {
if(!started) setInterval(updateMap, 2000)
setStarted(true)
}, [id])
return <div></div>
}This function sets an endless interval to poll the data every 2 seconds. The data will contain all the HLLs currently available, their CSS ID and their individual cardinality.
So once we have the list, we’ll calculate the total with the reduce one-liner and then we’ll proceed to calculate the rough percentage. I say “rough percentage” because I’m rounding all numbers to tens. So even if the actual percentage is 16.2654 I’m turning it into a 10.
That way I can have 10 pre-defined classes in my stylesheet where I set the actual border colors.
The endpoint associated with this component is inside api/click_maps.js :
import {createClient } from 'redis';
export default async function handler(req, res) {
if (req.method !== 'GET') {
res.status(405).json({ error: 'Method not allowed' });
return;
}
const redisClient = createClient();
await redisClient.connect()
try {
const keys = await redisClient.keys("*")
let returnValue = []
for(const key of keys) {
const result = await redisClient.pfCount(key)
const cssId = key.replace("myHyperLogLog_", "")
returnValue.push({
cssId,
total: result
})
}
res.status(200).json(returnValue)
} catch(err) {
console.error(err);
res.status(500).json({ error: 'Internal server error' });
return;
}
}I’m simply pulling all the keys with the keys method (if you have other keys in your Redis DB, you’ll want to use a better pattern).
Then I go through each one and create a list of objects with 2 properties:
- The cssId value (to identify it on the front-end).
- The cardinality in the total property.
That’s all there is to it.
Clearly, this endpoint and the communication between the front-end and back can be optimized a bit to make sure we’re only pulling HLLs that are present in the page and we can do some other calculations in the back-end as well.
But the example works, and every 2 seconds the border colors of all your clicked elements will update automatically.
Cool, uh?!
Did you like what you read? Consider subscribing to my FREE newsletter where I share my 2 decades’ worth of wisdom in the IT industry with everyone. Join “The Rambling of an old developer” !
Benefits of using HyperLogLogs for clickmaps
Using HyperLogLogs for clickmaps offers several benefits over traditional click-tracking methods.
- HyperLogLogs use probabilistic algorithms to estimate the number of clicks, which allows for accurate estimates while using minimal memory. This is particularly useful for large-scale applications where tracking individual clicks for every user would require too much memory.
- HyperLogLogs can be easily distributed across multiple Redis nodes using Redis Cluster. This allows for high availability and scalability while maintaining the accuracy of the click data.
- Even though we are storing the user ID (whatever that might be for your case), HyperLogLogs ensure privacy because the data you store in them can’t be retrieved. You can only know the cardinality of the set but now what’s inside. So if you’re worried about keeping “too much” track of your users and how they might see it, HLLs make sure you only keep what you really need.
- Finally, HyperLogLogs are designed for high write throughput, which makes them ideal for capturing user clicks in real-time. This is particularly useful for applications that require real-time analysis of user behavior, such as e-commerce sites or social networks.
Conclusion
HyperLogLogs are fun! I hope you, at least, got that out of this article.
While their name might seem like they’re this complex data structure that is very hard to use, the reality is quite different.
HyperLogLogs are a powerful data structure that allows for the accurate counting of unique items in a set while using minimal memory.
By tracking user clicks, like we did today, using HyperLogLogs, we can create a clickmap that provides valuable insights into user behavior and allows for real-time analysis of user engagement.
If you’re building a web application that requires click tracking, consider using HyperLogLogs in Redis to efficiently and accurately capture user behavior.
Have you used HLL before? What did you use them for?
Build Apps with reusable components, just like Lego
Bit’s open-source tool help 250,000+ devs to build apps with components.
Turn any UI, feature, or page into a reusable component — and share it across your applications. It’s easier to collaborate and build faster.
Split apps into components to make app development easier, and enjoy the best experience for the workflows you want:
→ Micro-Frontends
→ Design System
→ Code-Sharing and reuse
→ Monorepo
Learn more:
- How We Build Micro Frontends
- How we Build a Component Design System
- How to reuse React components across your projects
- 5 Ways to Build a React Monorepo
- How to Create a Composable React App with Bit
Track User Behavior in Real-Time: Redis HyperLogLogs for NextJS ClickMap was originally published in Bits and Pieces on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Bits and Pieces - Medium and was authored by Fernando Doglio
Fernando Doglio | Sciencx (2023-03-04T07:01:32+00:00) Track User Behavior in Real-Time: Redis HyperLogLogs for NextJS ClickMap. Retrieved from https://www.scien.cx/2023/03/04/track-user-behavior-in-real-time-redis-hyperloglogs-for-nextjs-clickmap/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.