This content originally appeared on Level Up Coding - Medium and was authored by TJ. Podobnik, @dorkamotorka
Anteon: Monitoring Encrypted Traffic on Kubernetes using Alaz eBPF Agent
The landscape of microservice applications is rapidly expanding, along with the variety of communication protocols and the complexity of interactions between services. In applications requiring high confidentiality, traditional traffic sniffing techniques often face limitations in accessing encrypted data, hindering the genuine observation of the communication channel. This restriction poses significant obstacles for real-time observability and monitoring tools used for system debugging and performance analysis.
In this blog post, we’ll introduce our open-source eBPF Agent solution, focusing on its ability to intercept and monitor TLS/SSL encrypted traffic. This feature enables our costumers to detect traffic patterns that could be overloading the system and impacting performance. We will start with a theoretical overview and end with a practical code example that you can try out yourself.
The Challenge of TLS/SSL on Kubernetes
In our recent post, we explored the observability of the PostgreSQL (Layer 7) protocol. This involved attaching our eBPF programs to the Kernel tracepoints that capture application data sent and received over the socket. You can find the post at the following link:
Monitoring PostgreSQL Database on Kubernetes using eBPF
However, secure communication channels pose a unique challenge. By the time data reaches the Linux send() and recv() syscalls for TLS/SSL traffic, it is already encrypted and indecipherable.
How can we effectively monitor this encrypted traffic without compromising security?
If you want to trace out this data in its unencrypted form, eBPF allows us to attach custom programs just before data is encrypted and sent, and after it is received and decrypted. While technically it’s possible to achieve this by hard-coding observability into your application, eBPF offers several significant advantages:
- No code changes required: eBPF operates independently of your application.
- Dynamic implementation: It can be added without restarting your app.
- Efficient resource usage: Often executed directly in the kernel, eBPF requires fewer CPU and memory resources compared to user-space telemetry collectors.
- Enhanced context extraction: It can gather much more application context than user-space provided attributes.
By leveraging eBPF, we can observe and monitor applications of all types, from monoliths to microservices, whether they are deployed as FaaS, on Kubernetes, or any other architecture you can imagine — even for TLS/SSL encrypted traffic. But the key question still remains: How does it work?
eBPF uprobes and Kubernetes Monitoring
In 2004, Linux introduced support for kernel probes (“kprobes”) with the 2.6 Linux Kernel, marking the beginning of the era of kernel function tracing. Kprobes allowed developers to attach small programs, or probes, to the kernel without modifying its source code. This innovation paved the way for numerous modern applications, ranging from performance monitoring and profiling to intrusion detection. Furthermore, with the release of the 3.5 Kernel, support for user probes (“uprobes”) was added, extending the capabilities of dynamically attached probes from the kernel into user space. While the primary distinction between kprobes and uprobes lies in their attachment points — kernel space and user space, respectively — there are also two other related types of probes:
- Kprobes/Uprobes: Gather information by intercepting functions as they are called.
- Kretprobes/Uretprobes: Executed when a function returns.
Nowadays, both types of probes can dynamically be attached to mostly all possible functions, but they differ in the data they capture. Kprobes and uprobes intercept input parameters to functions, while kretprobes and uretprobes capture returned values. These probes often work together, rather than as separate entities, to provide comprehensive tracing.
If you’re searching for a specific kernel source function name, you can use cat /proc/kallsymsor bpftrace -l ‘kprobe:*’. This command accesses the kernel's symbol table, which contains both function names and variable names.
As TLS/SSL traffic encryption is processed by user space libraries, kernel probes are of no help to us here. Thankfully, user probes can help us observe the data from any application that uses this shared library in the clear, before it is encrypted or after it has been decrypted — be it on bare metal servers or Kubernetes cluster. And there is no need for any keys, because those are already being provided by the application.
We, at Anteon have developed an eBPF agent that does just that. To be more specific, we provide support for observing traffic encrypted using OpenSSL library including its legacy versions. In order to achieve that we hook onto:
- SSL_write(): function used in the OpenSSL library to write data to an SSL/TLS connection. We primarily utilize it to intercept the data written by the client — not yet encrypted.
SEC("uprobe/SSL_write_v3")
void BPF_UPROBE(ssl_write_v3, void * ssl, void* buffer, int num) {
ssl_uprobe_write_v_3(ctx, ssl, buffer, num, 0);
}
- SSL_read(): function used in the OpenSSL library to read data from an SSL/TLS connection. It allows us to intercept the decrypted data received by the server as well as parsing the return code e.g. HTTP status code like 200 (Successful GET request).
SEC("uprobe/SSL_read_v3")
void BPF_UPROBE(ssl_read_enter_v3, void* ssl, void* buffer, int num) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u32 pid = pid_tgid >> 32;
__u64 id = pid_tgid | TLS_MASK;
ssl_uprobe_read_enter_v3(ctx, id, pid, ssl, buffer, num, 0);
}
SEC("uretprobe/SSL_read")
void BPF_URETPROBE(ssl_ret_read_v3) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
__u64 pid = pid_tgid >> 32;
__u64 id = pid_tgid | TLS_MASK;
int returnValue = PT_REGS_RC(ctx);
process_exit_of_syscalls_read_recvfrom(ctx, id, pid, returnValue, 1);
}
The interesting part is that we don’t even need any SSL/TLS certificates for decryption or encryption!
This allows us to observe the application protocol both before encryption and after decryption, and render the data onto our platform without compromising security. Here’s an example for HTTP/S traffic:
Anteon platform demo is available on demo.getanteon.com. Give it a look.
The code snippets follow patterns of our Alaz eBPF Agent. The complete source code is available in our GitHub repository:
To be honest, there’s quite a bit of code surrounding the described functionality, primarily focused on extracting the buffer and conducting other protocol-related checks. Presenting Alaz in its entirety might be slightly complex for now. However, to provide you with a tangible example, we’ve prepared a focused demo code that incorporates only the features relevant to TLS/SSL and HTTP traffic. You can access it at the following link:
blog_examples/012_ssl_observability at main · getanteon/blog_examples
Performance Evaluation
One of the first concerns when deploying a program that intercepts and parses every message sent over your network is how much latency does it infer as well how much load is that on the host running the program. So we ran a couple of tests for you. The tests involved measuring the average latency over 1000 requests for each HTTP method type.
Our results indicate that the eBPF program adds a constant eBPF overhead of approximately 0.2µs on average — yes, remarkably low. Additionally, the average CPU load introduced by each hook is as follows: 0.1% for uprobe/SSL_write*, 0.007% for uprobe/SSL_read*, and 0.3% for uretprobe/SSL_read.
You can find the load testing programs in the /perf directory of the repository referenced below.
These findings address the trade-off between the added latency and CPU load due to eBPF instrumentation and the benefits of SSL/TLS encrypted traffic observation and analysis.
Conclusion
By leveraging eBPF, our Alaz eBPF Agent can dynamically intercept and observe encrypted traffic, providing deep insights into application performance and traffic patterns. The remarkably low overhead and minimal CPU load make this approach ideal for high-confidentiality applications running in complex environments like Kubernetes. We invite you to explore our demo and GitHub repository to see the full potential of eBPF in action.
This post was originally published on the Anteon Blog.
What Insights Can eBPF Provide into Real-Time SSL/TLS Encrypted Traffic and How? was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by TJ. Podobnik, @dorkamotorka
TJ. Podobnik, @dorkamotorka | Sciencx (2024-08-09T12:57:29+00:00) What Insights Can eBPF Provide into Real-Time SSL/TLS Encrypted Traffic and How?. Retrieved from https://www.scien.cx/2024/08/09/what-insights-can-ebpf-provide-into-real-time-ssl-tls-encrypted-traffic-and-how/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.