This content originally appeared on Level Up Coding - Medium and was authored by Rushabh Gaherwar
In this article, we will discuss the architecture and steps we need to follow to analyse and monitor logs on Datadog.
Amazon Web Services (AWS) is the world’s most comprehensive and broadly adopted cloud platform, offering over 200 fully-featured services from data centres globally. CloudFront is one of the services provided by them. CloudFront is a content delivery network that helps organisations/customers to deliver their content to their consumers.
On the other hand, Datadog is a monitoring and analytics tool for information technology (IT) and DevOps teams that can be used to determine performance metrics as well as event monitoring for infrastructure and cloud services.
Now, one would be wondering why do they need Datadog when this can be achieved through Cloudwatch?
Yes, you are correct. “Monitor AWS resources” is the primary reason why developers consider Amazon CloudWatch over the competitors, whereas “Monitoring for many apps (databases, web servers, etc)” was stated as the key factor in picking Datadog. If you are using or want to use Datadog for monitoring your various apps (databases, web servers, etc), you would like to utilise the amazing functionalities of Datadog for monitoring your AWS logs as well.
Architecture
Below is the architecture designed to serve our purpose.
Before going further, let us understand the services used in this architecture.
AWS CloudFront
Amazon CloudFront is a content delivery network operated by Amazon Web Services. Content delivery networks provide a globally distributed network of proxy servers that cache content, such as web videos or other bulky media, more locally to consumers, thus improving access speed for downloading the content.
For CloudFront Standard log file fields refer here.
AWS Kinesis Data Stream
Amazon Kinesis Data Streams (KDS) is a massively scalable and durable real-time data streaming service. The data collected is available in milliseconds to enable real-time analytics use cases such as real-time dashboards, real-time anomaly detection, dynamic pricing, and more.
AWS Kinesis Firehose
Amazon Kinesis Data Firehose is a fully managed service that automatically provisions, manages and scales compute, memory, and network resources required to process and load your streaming data. Once set up, Kinesis Data Firehose loads data streams into your destinations continuously as they arrive.
AWS S3 bucket
An Amazon S3 bucket is a public cloud storage resource available in Amazon Web Services (AWS) Simple Storage Service (S3), an object storage offering. Amazon S3 buckets, which are similar to file folders, store objects, which consist of data and its descriptive metadata.
Datadog
Datadog is a monitoring and analytics tool for information technology (IT) and DevOps teams that can be used to determine performance metrics as well as event monitoring for infrastructure and cloud services.
We are setting up a Kinesis data stream to stream the CloudFront logs and then using Kinesis firehose to load the stream to the DataDog. If there are any failures while emitting logs to Datadog, they can be stored in the S3 bucket. On the Datadog side, we need to parse those logs to extract useful fields for us and creating custom metrics over them. These custom metrics then would be used to create time-series graphs on a dashboard, creating monitors and alerts.
Below are the detailed steps to follow to set up the above architecture.
Process steps
Pushing CloudFront logs to Datadog
- Identify the Cloudfront distribution whose traffic we want to monitor.
- Configure CloudFront to send real-time logs:-
A CloudFront real-time log configuration specifies the source and destination of your logs, as well as the fields they contain. To create your log configuration, provide a name for the configuration and specify its log sampling rate — the percentage of logs generated by CloudFront that you want to send to Kinesis. Next, select the fields to include in your logs. By default, your log configuration will include all of the available CloudWatch log fields. - Create Kinesis stream:-
Designate a Kinesis Data Stream as the endpoint to which CloudFront will send your logs. If you already have a stream you want to use, enter its Amazon Resource Name (ARN) in the Endpoint field. Or to create a new stream. - Navigate back to the Create real-time log configuration page and paste the ARN into the Endpoint field.
- Create a Datadog API key, that will be needed when we are creating AWS Kinesis firehose.
- Route logs through Kinesis Data Firehose to Datadog:-
To route your logs into Datadog, create a Kinesis Data Firehose delivery stream and choose the Kinesis Data Stream you created above as the source. Next, specify Datadog as your delivery stream’s destination. Finally, enter your Datadog API key, select the appropriate HTTP endpoint URL, and provide the required additional configuration details. See the AWS documentation to learn more about how to configure Datadog as a Kinesis Data Firehose destination. - Make sure you add key-value parameters in the AWS kinesis firehose, this will help in finding the logs on Datadog easily.
Once you are done with the above steps, you can see the logs are emitted to DataDog and can be seen on Live Tail using by searching for the parameter created in the above steps.
Next steps on Datadog
** Indexing the whole logs are not cost-efficient, rather we should go with the logging without limits approach. Creating custom metrics will save a lot of cost for us.
Parsing and creating custom metrics from the CloudFront logs on the Datadog
- Create a pipeline in Datadog to process the ingested logs.
- Add Grok parser to parse the logs and extract the useful fields from the logs.
- Add URL parser, category process, remappers in the pipeline to modify the extracted fields above or add new fields out of it.
- Create a custom metric using the parameter(Added in Kinesis firehose) as a filter and with the dimensions as per the requirement/use case.
Now we are ready to create our dashboard, monitors and alerts using the custom metrics we created above.
Here are some reference links to achieve the same:- Dashboard & Monitor
I hope you liked this article and helped you to achieve your task.
If you find this article useful, Grap me a coffee:- buymeacoffee.com/rushabh
An approach to analyse and monitor AWS CloudFront logs on Datadog was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Rushabh Gaherwar
Rushabh Gaherwar | Sciencx (2021-07-05T14:50:46+00:00) An approach to analyse and monitor AWS CloudFront logs on Datadog. Retrieved from https://www.scien.cx/2021/07/05/an-approach-to-analyse-and-monitor-aws-cloudfront-logs-on-datadog/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.