Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!)

This content originally appeared on DEV Community and was authored by Arbythecoder

Introduction

Today, we will look deep into the world of advanced logging and monitoring, but with a twist—we're doing it all on our phones! This is Day 14 of my Devops project, and we're going mobile with Termux( I don't have access to my windows machine because I will be on transit all day but I thought it would be fun to see what I can do using termux) to build a powerful monitoring system using Prometheus, Grafana, and Alertmanager. This is a great way to learn and practice these tools, even when you're on the go.

Objectives

Create a Node.js application that generates metrics and logs.
Dockerize the application for easy deployment.
Configure Prometheus to collect metrics and trigger alerts.
Set up Alertmanager to send email to notify you.
Use Grafana to visualize metrics and create dashboards.
Organize the project for easy sharing on GitHub.

A More detailed Guide

1. Setting up the Node.js Application (Termux)

Create the Project Directory:

 mkdir Day14-monitoring-app
 cd Day14-monitoring-app

Create the app Directory:
```
 mkdir app
 cd app
```
Create the index.js File:
```
 touch index.js
```
Open index.js in Termux's Nano editor:
```
 nano index.js
```

Paste the following code into index.js:

 // Import necessary modules
 const express = require('express');
 const app = express();
 const port = 4000;
 const client = require('prom-client');
 const morgan = require('morgan');
 const collectDefaultMetrics = client.collectDefaultMetrics;

 // Collect default metrics (CPU, memory, etc.)
 collectDefaultMetrics();

 // Middleware for logging requests
 app.use(morgan('combined')); // Logs HTTP requests

 // Define routes for our application
 app.get('/', (req, res) => res.send('Hello World!'));
 app.get('/api/data', (req, res) => res.json({ data: 'some data' }));

 // Endpoint to expose Prometheus metrics
 app.get('/metrics', (req, res) => {
   res.set('Content-Type', client.register.contentType);
   res.end(client.register.metrics());
 });

 // Start the server
 app.listen(port, () => console.log(`App listening at http://localhost:${port}`));

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.
Create the package.json File:
```
 touch package.json
```
Open package.json in Nano:
```
 nano package.json
```

Paste the following code into package.json:

 {
   "name": "advanced-monitoring-app",
   "version": "1.0.0",
   "description": "A Node.js application with advanced monitoring",
   "main": "index.js",
   "scripts": {
     "start": "node index.js"
   },
   "author": "Your Name",
   "license": "MIT",
   "dependencies": {
     "express": "^4.18.2",
     "morgan": "^1.10.0",
     "prom-client": "^16.0.0"
   }
 }

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.
Install Dependencies:
```
 npm install
```

2. Dockerizing the Application (Termux)

Create the Dockerfile:
```
 touch Dockerfile
```
Open Dockerfile in Nano:
```
 nano Dockerfile
```

Paste the following code into Dockerfile:

 # Use the Node.js 14 image as our base
 FROM node:14

 # Set the working directory inside the container
 WORKDIR /app

 # Copy the package.json file
 COPY package*.json ./

 # Install dependencies using npm
 RUN npm install

 # Copy the rest of the application code
 COPY . .

 # Expose port 4000 for our application to listen on
 EXPOSE 4000

 # Run the Node.js application when the container starts
 CMD ["node", "index.js"]

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

Build the Docker Image:

 docker build -t your_dockerhub_username/advanced-monitoring-app:latest .

Log in to Docker Hub:
```
 docker login
```

Push the Image to Docker Hub:

 docker push your_dockerhub_username/advanced-monitoring-app:latest

3. Configuring Prometheus (Termux)

Create the prometheus.yml File:
```
 touch prometheus.yml
```
Open prometheus.yml in Nano:
```
 nano prometheus.yml
```

Paste the following code into prometheus.yml:

 global:
   scrape_interval: 15s # Scrape metrics every 15 seconds

 scrape_configs:
   - job_name: 'node-app' # Name of the job for scraping our app
     static_configs:
       - targets: ['host.docker.internal:4000'] # Target to scrape (our app)
 rule_files:
   - 'alert.rules.yml'

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

4. Configuring Alertmanager (Termux)

Create the alertmanager.yml File:
```
 touch alertmanager.yml
```
Open alertmanager.yml in Nano:
```
 nano alertmanager.yml
```

Paste the following code into alertmanager.yml:

 global:
   resolve_timeout: 5m # Alerts are considered resolved after 5 minutes of no new alerts

 route:
   group_by: ['job'] # Group alerts by job name
   group_wait: 30s # Wait 30 seconds to group alerts
   group_interval: 5m # Group alerts every 5 minutes
   repeat_interval: 12h # Repeat alerts every 12 hours
   receiver: 'email' # Send alerts to the email receiver

 receivers:
   - name: 'email'
     email_configs:
       - to: 'your-email@example.com' # Your email address
         from: 'alertmanager@example.com' # Sender email
         smarthost: 'smtp.example.com:587' # SMTP server (replace with your settings)
         auth_username: 'alertmanager' # SMTP username
         auth_password: 'password' # SMTP password

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

5. Docker Compose Setup (Termux)

Create the docker-compose.yml File:
```
 touch docker-compose.yml
```
Open docker-compose.yml in Nano:
```
 nano docker-compose.yml
```

Paste the following code into docker-compose.yml:

 version: '3'

 services:
   prometheus:
     image: prom/prometheus
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml # Mount the prometheus.yml file
     ports:
       - "9090:9090" # Expose port 9090 for Prometheus

   grafana:
     image: grafana/grafana
     ports:
       - "3000:3000" # Expose port 3000 for Grafana

   alertmanager:
     image: prom/alertmanager
     volumes:
       - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml # Mount the alertmanager.yml file
     ports:
       - "9093:9093" # Expose port 9093 for Alertmanager

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

6. Adding Custom Alerts (Termux)

Create the alert.rules.yml File:
```
 touch alert.rules.yml
```
Open alert.rules.yml in Nano:
```
 nano alert.rules.yml
```

Paste the following code into alert.rules.yml:

 groups:
   - name: example
     rules:
       - alert: HighRequestLatency
         expr: http_request_duration_seconds_mean{job="node-app"} > 0.5 # Check if average request duration is over 0.5 seconds
         for: 2m # Trigger alert only if the condition is true for 2 minutes
         labels:
           severity: "page" # Add a label to the alert
         annotations:
           summary: "High request latency" # Alert summary
           description: "The request latency is above 0.5s for more than 2 minutes." # Detailed description

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

7. Deploying the Stack (Termux)

   # Start the Docker Compose stack in detached mode
   docker-compose up -d

8. Configuring Grafana Dashboards (Termux)

Access Grafana: Open your web browser and go to http://localhost:3000.
Add Prometheus as a data source: In Grafana, add Prometheus as a data source.
Create custom dashboards: Create dashboards to visualize your metrics.
Set up alerts in Grafana: Set up alerts in Grafana to trigger notifications via Alertmanager.

9. GitHub Actions Workflow (Termux)

Create the .github/workflows Directory:

 mkdir .github/workflows
 cd .github/workflows

Create the main.yml File:
```
 touch main.yml
```
Open main.yml in Nano:
```
 nano main.yml
```

Paste the following code into main.yml:

 name: Docker Build & Deploy

 on:
   push:
     branches:
       - main

 jobs:
   build-and-deploy:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout Repository
         uses: actions/checkout@v2

       - name: Login to Docker Hub
         uses: docker/login-action@v2
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}

       - name: Build and Push Docker Image
         run: |
           docker build -t ${{ secrets.DOCKER_USERNAME }}/advanced-monitoring-app:latest .
           docker push ${{ secrets.DOCKER_USERNAME }}/advanced-monitoring-app:latest

       - name: Deploy Docker Container
         run: |
           docker-compose up -d

Save and Exit Nano: Ctrl+O to save, then Ctrl+X to exit.

10. Setting Up Your GitHub Repository

Create a new repository on GitHub.
Initialize a Git repository in your project directory:
```
 git init
```
Add all files to the repository:
```
 git add .
```
Commit your changes:
```
 git commit -m "Initial commit"
```

Add the GitHub remote:

 git remote add origin <your_github_repo_url>

Push your changes to GitHub:
```
 git push origin main
```

Remember:

Replace your_dockerhub_username with your actual Docker Hub username.
Replace your-email@example.com with your email address.
Replace smtp.example.com:587 with your SMTP server address and port.
Set the DOCKER_USERNAME and DOCKER_PASSWORD secrets in your GitHub repository settings.

Conclusion

With this setup , you can indeed monitor your application and receive alerts directly on your phone using Termux. Here's how it works:

Data Collection: The Node.js application you've created acts as a source of data. It collects metrics about HTTP requests, system resources (CPU, memory), and network connections.
Prometheus: Prometheus is the data collector and storage engine. It regularly scrapes metrics from your Node.js application and stores them.
Alertmanager: Alertmanager is the notification system. You've configured it to send email alerts when certain conditions are met. For example, if your Node.js application experiences high request latency or if your system's CPU load exceeds a threshold, Alertmanager will send you an email notification.
Grafana: Grafana is the visualization tool. It allows you to create dashboards to view and analyze the collected metrics. You can create graphs, charts, and other visualizations to gain insights into your application's performance.
Termux: Termux is your mobile terminal environment. You can use it to run all the necessary commands to build, deploy, and manage your monitoring stack. You can also use Termux to access Grafana's web interface on your phone's browser to view your dashboards and monitor your application.

How to Receive Alerts on Your Phone:

Email: The most straightforward way to receive alerts is through email. You've already configured Alertmanager to send emails to your specified address.
Push Notifications: You can explore integrations with push notification services like Pushover or Prowl. These services allow you to receive alerts directly on your phone as push notifications.

Key Points:

Termux is your mobile command center: Which can start and stop the Docker Compose stack, view logs, and manage your monitoring system.
Grafana is your visual dashboard: It gives you a clear and intuitive way to see your application's performance.
Alertmanager is your early warning system: It notifies you of potential problems so you can address them quickly.

-To receive emails on your phone, you'll need to ensure that your email client is configured .

If you want to use push notifications, you'll need to set up an account with a push notification service and integrate it with Alertmanager.

With this setup, you have a powerful and flexible monitoring system at your fingertips, even when you're on the go!

This content originally appeared on DEV Community and was authored by Arbythecoder

Print Share Comment Cite Upload Translate Updates

APA

Arbythecoder | Sciencx (2024-07-18T06:16:02+00:00) Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!). Retrieved from https://www.scien.cx/2024/07/18/day-14-advanced-logging-and-monitoring-with-prometheus-grafana-and-alertmanager-termux-edition/

MLA

" » Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!)." Arbythecoder | Sciencx - Thursday July 18, 2024, https://www.scien.cx/2024/07/18/day-14-advanced-logging-and-monitoring-with-prometheus-grafana-and-alertmanager-termux-edition/

HARVARD

Arbythecoder | Sciencx Thursday July 18, 2024 » Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!)., viewed ,<https://www.scien.cx/2024/07/18/day-14-advanced-logging-and-monitoring-with-prometheus-grafana-and-alertmanager-termux-edition/>

VANCOUVER

Arbythecoder | Sciencx - » Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!). [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/07/18/day-14-advanced-logging-and-monitoring-with-prometheus-grafana-and-alertmanager-termux-edition/

CHICAGO

" » Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!)." Arbythecoder | Sciencx - Accessed . https://www.scien.cx/2024/07/18/day-14-advanced-logging-and-monitoring-with-prometheus-grafana-and-alertmanager-termux-edition/

IEEE

" » Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!)." Arbythecoder | Sciencx [Online]. Available: https://www.scien.cx/2024/07/18/day-14-advanced-logging-and-monitoring-with-prometheus-grafana-and-alertmanager-termux-edition/. [Accessed: ]

rf:citation

» Day 14: Advanced Logging and Monitoring with Prometheus, Grafana, and Alertmanager (Termux Edition!) | Arbythecoder | Sciencx | https://www.scien.cx/2024/07/18/day-14-advanced-logging-and-monitoring-with-prometheus-grafana-and-alertmanager-termux-edition/ |

Please log in to upload a file.

There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts