Custom Monitor Capture Machine Metrics using Prometheus and Node Exporter

Introduction

Prometheus is an open-source monitoring tool, developed and maintained by SoundCloud. Prometheus scrapes and stores its metrics in a time series database. To query the data stored in the database, it supports a query language called PromQL. Node Exporter is a library able to scrape all the system-level metrics and make those metrics accessible over HTTP. By default, the metrics can be accessed over port 9100.

With the help of Node Exporter, we can get the machine metrics. Later we can configure Prometheus to scrape these values and store them in its time-series database. Once the values are stored, we can use Custom Monitor to query Prometheus using PromQL and fetch the required information.

Prerequisites

A Linux System.
NodeJS installed on your Linux System.

Installation and Configuration

Installing exporter

For this guide, we will use Node Exporter to capture machine metrics.

Navigate to the opt folder.
$ cd /opt
Create a folder to store the Prometheus and Node Exporter files.
$ mkdir PrometheusFiles
Navigate into the new folder.
$ cd PrometheusFiles
Download the latest version of node exporter using the wget command.
$ wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz
Unzip the file in the same folder.
$ tar -xvzf node_exporter-0.17.0.linux-amd64.tar.gz
Delete the .gz file after unzip.
$ rm node_exporter-0.17.0.linux-amd64.tar.gz

Running Node Exporter as a Service

This will ensure that we can easily start and stop Node Exporter.

Create a file for service configuration.
$ sudo vi /etc/systemd/system/node_exporter.service
Save the file with the content below, where we define the user and the path pointing to Node Exporter.

[Unit]
Description=Node Exporter
 
[Service]
User=root
ExecStart=/opt/PrometheusFiles/node_exporter-0.17.0.linux-amd64/node_exporter

[Install]
WantedBy=default.target

Reload systemd so that it reads the new configuration file you created.
$ sudo systemctl daemon-reload
Now that Node Exporter is available as a service, enable it so that it starts automatically at boot time.
$ sudo systemctl enable node_exporter.service
Start the service manually.
$ sudo systemctl start node_exporter.service
Test that the Node Exporter service is up and running.
$ curl http://localhost:9100/metrics

Installing Prometheus

Prometheus will scrape all the metrics exposed by Node Exporter.

Go to the folder where all files for this project are saved.
$ cd /opt/PrometheusFiles/
Download the latest version of Prometheus using the wget command..
$ wget https://github.com/prometheus/prometheus/releases/download/v2.7.1/prometheus-2.7.1.linux-amd64.tar.gz
Unzip the .gz file in the same folder.
$ tar -xvzf prometheus-2.7.1.linux-amd64.tar.gz
Verify that Prometheus is installed correctly.
$ prometheus-2.7.1.linux-amd64/prometheus --version
Delete the .gz file after unzip.
$ rm prometheus-2.7.1.linux-amd64.tar.gz

4. Running Prometheus as a service

This will ensure that we can easily start and stop Prometheus.

Create a file for service configuration.
$ sudo vi /etc/systemd/system/prometheus.service
Save the file with the content below, where we define the user and the path pointing to Prometheus.

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target[Service]
User=root
Group=prometheus
Type=simple
ExecStart=/opt/PrometheusFiles/prometheus-2.7.1.linux-amd64/prometheus \
    --config.file /opt/PrometheusFiles/prometheus-2.7.1.linux-amd64/prometheus.yml \
    --storage.tsdb.path /opt/PrometheusFiles/prometheus-2.7.1.linux-amd64/ \
    --web.console.templates=/opt/PrometheusFiles/prometheus-2.7.1.linux-amd64/consoles \
    --web.console.libraries=/opt/PrometheusFiles/prometheus-2.7.1.linux-amd64/console_libraries

[Install]
WantedBy=multi-user.target

Reload systemd so that it reads the new configuration file you created.
$ sudo systemctl daemon-reload
Now that Prometheus is available as a service, enable it so that it starts automatically at boot time.
$ sudo systemctl start prometheus.service
Start the service manually.
$ sudo systemctl enable prometheus.service
Test that Prometheus service is up and running.
$ systemctl status prometheus

Implementation

Write the Custom Script

Navigate to /opt/3genlabs/hawk/syntheticnode/service/shellmonitor/sandbox to create a shell script.
Create a new file to save the NodeJS code.
$ vi prom_node_exporter.js
Paste the code below and save it.

#!/usr/bin/env node
var request  = require('request');

//End and Start Epochtime
var Et = Math.round((new Date()).getTime() / 1000);
var St=Et-10;
var query= [      process.env.CP_UNSAFE_VAR_used,
                  process.env.CP_UNSAFE_VAR_free,
                  process.env.CP_UNSAFE_VAR_cached,
                  process.env.CP_UNSAFE_VAR_buffer,
                  process.env.CP_UNSAFE_VAR_CPU
                ]
for(var i=0;i<query.length;i++){
        var q_encode= encodeURIComponent(query[i]);

        //Create object
        var getMetric = {
                uri: 'http://localhost:9090/api/v1/query?query='+q_encode+'&time='+Et+'&_='+St,
                method: 'GET',
                headers: {
                'Content-Type': 'application/json',
                'Accept': 'application/json',
                        }
                };

        request(getMetric, function(error, response, body) {

                if(error){
                        console.log(error)
                                }
                else{
                        console.log(body);
                }
        });
}

Note: You can modify the above code as per your requirement.

Install the NodeJs request library.
$ npm install request
Run these commands to change the permission and owner of the shell script:
$ chmod 500 prom_node_exporter.js
$ chown serveruser prom_node_exporter.js

Create the Custom Test

Create a Custom Test within the Portal using the JSON template below. Make sure that you replace the necessary values.

{ 
   "shell_command_file_name" : "prom_node_exporter.js",
   "used" : "node_memory_MemTotal_bytes-(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes)",
   "free" : "node_memory_MemFree_bytes",
   "cached" : "node_memory_Cached_bytes",
   "buffer" : "node_memory_Buffers_bytes",
   "CPU" : "sum by (mode)(irate(node_cpu_seconds_total{job='node',instance='localhost:9100'}[1m])) * 100 / scalar(count(count by (cpu)(node_cpu_seconds_total{job='node',instance='localhost:9100'})))"
}

Results

Based on the output in the console, you can create Tracepoints and Indicators to capture the metrics and analyze those metrics in the Portal.

This is an example used for monitoring the CPU and memory usage of a particular machine. Based on the queries in custom test script results will be fetched from Node Exporter through Prometheus.

Here are the results of the test:

Now we can set up Insight to capture and analyze the results.

Regex for the same:

Free Memory (bytes): node_memory_MemFree_bytes.*\"([\d]+)\"\]
Used Memory (bytes): metric\"\:{"instance.*value.*\,\"([\d]+)\"\]
CPU_idle (%): idle\"\}\,\"value\"\:\[\d+,\"([\d.\d]+)
CPU_system (%): system\"\}\,\"value\"\:\[\d+,\"([\d.\d]+)