How To Monitor Your Node.js Apps With Prometheus"

In the age of large monolithic apps and app servers, it was not too difficult to monitor the status of your application as well as some more detailed monitoring of transactions. In today’s age of microservices, that whole monitoring thing is getting more and more complex. You not only have to deal with a bunch of more servers, you also have to deal with more (micro)services. You have different options depending on what you want to monitor. For example, you can use Nagios, Zabbix or Prometheus. My personal preference goes out to Greek deity that stole fire from Mount Olympus and brought it to us.

Deity? Prometheus

Copied from the Prometheus website, they describe it as:

(…) an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community.

As SoundCloud moved towards microservices they needed to monitor thousands of running services and as their current monitoring systems had too many limitations they invented Prometheus. The Prometheus website goes on to say that:

It is now a standalone open source project and maintained independently of any company. To emphasize this and clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project after Kubernetes.

Prometheus architecture

Prometheus was designed with 4 main requirements in mind:

• A multi-dimensional data model, so that data can be sliced and diced at will, along dimensions like instance, service, endpoint, and method.
• Operational simplicity, so that you can spin up a monitoring server where and when you want, even on your local workstation, without setting up a distributed storage backend or reconfiguring the world.
• Scalable data collection and decentralized architecture, so that you can reliably monitor the many instances of your services, and independent teams can set up independent monitoring servers.
• Finally, a powerful query language that leverages the data model for meaningful alerting (including easy silencing) and graphing (for dashboards and for ad-hoc exploration).

Image from Prometheus documentation

As there are many, many good explanations on what the different components of Prometheus do, I’ll leave that to those websites.

Building a Node.js app

Before we continue with Prometheus we do need to have a service running somewhere to monitor. As I enjoy building Node.js code, I’ll walk you through what you need to add to your Node.js apps to start monitoring them with Prometheus. I do assume that you’ve created an API spec in TIBCO Cloud Integration and exported the Node.js code from there. If not, you can get started with the tutorial on the TIBCO Community and come back later (or simply download the code from my GitHub repo ;-)).

Adding dependencies to package.json

To start monitoring the Node.js engine we need a prometheus client for node.js that supports histogram, summaries, gauges and counters. Prometheus themselves recommend prom-client as the best option, which you can install with

npm install --save prom-client

Not only do we want to monitor the health of the app, we also want to see the response times for the various actions. So we need a middleware that records the response time for requests in HTTP servers.

npm install --save response-time

These two dependencies will make sure we can monitor the Node.js engine as well as collect the response times.

Creating a module

Node.js has a simple module loading system. In Node.js, files and modules are in one-to-one correspondence (each file is treated as a separate module). To follow the best practices of Node.js, and make our code as reusable and readable as possible, we’ll create a new module and use that in our main code later on. It is a lot of JavaScript code, but I’ve really done my best to document it.

/**
 * Newly added requires
 */
var Register = require('prom-client').register;
var Counter = require('prom-client').Counter;
var Histogram = require('prom-client').Histogram;
var Summary = require('prom-client').Summary;
var ResponseTime = require('response-time');
var Logger = require('./logger');

/**
 * A Prometheus counter that counts the invocations of the different HTTP verbs
 * e.g. a GET and a POST call will be counted as 2 different calls
 */
module.exports.numOfRequests = numOfRequests = new Counter({
    name: 'numOfRequests',
    help: 'Number of requests made',
    labelNames: ['method']
});

/**
 * A Prometheus counter that counts the invocations with different paths
 * e.g. /foo and /bar will be counted as 2 different paths
 */
module.exports.pathsTaken = pathsTaken = new Counter({
    name: 'pathsTaken',
    help: 'Paths taken in the app',
    labelNames: ['path']
});

/**
 * A Prometheus summary to record the HTTP method, path, response code and response time
 */
module.exports.responses = responses = new Summary({
    name: 'responses',
    help: 'Response time in millis',
    labelNames: ['method', 'path', 'status']
});

/**
 * This funtion will start the collection of metrics and should be called from within in the main js file
 */
module.exports.startCollection = function () {
    Logger.log(Logger.LOG_INFO, `Starting the collection of metrics, the metrics are available on /metrics`);
    require('prom-client').collectDefaultMetrics();
};

/**
 * This function increments the counters that are executed on the request side of an invocation
 * Currently it increments the counters for numOfPaths and pathsTaken
 */
module.exports.requestCounters = function (req, res, next) {
    if (req.path != '/metrics') {
        numOfRequests.inc({ method: req.method });
        pathsTaken.inc({ path: req.path });
    }
    next();
}

/**
 * This function increments the counters that are executed on the response side of an invocation
 * Currently it updates the responses summary
 */
module.exports.responseCounters = ResponseTime(function (req, res, time) {
    if(req.url != '/metrics') {
        responses.labels(req.method, req.url, res.statusCode).observe(time);
    }
})

/**
 * In order to have Prometheus get the data from this app a specific URL is registered
 */
module.exports.injectMetricsRoute = function (App) {
    App.get('/metrics', (req, res) => {
        res.set('Content-Type', Register.contentType);
        res.end(Register.metrics());
    });
};

Adding code to server.js

Now from the server.js file you only need a few lines of JavaScript code

'use strict';

...
/**
 * This creates the module that we created in the step before.
 * In my case it is stored in the util folder.
 */
var Prometheus = require('./util/prometheus');
...

/**
 * The below arguments start the counter functions
 */
App.use(Prometheus.requestCounters);
App.use(Prometheus.responseCounters);

/**
 * Enable metrics endpoint
 */
Prometheus.injectMetricsRoute(App);

/**
 * Enable collection of default metrics
 */
Prometheus.startCollection();
...

By only adding 5 lines of code in the server.js file you can effectively monitor your app!

Deloy

The last part is pushing the app to TIBCO Cloud Integration, using either the tibcli utility or the Visual Studio Code extension. You can find more details in the TCI docs or in this post on the TIBCO Community. After you’re done, you need to get the URL of your app so we can add it to Prometheus. For example a URL could be https://integration.cloud.tibcoapps.com/ijdc72jg2ugg2dikkkl236f2rhma6qaz.

Running Prometheus

There are a whole bunch of ways to run Prometheus, but I think one of the easiest is to run it inside a Docker container. In that case you don’t have to worry about installation, dependencies and all that other hassle a container solves for you. We do need to tell Prometheus which endpoint to monitor and to do that you need to create a prometheus.yml and bind-mount it from the host.

Prometheus can only monitor targets with a hostname and a port, so it makes it a little more difficult to monitor apps running on an iPaaS or PaaS platform. Luckily for each job you can add a parameter called metrics_path that instructs Prometheus to go to a specific path on that server.

A very basic, but correct and useful, prometheus.yml can look like this.

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
rule_files:
- /etc/prometheus/alert.rules
scrape_configs:
- job_name: PrometheusApp
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /ijdc72jg2ugg2dikkkl236f2rhma6qaz/metrics
  scheme: https
  static_configs:
  - targets:
    - integration.cloud.tibcoapps.com
    labels:
      app: PrometheusApp
      sandbox: MyDefaultSandbox

I’ve set the job_name and the label app to the name of my app in TCI so I can easily correlate between them, but you can make your own decision there. The metrics_path contains part of the URL from your app and added the part /metrics for Prometheus to access the data.

If you’ve created your file you can run a docker container with the below command and it will start to monitor your app on TIBCO Cloud Integration.

docker run -p 9090:9090 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Looking at Prometheus data

Let’s open up Prometheus http://localhost:9090/graph and enter the expression up in the white box.

That expression will check whether your app is running or not. An expression like sum(numOfRequests) will get you a number of all requests processes by your app.

Without the sum() it will give you an overview of the individual HTTP verbs

Reporting with Grafana

Grafana is the open platform for beautiful analytics and monitoring, which is capable of taking the data from Prometheus and showing it in a nice and easy way. With Grafana you can very easily make dashboards to show the status of the apps like:

A sample dashboard that shows the status, the number of requests and what kind of requests were served

Conclusion

With a combination of Prometheus you can not only monitor the health and status of not only the Node.js apps you run in TIBCO Cloud Integration (or anywhere else), but also other types of apps that can export (or push) their data to Prometheus. With Grafana you can create custom dashboards that take the data from Prometheus and display it in a way that you want!