Production Metrics in Node.js: Instrumenting Custom Prometheus Metrics and Grafana Dashboards

When your Node.js application hits production, console logs and basic process monitoring (like CPU and memory usage) are no longer enough. You need deep visibility into application performance: request latencies, active database connections, custom business logic error rates, and throughput.

In this guide, we will walk through setting up production-grade observability in a Node.js application using Prometheus for metrics collection and Grafana for visualization. We will use the industry-standard prom-client library to instrument custom metrics, expose them securely, and build a powerful Grafana dashboard.

Architecture Overview

Before diving into the code, let’s look at the telemetry pipeline:

1. Node.js Application: Uses prom-client to track system and custom metrics.
2. Exposed Endpoint: The application hosts an HTTP endpoint (GET /metrics) in the format Prometheus expects.
3. Prometheus Server: Periodically scrapes the Node.js /metrics endpoint and stores the time-series data.
4. Grafana: Connects to Prometheus as a data source to query and visualize the metrics on customizable dashboards.

Step 1: Setting Up the Node.js Project

Let’s initialize a modern Node.js project using Express. We will install the required dependencies: express for our web server and prom-client for metric instrumentation.

mkdir node-prometheus-demo
cd node-prometheus-demo
npm init -y
npm install express prom-client

Create a file named server.js. We will structure our code to separate our Express application from our metrics collection setup.

Step 2: Configuring Prometheus Client (prom-client)

Prometheus relies on different metric types: Counters (only go up, like total requests), Gauges (go up and down, like active connections), and Histograms (sample observations and count them in configurable buckets, like request latency).

Let’s create a dedicated module to manage our metrics registry.

// metrics.js
const client = require('prom-client');

// 1. Create a Registry to store metrics
const register = new client.Registry();

// 2. Add default Node.js runtime metrics (CPU, memory, event loop lag, GC stats)
client.collectDefaultMetrics({
  register,
  prefix: 'node_app_',
});

// 3. Define Custom Metrics

// HTTP Request Duration Histogram
const httpRequestDurationMicroseconds = new client.Histogram({
  name: 'node_app_http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  // Custom buckets tailored for API response times (in seconds)
  buckets: [0.01, 0.05, 0.1, 0.3, 0.5, 1, 2, 5],
  registers: [register],
});

// Active Connections Gauge
const activeConnections = new client.Gauge({
  name: 'node_app_active_connections',
  help: 'Number of active connections currently being handled',
  registers: [register],
});

// Business Logic Counter (e.g., failed orders or payments)
const businessErrorsTotal = new client.Counter({
  name: 'node_app_business_errors_total',
  help: 'Total count of business logic errors',
  labelNames: ['error_type'],
  registers: [register],
});

module.exports = {
  register,
  httpRequestDurationMicroseconds,
  activeConnections,
  businessErrorsTotal,
};

Step 3: Integrating Metrics into the Express Application

Next, we need to wire up middleware to automatically time incoming HTTP requests and record status codes, while also exposing the /metrics scrapable endpoint.

// server.js
const express = require('express');
const {
  register,
  httpRequestDurationMicroseconds,
  activeConnections,
  businessErrorsTotal,
} = require('./metrics');

const app = express();
const PORT = process.env.PORT || 3000;

app.use(express.json());

// Middleware: Track active connections and request duration
app.use((req, res, next) => {
  activeConnections.inc();
  const startTimer = httpRequestDurationMicroseconds.startTimer();

  // Ensure we decrement active connections and record duration once response finishes
  res.on('finish', () => {
    activeConnections.dec();
    startTimer({
      method: req.method,
      // Use req.path or a normalized route pattern to prevent high cardinality
      route: req.route ? req.route.path : req.path,
      status_code: res.statusCode,
    });
  });

  next();
});

// Sample route: Healthy API response
app.get('/api/users', (req, res) => {
  // Simulate random processing time
  setTimeout(() => {
    res.json({ users: [{ id: 1, name: 'Alice' }, { id: 2, name: 'Bob' }] });
  }, Math.random() * 200);
});

// Sample route: Simulating a business logic error
app.post('/api/checkout', (req, res) => {
  const success = Math.random() > 0.3;
  if (!success) {
    businessErrorsTotal.inc({ error_type: 'payment_declined' });
    return res.status(400).json({ error: 'Payment declined by gateway' });
  }
  res.status(200).json({ status: 'success' });
});

// The Prometheus Scrape Endpoint
app.get('/metrics', async (req, res) => {
  try {
    res.setHeader('ContentType', register.contentType);
    res.send(await register.metrics());
  } catch (ex) {
    res.status(500).end(ex);
  });
});

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`);
});

Best Practice Alert: Avoiding Metric Cardinality Explosions

Notice how we used req.route ? req.route.path : req.path for our route label. Never pass raw user inputs, dynamic UUIDs, or un-normalized database IDs directly into metric labels. Doing so creates unique time-series for every single request, which will quickly exhaust Prometheus memory and crash your observability backend.

Step 4: Configuring Prometheus

To make Prometheus scrape your Node.js application, configure your prometheus.yml file to include a scrape job targeting your server.

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'nodejs-app'
    static_configs:
      - targets: ['localhost:3000'] # Point this to your Node.js service

Start your Prometheus instance using Docker or a local binary pointing to this configuration file:

docker run -d \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Navigate to http://localhost:9090/targets to verify that your Node.js application is being successfully scraped.

Step 5: Visualizing Metrics in Grafana

Now that Prometheus is ingesting your metrics, you can spin up Grafana and configure Prometheus as a data source.

1. Run Grafana locally via Docker:

   docker run -d -p 3001:3000 --name=grafana grafana/grafana
   

2. Open http://localhost:3001 (default credentials: admin / admin).
3. Navigate to Connections > Data Sources > Add data source and select Prometheus.
4. Set the connection URL to your Prometheus instance (e.g., http://host.docker.internal:9090 if running both in Docker on macOS/Windows) and click Save & test.

Building a Production Dashboard

Create a new dashboard and add the following Prometheus PromQL queries to your panels:

1. Request Throughput (Requests Per Second)

Calculates the per-second rate of HTTP requests over a 1-minute window, broken down by route and status code.

sum(rate(node_app_http_request_duration_seconds_count[1m])) by (route, status_code)

2. P95 Request Latency

Measures the 95th percentile response latency across your application routes.

histogram_quantile(0.95, sum(rate(node_app_http_request_duration_seconds_bucket[1m])) by (le, route))

3. Active Connections

Monitors real-time concurrent requests being processed.

node_app_active_connections

4. Business Error Rate

Tracks business exceptions over time.

sum(rate(node_app_business_errors_total[1m])) by (error_type)

Conclusion

By instrumenting your Node.js application with prom-client, exposing a standard /metrics endpoint, and pairing it with Prometheus and Grafana, you unlock deep, production-grade observability. You are no longer guessing how your system behaves under load—you have hard data regarding CPU bottlenecks, memory leaks, latency spikes, and business logic exceptions.

Key Takeaways

• Use default system metrics to spot memory leaks and event loop degradation early.
• Wrap HTTP workflows in histograms to track latency distributions accurately without high cardinality issues.
• Use PromQL aggregations like rate() and histogram_quantile() to extract actionable insights inside Grafana dashboards.