Production Logging in Node.js: Structured Logging, Pino, and Log Aggregation with Loki

Every backend engineer eventually learns the hard way that console.log('User logged in', userId) is an anti-pattern in production. When you are scaling across multiple instances, containerized in Kubernetes, or trying to debug a race condition under heavy load, unstructured stdout strings become useless noise.

To build resilient, observable distributed systems in Node.js, you need three things:

1. Structured JSON Logging: Machine-readable logs that include context (timestamps, log levels, request IDs).
2. High Performance: A logging pipeline that doesn’t block the Node.js single-threaded event loop.
3. Centralized Aggregation: A system to collect, index, and query logs across your entire infrastructure (like Grafana Loki).

In this practical guide, we will implement a production-grade logging architecture using Pino in Node.js and ship those logs to Grafana Loki using Promtail.

Why Pino? The Need for Speed

Many Node.js developers reach for Winston out of habit. While Winston is feature-rich, it is also notoriously heavy. Pino, on the other hand, was built from the ground up for extreme performance. It serializes objects directly to JSON strings using internal optimizations, often running up to 5x faster than Winston while consuming a fraction of the CPU and memory.

In a single-threaded runtime like Node.js, a slow logger means a blocked event loop, degraded HTTP throughput, and increased latency.

Installing Dependencies

Let’s set up a clean Node.js project. Initialize your project and install Pino along with pino-http for automatic HTTP request logging:

npm init -y
npm install express pino pino-http
npm install -D pino-pretty

Note: pino-pretty is strictly for local development. In production, you should output raw JSON.

Setting Up Structured Logging

Let’s create a centralized logger module (logger.js) that configures Pino appropriately based on the environment.

// logger.js
const pino = require('pino');

const isProduction = process.env.NODE_ENV === 'production';

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  // In production, emit raw JSON. In development, use pino-pretty via transport.
  transport: !isProduction
    ? {
        target: 'pino-pretty',
        options: {
          colorize: true,
          translateTime: 'SYS:standard',
          ignore: 'pid,hostname',
        },
      }
    : undefined,
  // Standard base properties added to every log line
  base: {
    env: process.env.NODE_ENV,
    service: 'user-payment-service',
  },
  // Custom timestamp format for consistency
  timestamp: pino.stdTimeFunctions.isoTime,
});

module.exports = logger;

Writing Structured Logs

When using Pino, always pass an object as the first argument, followed by your log message. This ensures the keys become first-class queryable fields in your log aggregation system.

// paymentService.js
const logger = require('./logger');

async function processPayment(userId, amount, currency) {
  const transactionId = 'txn_' + Math.random().toString(36).substring(7);

  logger.info({ userId, amount, currency, transactionId }, 'Initiating payment processing');

  try {
    // Simulate payment gateway call
    if (amount > 10000) {
      throw new Error('Amount exceeds regulatory limit');
    }

    logger.info({ transactionId, status: 'SUCCESS' }, 'Payment processed successfully');
    return { transactionId, status: 'SUCCESS' };
  } catch (error) {
    logger.error(
      { err: error, userId, amount, transactionId }, 
      'Failed to process payment'
    );
    throw error;
  }
}

module.exports = { processPayment };

Notice how we pass { err: error }. Pino has a native error serializer that automatically extracts the stack trace, error name, and message into JSON fields.

Propagating Trace IDs Across Async Boundaries

In microservice architectures, a single user action may trigger downstream calls across multiple services. To trace a request from end to end, every log generated during that request lifecycle must share a unique traceId (or requestId).

We can achieve this in Node.js using pino-http and Node’s native AsyncLocalStorage (via pino-http’s built-in hooks) to maintain context across asynchronous boundaries without manual propogation.

// server.js
const express = require('express');
const pinoHttp = require('pino-http');
const logger = require('./logger');
const { processPayment } = require('./paymentService');

const app = express();
app.use(express.json());

// Configure pino-http to inject req.id and log requests
app.use(
  pinoHttp({
    logger,
    // Generate or extract incoming request ID
    genReqId: (req, res) => {
      const existingId = req.headers['x-request-id'];
      const id = existingId || 'req_' + Math.random().toString(36).substring(7);
      res.setHeader('x-request-id', id);
      return id;
    },
    customLogLevel: (res, err) => {
      if (res.statusCode >= 500 || err) return 'error';
      if (res.statusCode >= 400) return 'warn';
      return 'info';
    },
  })
);

app.post('/pay', async (req, res) => {
  const { userId, amount, currency } = req.body;
  
  // req.log is automatically bound to the request context (including req.id)
  req.log.info({ userId, amount }, 'Received payment request');

  try {
    const result = await processPayment(userId, amount, currency);
    return res.status(200).json(result);
  } catch (err) {
    return res.status(500).json({ error: err.message });
  fi}
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  logger.info({ port: PORT }, 'Server started successfully');
});

Every log line generated via req.log.info() inside the /pay route will now automatically include reqId, allowing you to query all logs associated with a single HTTP request.

Shipping Logs to Grafana Loki

Now that our Node.js app outputs pristine, high-performance structured JSON logs to stdout, we need a system to aggregate them. Grafana Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus.

Unlike traditional log stores (like Elasticsearch) that index the full text of every log, Loki only indexes metadata labels (like service, env, and level), keeping storage and compute costs drastically lower.

The Architecture

+------------------+       +-------------------+       +-------------+
| Node.js App      | ----> | Promtail / Fluent | ----> | Grafana     | 
| (stdout / JSON)  |       | (Scrapes & Ships) |       | Loki        | 
+------------------+       +-------------------+       +-------------+
                                                                ^
                                                                |
                                                       +-----------------+
                                                       | Grafana UI      |
                                                       +-----------------+

1. Docker Compose Setup

Let’s orchestrate our Node app, Loki, and Grafana using Docker Compose. Create a docker-compose.yml file in your project root:

version: '3.8'

services:
  loki:
    image: grafana/loki:2.9.2
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  grafana:
    image: grafana/grafana:10.2.2
    ports:
      - "3001:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    depends_on:
      - loki

  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - LOG_LEVEL=info
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

2. Configuring Promtail

Promtail is the agent that ships local container logs to Loki. Create a promtail-config.yaml file:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_name: container
      # Extract service label from container name
      - source_labels: ['__meta_docker_container_name']
        target_label: 'service'

(Alternatively, if running in Kubernetes, Promtail or Grafana Alloy will automatically scrape pod stdout and extract labels).

Querying Logs in Grafana (LogQL)

Once your stack is running and your Node.js application is receiving traffic, open Grafana at http://localhost:3001.

1. Navigate to Connections > Data Sources and add Loki with the URL: http://loki:3100.
2. Go to the Explore tab.

Loki uses LogQL, a query language similar to Prometheus PromQL. LogQL queries consist of two parts: Stream selectors (for filtering by labels) and Line filters (for text/JSON parsing).

Example Queries:

• View all error logs for our service:

  {service="app"} | json | level == "50"
  

  (Note: Pino maps error level to numeric code 50, info to 30, etc.)

• Trace a specific request ID across logs:

  {service="app"} | json | reqId="req_abc123"
  

• Calculate error rates per minute:

  sum(rate({service="app"} | json | level == "50" [1m]))
  

Because Pino formats logs as valid JSON, Loki’s | json parser automatically unpacks every field (userId, amount, transactionId, err.stack) into queryable attributes without requiring expensive regex parsing.

Production Best Practices Checklist

Before deploying your logging pipeline to production, keep these operational tips in mind:

• Never log sensitive data: Ensure tokens, passwords, PII, and credit card numbers are scrubbed or omitted before passing objects to Pino.
• Adjust log levels dynamically: Avoid running debug or trace levels in production unless actively troubleshooting, as the I/O and disk space overhead can become substantial.
• Handle unhandled rejections: Hook into Node.js process-level events to log catastrophic crashes before the process exits:

  process.on('unhandledRejection', (reason, promise) => {
    logger.fatal({ err: reason }, 'Unhandled Rejection detected. Shutting down...');
    process.exit(1);
  });
  

Conclusion

Moving away from unstructured console.log statements to structured JSON logging with Pino and Grafana Loki is a game-changer for Node.js backend reliability. By offloading serialization performance to Pino, preserving context with AsyncLocalStorage, and leveraging Loki’s label-based indexing, you create a lightning-fast, highly scalable observability pipeline that makes debugging production incidents painless.