Beyond Correlation IDs: Implementing OpenTelemetry and Distributed Tracing in Node.js

For years, the standard approach to debugging distributed Node.js architectures involved passing a X-Request-ID header down the wire. You would log this correlation ID at every hop, grep through your log aggregator (ELK, Datadog, or CloudWatch), and try to reconstruct the timeline of a request.

While correlation IDs are better than flying blind, they fall short in modern microservice architectures. They do not tell you:

• Which exact function or database query caused a latency spike.
• The hierarchical relationship between downstream service calls.
• The exact timing and payload metadata of asynchronous operations.

Enter Distributed Tracing. By adopting the OpenTelemetry (OTel) standard, you can automatically capture metrics, logs, and traces, and stream them to visualization tools like Jaeger.

In this tutorial, we will build a production-grade OpenTelemetry setup in Node.js, auto-instrument HTTP and database calls, manually create custom spans, and view the resulting traces in Jaeger.

The Architecture

To demonstrate a distributed setup, we will create a small ecosystem:

1. API Gateway (gateway-service): Accepts client requests and forwards them downstream.
2. User Service (user-service): Handles user queries and talks to a PostgreSQL database.
3. Jaeger: Collects and visualizes the trace data.

[Client] ---> (HTTP) ---> [Gateway Service] ---> (HTTP) ---> [User Service] ---> (pg) ---> [PostgreSQL]
                                   |
                                   v (OTLP/gRPC)
                               [Jaeger]

Step 1: Setting up the OTel Initialization File

In OpenTelemetry, initialization must happen before any application code (like Express or Mongoose/pg) is loaded. This is because auto-instrumentation libraries patch Node.js modules upon require.

Create a file named tracing.js in the root of your project:

'use strict';

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { SEMRESATTRS_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');

// Configure the OTLP exporter to send traces to Jaeger
const traceExporter = new OTLPTraceExporter({
  url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4317',
});

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: process.env.SERVICE_NAME || 'unknown-service',
  }),
  traceExporter,
  instrumentations: [
    getNodeAutoInstrumentations({
      // Configure specific auto-instrumentations if needed
      '@opentelemetry/instrumentation-fs': { enabled: false }, // Reduce noise
    }),
  ],
});

// Initialize the SDK and register with the OpenTelemetry API
sdk.start();

console.log(`[OpenTelemetry] Tracing initialized for service: ${process.env.SERVICE_NAME}`);

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk
    .shutdown()
    .then(() => console.log('Tracing terminated successfully'))
    .catch((error) => console.log('Error terminating tracing', error))
    .finally(() => process.exit(0));
});

Step 2: Installing Dependencies

Install the required core OpenTelemetry packages, the auto-instrumentation bundle, and the OTLP gRPC exporter:

npm install \
  @opentelemetry/api \
  @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-grpc \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions \
  express \
  pg

Step 3: Implementing the User Service

The User Service connects to PostgreSQL and fetches user data. Because we imported @opentelemetry/auto-instrumentations-node, our pg (PostgreSQL) and express libraries will be instrumented automatically without writing custom tracing code for database calls.

Create user-service.js:

// MUST be the very first import
require('./tracing');

const express = require('express');
const { Pool } = require('pg');

const app = express();
const port = process.env.PORT || 3002;

// PostgreSQL connection pool
const pool = new Pool({
  connectionString: process.env.DATABASE_URL || 'postgres://postgres:postgres@localhost:5432/usersdb',
});

app.use(express.json());

app.get('/users/:id', async (req, res) => {
  const userId = req.params.id;
  
  try {
    // This query span will be automatically captured and linked to the incoming trace
    const query = 'SELECT id, name, email, created_at FROM users WHERE id = $1';
    const { rows } = await pool.query(query, [userId]);

    if (rows.length === 0) {
      return res.status(404).json({ error: 'User not found' });
    }

    res.json(rows[0]);
  } catch (error) {
    console.error('Database error:', error);
    res.status(500).json({ error: 'Internal Server Error' });
  });
});

app.listen(port, () => {
  console.log(`User Service running on port ${port}`);
});

Step 4: Implementing the API Gateway with Custom Spans

The API Gateway receives external requests, performs downstream HTTP calls to the user-service, and propagates context headers (traceparent) automatically via the HTTP auto-instrumentation.

Sometimes, auto-instrumentation doesn’t capture business logic context. For this, we use the OpenTelemetry API to create custom spans.

Create gateway-service.js:

// MUST be the very first import
require('./tracing');

const express = require('express');
const http = require('http');
const { trace, context } = require('@opentelemetry/api');

const app = express();
const port = process.env.PORT || 3001;

// Get a tracer instance
const tracer = trace.getTracer('gateway-service');

// Helper function for downstream HTTP calls
const fetchUserFromService = (userId) => {
  return new Promise((resolve, reject) => {
    const userServiceUrl = process.env.USER_SERVICE_URL || 'http://localhost:3002';
    
    http.get(`${userServiceUrl}/users/${userId}`, (res) => {
      let data = '';
      res.on('data', (chunk) => (data += chunk));
      res.on('end', () => {
        if (res.statusCode >= 400) {
          return reject(new Error(`Downstream service error: ${res.statusCode}`));
        }
        resolve(JSON.parse(data));
      });
    }).on('error', reject);
  });
};

app.get('/api/profile/:id', async (req, res) => {
  const userId = req.params.id;

  // Create a custom span for business logic processing
  return tracer.startActiveSpan('process-user-profile', async (span) => {
    // Set attributes on the custom span
    span.setAttribute('user.id', userId);
    span.setAttribute('profile.version', 'v1');

    try {
      console.log(`Fetching profile for user: ${userId}`);
      
      // Downstream HTTP call (auto-instrumented and context-propagated)
      const user = await fetchUserFromService(userId);

      span.addEvent('user_data_fetched', { 'user.email': user.email });
      
      res.json({
        status: 'success',
        data: user,
      });
    } catch (error) {
      // Record exceptions directly into the span
      span.recordException(error);
      span.setStatus({ code: 2, message: error.message });
      
      res.status(500).json({ error: error.message });
    } finally {
      // Always end your custom spans
      span.end();
    }
  });
});

app.listen(port, () => {
  console.log(`Gateway Service running on port ${port}`);
});

Key Takeaway: Notice how we didn’t manually extract or inject headers for fetchUserFromService. The @opentelemetry/instrumentation-http package automatically injects the W3C Trace Context headers (traceparent) into outgoing HTTP requests, allowing Jaeger to stitch the gateway-service and user-service traces together seamlessly.

Step 5: Running Jaeger Locally with Docker

Jaeger provides an all-in-one Docker image that includes the collector, query service, and UI. Run the following command to spin it up:

docker run --rm --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/all-in-one:latest

• Port 16686: Jaeger Query UI.
• Port 4317: OTLP gRPC receiver endpoint used by our Node.js SDKs.

Step 6: Running the Services and Generating Traces

Start your services in separate terminal windows with their respective environment variables:

Terminal 1: User Service

export SERVICE_NAME=user-service
export PORT=3002
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
node user-service.js

Terminal 2: Gateway Service

export SERVICE_NAME=gateway-service
export PORT=3001
export USER_SERVICE_URL=http://localhost:3002
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
node gateway-service.js

Terminal 3: Fire a Request

Use curl to hit the API Gateway:

curl http://localhost:3001/api/profile/1

Step 7: Visualizing in Jaeger

1. Open your browser and navigate to http://localhost:16686.
2. In the left sidebar, select Service: gateway-service.
3. Click Find Traces.

You will see a waterfall chart depicting the full lifecycle of your request:

gateway-service (GET /api/profile/:id)
 └── process-user-profile (Custom Span)
      └── HTTP GET /users/1 (Auto-instrumented)
           └── user-service (GET /users/:id)
                └── pg: SELECT id, name, email... (Auto-instrumented DB Span)

Clicking on individual spans reveals metadata such as HTTP status codes, SQL query strings, execution durations, and custom attributes (user.id = 1) that you added manually.

Best Practices for Production OTLP in Node.js

1. Always load tracing.js first: Use the Node.js -r flag to avoid race conditions with module loading:

   node -r ./tracing.js gateway-service.js
   

2. Avoid High Cardinality Attributes: Never put UUIDs, full request bodies, or user passwords into span names. Use attributes (span.setAttribute()) for searchable metadata instead.
3. Handle Sampling: In high-throughput production environments, exporting 100% of traces will saturate your network and backend storage. Configure a probabilistic sampler in your NodeSDK configuration:

   const { ParentBasedSampler, TraceIdRatioBasedSampler } = require('@opentelemetry/sdk-trace-base');
   
   const sdk = new NodeSDK({
     // Sample 20% of traces
     sampler: new ParentBasedSampler({
       root: new TraceIdRatioBasedSampler(0.2),
     }),
     // ...
   });
   

Conclusion

Correlation IDs served us well, but distributed tracing gives us the granular visibility required to debug complex microservices. By combining OpenTelemetry’s auto-instrumentation for HTTP and PostgreSQL with custom spans for critical business logic, you unlock deep insights into latency bottlenecks and error paths without cluttering your codebase with manual logging boilerplates.