Preventing Race Conditions in Distributed Node.js Apps with Redis and Redlock

As applications scale, moving from a monolithic, single-instance Node.js server to a distributed, horizontally-scaled architecture introduces a host of concurrency challenges. While in-memory synchronization primitives like mutexes, semaphores, or even simple JavaScript variables are sufficient for single-process applications, they break down immediately when multiple Node.js instances run across different servers or containers.

Imagine a scenario where two API instances process requests for the same user simultaneously. Both check the database for remaining inventory or account balance, both see sufficient funds, and both proceed to execute a financial transaction. Suddenly, you have a race condition, double-spending, or inventory overselling.

To solve this, we need distributed locks. In this guide, we will explore how Redis handles atomic operations and how to safely implement the Redlock algorithm in Node.js without introducing deadlocks or performance bottlenecks.

The Anatomy of a Distributed Lock

A distributed lock is a mechanism that restricts access to a shared resource in a distributed system. Unlike a local lock, a distributed lock must coordinate state across network boundaries.

At its core, a robust distributed lock must guarantee three properties (as defined by antirez, the creator of Redis):

1. Mutual Exclusion: At any given moment, only one client can hold a lock.
2. Deadlock Free: Eventually, a lock can always be acquired, even if the client that locked it crashes or gets partitioned.
3. Fault Tolerance: As long as the majority of Redis nodes are up, clients can acquire and release locks.

Why a Simple SETNX Isn’t Enough

If you are familiar with Redis, you might think of using SETNX (Set if Not Exists) to acquire a lock:

SETNX resource_name random_value

While this sets a lock if it doesn’t exist, it has critical flaws:

• No Expiration: If the Node.js process crashes immediately after acquiring the lock, the key remains in Redis forever, causing a permanent deadlock.
• Unsafe Release: If a process takes longer than expected to execute its critical section, its lock might expire. If another process acquires the lock, the original process might eventually finish and delete the other process’s lock.

To fix this, we need expiration times (TTL) and unique tokens to ensure a client only deletes the lock it actually created.

—

Entering Redlock: The Multi-Node Algorithm

In a standard master-slave Redis setup, what happens if the master node crashes right after a client acquires a lock, but before the lock replication reaches the replica? A failover occurs, a replica is promoted to master, and another client can acquire the same lock, breaking mutual exclusion.

To solve this, the Redlock algorithm was introduced. Instead of relying on a single Redis instance, Redlock assumes you have $N$ independent Redis masters (typically 5 distinct instances running on separate physical machines or VMs).

The algorithm executes the following steps to acquire a lock:

1. Get Current Time: The client gets the current time in milliseconds.
2. Sequential Acquisition: The client tries to acquire the lock in all $N$ instances sequentially, using the same key and random value (token), with a very small timeout compared to the auto-release time (e.g., if TTL is 10s, timeout per node should be 5-50ms).
3. Calculate Elapsed Time: The client computes how much time elapsed to acquire the locks. The lock is considered successfully acquired only if the client managed to acquire the lock in a majority of the instances (at least 3 out of 5), and the total time taken was less than the lock validity time.
4. Adjust Validity: If the lock was acquired, its true validity time is considered the initial TTL minus the elapsed time.
5. Failure Cleanup: If the client failed to acquire the lock (either because it couldn’t lock a majority, or validity time is negative), it unlocks all instances.

Implementing Redlock in Node.js

Let’s build a practical implementation using Node.js, ioredis (a robust Redis client), and the official redlock npm package.

1. Project Setup

Initialize a new Node.js project and install the required dependencies:

npm init -y
npm install ioredis redlock dotenv

2. Configuring Redis Clients

Create a file named redlock-client.js. In a production environment, your Redlock instance should connect to independent Redis nodes. For local development, you can spin up multiple Redis instances on different ports using Docker.

// redlock-client.js
import Redis from 'ioredis';
import Redlock from 'redlock';

// Initialize independent Redis clients for Redlock
// In production, these should point to separate physical nodes/servers
const client1 = new Redis({ host: '127.0.0.1', port: 6379 });
const client2 = new Redis({ host: '127.0.0.1', port: 6380 });
const client3 = new Redis({ host: '127.0.0.1', port: 6381 });

const redlock = new Redlock(
  [client1, client2, client3],
  {
    // The expected clock drift; for more details see:
    // http://redis.io/topics/distlock
    driftFactor: 0.01, // multiplied by lock ttl to determine drift

    // The maximum number of times an attempt will be made to lock a resource
    // before a error is thrown
    retryCount: 10,

    // the time in ms between retries
    retryDelay: 200, // time in ms

    // the max variance in retry delay for randomized exponential backoff
    retryJitter: 200, // time in ms

    // The maximum duration of a command execution
    automaticExtensionThreshold: 500, // ms
  }
);

redlock.on('error', (error) => {
  // Ignore resource contention errors, but log infrastructure errors
  if (error instanceof Redlock.ResourceLockedError) {
    return;
  }
  console.error('Redlock Error:', error);
});

export default redlock;

3. Handling Race Conditions in a Service

Now let’s use our configured Redlock instance inside a critical section, such as processing a payment or updating a constrained resource.

// orderService.js
import redlock from './redlock-client.js';

/**
 * Simulates updating a user's account balance safely across distributed instances.
 */
export async function processUserCheckout(userId, orderAmount) {
  const resource = `locks:user:${userId}`;
  let lock;

  try {
    // Try to acquire a lock with a 5-second TTL
    // acquire(resource, ttl)
    lock = await redlock.acquire([resource], 5000);
    
    console.log(`[Instance PID: ${process.pid}] Acquired lock for user ${userId}`);

    // --- CRITICAL SECTION START ---
    
    // 1. Fetch current user state from database
    const userAccount = await simulateDatabaseFetch(userId);
    
    if (userAccount.balance < orderAmount) {
      throw new Error('Insufficient funds');
    }

    // 2. Perform slow business logic / DB updates
    await simulateDatabaseUpdate(userId, userAccount.balance - orderAmount);
    
    console.log(`[Instance PID: ${process.pid}] Successfully processed order for user ${userId}`);
    
    // --- CRITICAL SECTION END ---

  } catch (error) {
    console.error(`[Instance PID: ${process.pid}] Failed to process order:`, error.message);
    throw error;
  } finally {
    // ALWAYS release the lock in the finally block
    if (lock) {
      try {
        await lock.release();
        console.log(`[Instance PID: ${process.pid}] Released lock for user ${userId}`);
      } catch (releaseError) {
        // Lock may have already expired or been released
        console.error(`[Instance PID: ${process.pid}] Failed to release lock:`, releaseError.message);
      }
    }
  }
}

// Mock database helpers
async function simulateDatabaseFetch(userId) {
  await new Promise((resolve) => setTimeout(resolve, 100));
  return { userId, balance: 100 };
}

async function simulateDatabaseUpdate(userId, newBalance) {
  await new Promise((resolve) => setTimeout(resolve, 200));
  return { userId, balance: newBalance };
}

Advanced Patterns: Extending Locks and Auto-Extension

What happens if your critical section takes longer than your lock’s TTL (e.g., your TTL is 5 seconds, but an external API call takes 7 seconds)?

If the lock expires mid-execution, another Node.js instance could acquire the lock, leading to concurrent execution.

Using using() for Automatic Lock Extension

The redlock library provides a helper method called using() that automatically extends the lock’s TTL while your asynchronous function is still executing, provided the operation doesn’t exceed your safety limits.

import redlock from './redlock-client.js';

export async function processLongRunningJob(jobId) {
  const resource = `locks:job:${jobId}`;
  const ttl = 5000; // Initial 5 seconds

  try {
    await redlock.using([resource], ttl, async (signal) => {
      // This callback will be executed.
      // If execution takes longer than expected, Redlock will automatically
      // attempt to extend the lock TTL in the background.
      
      while (!signal.aborted) {
        await performStepOfJob();
        
        if (isJobFinished()) {
          break;
        }
      }
    });
    
    console.log('Job completed successfully with auto-extended lock.');
  } catch (error) {
    console.error('Job execution failed or lock expired:', error);
  }
}

async function performStepOfJob() {
  await new Promise((res) => setTimeout(res, 2000));
}

let steps = 0;
function isJobFinished() {
  steps++;
  return steps >= 4; // Takes ~8 seconds total, requiring extension
}

Warning on Auto-Extensions: While convenient, relying heavily on auto-extensions can mask underlying performance issues or deadlocks. Always try to size your initial TTL appropriately for your expected execution bounds.

Architectural Best Practices & Pitfalls

Implementing distributed locks adds network overhead and failure modes. Keep these production best practices in mind:

1. Clock Drift Matters

Redlock relies on system clocks being reasonably synchronized. If a server’s system clock jumps forward due to NTP synchronization issues, a lock might expire prematurely, breaking safety guarantees. Ensure ntpd or chrony is actively running and configured to avoid large time jumps on your Redis nodes.

2. Don’t Use Locks for Everything

Distributed locks are a tool of last resort. Whenever possible, design your architecture around idempotent operations, optimistic concurrency control (OCC) using version numbers, or database-level unique constraints.

• Bad: Locking a user table row for every basic profile read.
• Good: Using a distributed lock only when mutating critical financial records or running single-instance background cron tasks across a cluster.

3. Handle Network Partitions Gracefully

If your Node.js application experiences a network partition and loses connection to the Redis cluster, it must fail fast rather than assuming it still holds the lock. Always catch lock release and acquisition errors explicitly.

Conclusion

As Node.js applications scale into clustered and distributed environments, race conditions become an inevitable hazard. While simple Redis commands like SETNX fall short in multi-node topologies, the Redlock algorithm provides a mathematically sound, fault-tolerant foundation for distributed synchronization.

By carefully managing lock TTLs, utilizing libraries like redlock for Node.js, and wrapping critical sections in robust try/finally blocks, you can ensure data integrity and bulletproof concurrency across all your microservices and API instances.