Real-Time Audio Transcription in Node.js: Streaming Voice Streams to OpenAI Whisper via WebSockets

Building real-time features into modern web applications often involves text, but voice is rapidly becoming the primary interface for natural human-computer interaction. Whether you are building an AI meeting assistant, a voice-controlled dashboard, or a live accessibility tool, the ability to transcribe speech-to-text with minimal latency is essential.

While OpenAI’s Whisper API is the gold standard for transcription accuracy, it was originally designed to process static files rather than live streams. In this architectural guide, we will build a robust, production-grade Node.js pipeline that accepts raw binary audio chunks over WebSockets, buffers them intelligently, and streams them to the Whisper API for real-time transcription.

The Architecture Challenge

Bridging continuous WebSockets with a stateless REST API like OpenAI Whisper presents a fundamental architectural mismatch:

1. Continuous vs. Discrete: WebSockets are persistent, bi-directional, event-driven pipes delivering small, frequent binary chunks (e.g., PCM or WebM audio packets every 100ms).
2. File-Based Processing: The Whisper API expects a complete audio file container (like .mp3, .wav, or .webm) with proper headers, sizes, and file boundaries.

To bridge this gap, our Node.js backend must:

• Terminate WebSocket connections from the client.
• Accumulate binary audio frames into an in-memory buffer.
• Implement a time- or size-based chunking strategy to package these buffers into temporary file structures.
• Dispatch these chunks asynchronously to the OpenAI API.
• Broadcast the transcribed results back to the client in real time.

[Client Browser] --(Binary Audio via WebSocket)--> [Node.js Backend] --(Buffered Audio Chunks)--> [OpenAI Whisper API]
[Client Browser] <--(JSON Transcript via WebSocket)----- [Node.js Backend] <--(Transcription Result)------------/

Setting Up the Project

Let’s start by initializing our Node.js environment and installing the required dependencies. We will need ws for handling WebSocket connections, openai for interacting with the Whisper API, and standard Node.js utilities like fs and path for managing temporary audio files.

mkdir whisper-websocket-pipeline
cd whisper-websocket-pipeline
npm init -y
npm install ws openai dotenv
npm install -D nodemon

Update your package.json to include a start script:

{
  "name": "whisper-websocket-pipeline",
  "version": "1.0.0",
  "main": "server.js",
  "scripts": {
    "start": "node server.js",
    "dev": "nodemon server.js"
  },
  "dependencies": {
    "dotenv": "^16.4.5",
    "openai": "^4.28.0",
    "ws": "^8.16.0"
  }
}

Create a .env file in the root directory to store your OpenAI API key:

PORT=8080
OPENAI_API_KEY=your_openai_api_key_here

Building the WebSocket and Audio Buffer Pipeline

Now, let’s build the core backend logic. We need a server that listens for incoming WebSocket connections, instantiates an audio buffer per client session, and flushes the buffer to OpenAI at regular intervals (e.g., every 5 seconds) to maintain a near real-time stream.

Create server.js:

import { WebSocketServer } from 'ws';
import OpenAI from 'openai';
import dotenv from 'dotenv';
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import os from 'os';

dotenv.config();

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const wss = new WebSocketServer({ port: process.env.PORT || 8080 });

console.log(`WebSocket server running on ws://localhost:${process.env.PORT || 8080}`);

wss.on('connection', (ws) => {
  console.log('Client connected');

  // Client-specific state management
  let audioChunks = [];
  let isProcessing = false;

  // Buffer interval timer (e.g., process accumulated audio every 5 seconds)
  const BUFFER_INTERVAL_MS = 5000;

  const processingInterval = setInterval(async () => {
    if (audioChunks.length === 0 || isProcessing) return;

    // Capture current buffer and reset
    const chunksToProcess = [...audioChunks];
    audioChunks = [];
    isProcessing = true;

    try {
      // Combine chunks into a single Buffer
      const combinedBuffer = Buffer.concat(chunksToProcess);
      
      // Write to a temporary file for the OpenAI SDK
      const tempFilePath = path.join(os.tmpdir(), `audio-${Date.now()}.webm`);
      fs.writeFileSync(tempFilePath, combinedBuffer);

      console.log(`Sending ${combinedBuffer.length} bytes to Whisper API...`);

      // Call OpenAI Whisper API
      const transcription = await openai.audio.transcriptions.create({
        file: fs.createReadStream(tempFilePath),
        model: 'whisper-1',
        language: 'en',
        response_format: 'json',
      });

      // Clean up temp file
      fs.unlinkSync(tempFilePath);

      // Send transcript back to client
      if (ws.readyState === ws.OPEN) {
        ws.send(JSON.stringify({
          event: 'transcript',
          text: transcription.text,
        }));
      }
    } catch (error) {
      console.error('Error processing audio chunk:', error);
      if (ws.readyState === ws.OPEN) {
        ws.send(JSON.stringify({
          event: 'error',
          message: 'Failed to transcribe audio',
        }));
      }
    } finally {
      isProcessing = false;
    }
  }, BUFFER_INTERVAL_MS);

  // Handle incoming binary audio streams from client
  ws.on('message', (message, isBinary) => {
    if (isBinary) {
      audioChunks.push(message);
    } else {
      try {
        const data = JSON.parse(message.toString());
        if (data.event === 'stop') {
          console.log('Client requested stream termination.');
        }
      } catch (err) {
        console.error('Invalid JSON message received');
      }
    }
  });

  ws.on('close', () => {
    console.log('Client disconnected');
    clearInterval(processingInterval);
  });
});

Optimizing Chunking and Context Continuity

While fixed-interval buffering (e.g., every 5 seconds) is simple to implement, it introduces a hard edge. If a user speaks across the 5-second boundary, words can get clipped or cut off mid-syllable, leading to hallucinations or missing words in the Whisper output.

Overlapping Sliding Windows

To solve context clipping, advanced real-time pipelines implement an overlapping sliding window. Instead of completely flushing the buffer, you retain the last 1–2 seconds of audio to prepend to the next batch.

// Conceptual sliding window extraction
const OVERLAP_DURATION_BYTES = 16000 * 2; // Example for 16kHz 16-bit audio
let overlapBuffer = Buffer.alloc(0);

// When slicing chunks for processing:
const currentBatch = Buffer.concat([overlapBuffer, ...newChunks]);
// Save the tail end for the next iteration
overlapBuffer = currentBatch.subarray(currentBatch.length - OVERLAP_DURATION_BYTES);

Crafting the Client-Side Audio Capture

To test our Node.js WebSocket pipeline, we need a lightweight client capable of capturing microphone input via the browser’s MediaRecorder API and streaming raw binary data over WebSockets.

Create an index.html file in your project directory:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Real-Time Whisper Transcription</title>
    <style>
        body { font-family: sans-serif; max-width: 600px; margin: 40px auto; padding: 20px; }
        #transcript { background: #f4f4f4; padding: 15px; border-radius: 5px; min-height: 150px; white-space: pre-wrap; }
        button { padding: 10px 20px; font-size: 16px; cursor: pointer; }
        .recording { background: #ff4d4d; color: white; }
    </style>
</head>
<body>
    <h1>Live Speech-to-Text with Whisper</h1>
    <div>
        <button id="toggleBtn" onclick="toggleRecording()">Start Recording</button>
    </div>
    <h3>Live Transcript:</h3>
    <div id="transcript"></div>

    <script>
        let ws;
        let mediaRecorder;
        let isRecording = false;

        const toggleBtn = document.getElementById('toggleBtn');
        const transcriptDiv = document.getElementById('transcript');

        async function toggleRecording() {
            if (!isRecording) {
                // Connect to Node.js WebSocket server
                ws = new WebSocket('ws://localhost:8080');

                ws.onopen = async () => {
                    const stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
                    
                    // Use webm/opus codec which streams efficiently
                    mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm;codecs=opus' });

                    mediaRecorder.ondataavailable = async (event) => {
                        if (event.data.size > 0 && ws.readyState === WebSocket.OPEN) {
                            // Send raw binary blob directly over WebSocket
                            const arrayBuffer = await event.data.arrayBuffer();
                            ws.send(arrayBuffer);
                        }
                    };

                    // Fire dataavailable every 1000ms (1 second)
                    mediaRecorder.start(1000);
                    isRecording = true;
                    toggleBtn.textContent = 'Stop Recording';
                    toggleBtn.classList.add('recording');
                };

                ws.onmessage = (event) => {
                    const data = JSON.parse(event.data);
                    if (data.event === 'transcript') {
                        transcriptDiv.textContent += data.text + ' ';
                    }
                };

                ws.onclose = () => {
                    stopRecordingUI();
                };

            } else {
                stopRecording();
            }
        }

        function stopRecording() {
            if (mediaRecorder) {
                mediaRecorder.stop();
                mediaRecorder.stream.getTracks().forEach(track => track.stop());
            }
            if (ws) {
                ws.close();
            }
            stopRecordingUI();
        }

        function stopRecordingUI() {
            isRecording = false;
            toggleBtn.textContent = 'Start Recording';
            toggleBtn.classList.remove('recording');
        }
    </script>
</body>
</html>

To test this locally, serve index.html using a simple static file server (like npx serve) and open it in your browser while running your Node.js backend (npm run dev).

Production Considerations and Edge Cases

Running a streaming transcription pipeline in production requires accounting for latency, resource management, and cost:

Cost Warning: The OpenAI Whisper API charges per second of audio processed. Sending audio clips every 5 seconds creates high request volume. For heavy production usage, consider deploying an open-source Whisper model (like whisper.cpp or faster-whisper) locally on a GPU server (e.g., AWS EC2 with NVIDIA T4) to eliminate API costs and reduce network latency further.

1. Backpressure Management

If a client connection drops or network congestion slows down API requests, your Node.js server’s memory usage can spike as audio chunks accumulate. Implement strict buffer limits:

const MAX_BUFFER_SIZE = 1024 * 1024 * 5; // 5MB limit per client
if (combinedBuffer.length > MAX_BUFFER_SIZE) {
    ws.send(JSON.stringify({ event: 'error', message: 'Buffer overflow exceeded.' }));
    ws.close();
}

2. Audio Format Normalization

Browsers vary in their native audio recording codecs (audio/webm, audio/ogg, audio/mp4). If you run into parsing issues with OpenAI Whisper, you can use fluent-ffmpeg in Node.js to normalize incoming byte streams into standard 16kHz mono WAV files before forwarding them to the API.

Conclusion

By pairing WebSockets with Node.js and the OpenAI Whisper API, you can construct a powerful, responsive real-time transcription pipeline. While managing buffer intervals and temporary files requires careful system design, this pattern unlocks seamless voice-enabled features for modern web applications.

Whether you stick with OpenAI’s cloud API for effortless scalability or migrate to a self-hosted faster-whisper instance for ultra-low latency, the WebSocket ingestion architecture outlined here provides a rock-solid foundation for real-time AI audio processing.