Offline-First Collaboration: Persisting Yjs State with IndexedDB and Service Workers
{
{ “title”: “Offline-First Collaboration: Persisting Yjs State with IndexedDB and Service Workers”, “summary”: “A technical architecture guide walking through setting up robust local persistence for Yjs documents, handling IndexedDB bottlenecks, and managing reconnect sync queues.”, “tags”: [“Frontend”, “IndexedDB”, “Distributed Systems”, “Offline-First”, “Architecture”], “body”: “Modern web applications demand real-time collaboration. Libraries like Yjs make this possible by using Conflict-free Replicated Data Types (CRDTs), enabling multiple users to edit the same document concurrently and converge automatically. However, real-time collaboration often assumes a persistent, stable network connection. What happens when a user loses internet access, closes their browser abruptly, or refreshes mid-edit?
Building a truly resilient offline-first architecture requires combining Yjs with a robust local persistence layer and a reliable synchronization strategy. This article explores how to architect an offline-first persistence and synchronization pipeline using IndexedDB, Service Workers, and Yjs without losing a single byte of user state.
The Architecture of Offline Yjs State
At its core, a Yjs document (Y.Doc) is an event emitter containing a shared data structure. It manages state as a binary log of updates. To achieve offline persistence, we need to capture these updates locally, store them efficiently, and replay them upon reconnection.
+-------------------------------------------------------+
| Browser |
| |
| +-------------+ Updates +--------------+ |
| | Y.Doc | <----------------> | IndexedDB | |
| +-------------+ +--------------+ |
| | | |
| | Sync Queue | |
| v v |
| +-------------------------------------------------+ |
| | Service Worker | |
| +-------------------------------------------------+ |
| | |
+--------------------------|----------------------------+
v Network
+-------------------+
| Remote Server |
+-------------------+
Our architecture consists of three main pillars:
- Local-First Storage: Capturing every Yjs transaction and persisting it to IndexedDB.
- Background Sync: Utilizing Service Workers to queue and flush pending mutations when network connectivity returns.
- Graceful Shutdown Handling: Flushing memory buffers to disk when the browser or tab unloads unexpectedly.
1. Setting Up IndexedDB Persistence for Yjs
While localStorage is synchronous and limited to 5MB, IndexedDB is asynchronous, transactional, and capable of storing gigabytes of binary data. Because Yjs updates are binary Uint8Arrays, IndexedDB is the ideal native browser store.
The Pitfall of Naive Storage
If you save the entire document state as a single monolithic blob on every transaction, write amplification will eventually freeze the main thread. As the document grows, serializing and writing the entire state tree degrades performance.
Instead, we use an incremental update log combined with occasional state snapshots.
Implementing the Storage Engine
We can implement a robust persistence layer using y-indexeddb as a conceptual blueprint, or build a tailored engine that stores updates sequentially:
import * as Y from 'yjs';
import { openDB } from 'idb';
const DB_NAME = 'collaborative-editor-db';
const STORE_NAME = 'yjs-updates';
export async function initDB(docId) {
return openDB(`${DB_NAME}-${docId}`, 1, {
upgrade(db) {
if (!db.objectStoreNames.contains(STORE_NAME)) {
db.createObjectStore(STORE_NAME, { autoIncrement: true });
}
},
});
}
// Persist individual updates asynchronously
export async function storeUpdate(db, update) {
const tx = db.transaction(STORE_NAME, 'readwrite');
await tx.store.add(update);
await tx.done;
}
// Load all updates and apply them to bootstrap the Y.Doc
export async function loadPersistedState(doc) {
const db = await initDB(doc.guid);
const tx = db.transaction(STORE_NAME, 'readonly');
const updates = await tx.store.getAll();
if (updates.length > 0) {
Y.transact(doc, () => {
updates.forEach(update => {
Y.applyUpdate(doc, update);
});
}, 'init-from-idb');
}
return { db, updatesCount: updates.length };
}
By listening to the update event on the Y.Doc, we stream every incremental change directly into IndexedDB:
export function bindPersistence(doc, db) {
doc.on('update', async (update, origin) => {
// Avoid writing updates back to IDB that originated from IDB itself
if (origin === 'init-from-idb') return;
await storeUpdate(db, update);
});
}
2. Preventing Data Loss on Abrupt Browser Closures
Users frequently close tabs, experience crashes, or lose power. Relying solely on asynchronous IndexedDB promises inside standard event listeners is risky because the browser may terminate execution before the write transaction completes.
The beforeunload and visibilitychange Strategy
To prevent data loss during sudden closures, we must combine standard event hooks with synchronous state checkpoints where possible, or leverage the pagehide event alongside navigator.sendBeacon for network syncs.
window.addEventListener('visibilitychange', async () => {
if (document.visibilityState === 'hidden') {
// Force a local checkpoint or flush memory buffers
await checkpointDocumentState(doc);
}
});
window.addEventListener('pagehide', (event) => {
if (!event.persisted) {
// The page is being unloaded. Ensure final state is safely queued.
const currentState = Y.encodeStateAsUpdate(doc);
navigator.sendBeacon('/api/sync/beacon', currentState);
}
});
State Compaction (Garbage Collection)
Over time, a document with thousands of incremental updates will suffer from bloated storage. Periodically, you should compute a full state vector snapshot, write that snapshot as a single entry, and safely prune the historical micro-updates.
async function compactDatabase(doc, db) {
const stateVector = Y.encodeStateAsUpdate(doc);
const tx = db.transaction(STORE_NAME, 'readwrite');
const store = tx.store;
// Clear old updates and replace with the definitive snapshot
await store.clear();
await store.add(stateVector);
await tx.done;
}
3. Managing Sync Queues and Network Reconnections
When a user works offline, changes accumulate locally in IndexedDB. When network connectivity is restored, these updates must be synchronized with the central server without causing race conditions or data duplication.
The Sync Queue State Machine
We track network states explicitly and maintain an outgoing queue of updates:
class OfflineSyncManager {
constructor(doc, websocketEndpoint) {
this.doc = doc;
this.endpoint = websocketEndpoint;
this.isOnline = navigator.onLine;
this.syncQueue = [];
window.addEventListener('online', () => this.handleOnline());
window.addEventListener('offline', () => this.handleOffline());
}
handleOffline() {
this.isOnline = false;
console.warn('[Sync] Network lost. Switching to offline-first mode.');
}
async handleOnline() {
this.isOnline = true;
console.log('[Sync] Network restored. Flushing sync queue...');
await this.flushQueue();
}
async queueUpdate(update) {
if (this.isOnline) {
this.sendToServer(update);
} else {
this.syncQueue.push(update);
}
}
async flushQueue() {
while (this.syncQueue.length > 0 && this.isOnline) {
const update = this.syncQueue.shift();
try {
await this.sendToServer(update);
} catch (err) {
// Re-queue on failure and break loop
this.syncQueue.unshift(update);
this.isOnline = false;
break;
}
}
}
async sendToServer(update) {
// Implementation of WebSocket or HTTP chunk upload
return window.collaborativeSocket.send(update);
}
}
4. Leveraging Service Workers for Background Sync
For web applications that need to sync data even if the user navigates away from the active tab, the Background Synchronization API (managed via Service Workers) provides a reliable mechanism.
Registering a Sync Event
In your main application thread:
async function registerBackgroundSync(docId) {
if ('serviceWorker' in navigator && 'SyncManager' in window) {
const registration = await navigator.serviceWorker.ready;
try {
await registration.sync.register(`sync-yjs-doc-${docId}`);
console.log('[ServiceWorker] Background sync registered.');
} catch (err) {
console.error('[ServiceWorker] Background sync registration failed:', err);
}
}
}
Handling Sync in the Service Worker (sw.js)
self.addEventListener('sync', (event) => {
if (event.tag.startsWith('sync-yjs-doc-')) {
event.waitUntil(syncPendingData(event.tag));
}
});
async function syncPendingData(tag) {
const docId = tag.replace('sync-yjs-doc-', '');
// Open IDB from Service Worker context, extract unsynced mutations,
// and push them to the upstream coordination server via Fetch API.
const updates = await fetchUnsyncedUpdatesFromIDB(docId);
for (const update of updates) {
const response = await fetch(`/api/docs/${docId}/sync`, {
method: 'POST',
body: update,
headers: { 'Content-Type': 'application/octet-stream' }
});
if (response.ok) {
await markUpdateAsSyncedInIDB(docId, update.id);
} else {
throw new Error('Sync failed, will retry.');
}
}
}
Summary and Production Checklist
Implementing rock-solid offline persistence for Yjs requires treating local storage as a first-class citizen rather than an afterthought. By following these architectural patterns, your application will handle flaky mobile connections, sudden browser crashes, and lengthy offline sessions seamlessly.
Production Checklist:
- Incremental Writes: Never rewrite the entire Yjs state blob on every keystroke; stream incremental updates to IndexedDB.
- State Compaction: Periodically compute snapshots and clean up historical logs to prevent database bloat.
- Event Safety: Listen to
visibilitychangeandpagehideto capture data before abrupt unloads. - Resilient Queuing: Build an explicit offline queue that gracefully retries when network conditions stabilize.
- Background Sync: Use Service Workers to ensure data syncs even if the user closes the application tab.” }