Browser-Side Agentic Engine Architecture: Multithreading, ONNX, and WebGPU Memory Management

If your browser-side AI app freezes the main UI thread during matrix multiplication, your architecture is broken.

Running neural networks inside browser tabs requires treating the client runtime like an embedded operating system.

You cannot rely on server-side vLLM clusters or endless VRAM allocations. Every megabyte of RAM and GPU compute slice must be managed defensively.

The Setup

When expanding our Systems Lab to support dual-model parallel execution in the Edge Arena (/lab/edge-arena), we needed to run two distinct LLMs simultaneously in the browser.

The goal was to benchmark latency (Time-To-First-Token) and generation speed (tokens per second) across models like SmolLM2-135M vs Qwen2.5-0.5B.

The engine had to maintain smooth 60 FPS scrolling, respond instantly to user input, and manage model weight downloads without exhausting browser memory limits.

The Mess

Our initial implementation allocated WebGPU device contexts directly on the main window thread.

When loading quantized 4-bit ONNX weights (~120MB to 350MB), the browser frozen for 4.2 seconds during shader compilation.

[DOM Exception] Rendering thread blocked for 4180ms by synchronous WebGPU pipeline creation.
[Chrome DevTools] Frame drop detected: 251 frames dropped during ONNX model initialization.

Worst of all, switching browser tabs purged the WebGPU device context, causing active inference streams to fail with unrecoverable buffer loss errors.

The Solution

I built a decoupled, multi-threaded Web Worker architecture governed by zero-cost intent gates.

1. Dual Isolated Web Workers: Each model runs inside an independent Web Worker thread (model-a.worker.ts and model-b.worker.ts), preventing main thread UI blocking.
2. Intent Gate Token Cache (IntentGate.astro): Defer neural network downloading and shader compilation until the user confirms intent or presents a cached token in localStorage.
3. CacheStorage Model Seeding: Model weights are cached using the browser’s CacheStorage API, turning 120MB network downloads into instant 5ms local disk reads on subsequent visits.

// Multi-Worker Dual Execution Engine
export class EdgeArenaManager {
  private workerA: Worker;
  private workerB: Worker;

  constructor() {
    this.workerA = new Worker(new URL('./engine.worker.ts', import.meta.url), {
      type: 'module',
    });
    this.workerB = new Worker(new URL('./engine.worker.ts', import.meta.url), {
      type: 'module',
    });
  }

  public async runParallelBenchmark(
    prompt: string,
    modelA: string,
    modelB: string,
  ) {
    this.workerA.postMessage({ command: 'INFER', model: modelA, prompt });
    this.workerB.postMessage({ command: 'INFER', model: modelB, prompt });
  }
}

Measured Architectural Gains

• Main Thread UI Frame Rate: Maintained at 60 FPS during dual 100% GPU utilization.
• Warm Model Load Time: Reduced from 4,200ms to 8ms via CacheStorage API.
• Server KV Writes: Reduced to 0/day by storing state and theme affinity entirely in localStorage.

Key Takeaway

Treat client-side WebGPU applications like systems programming. Isolate heavy compute into dedicated Web Workers, guard GPU memory allocations behind explicit user intent, and cache compiled model weights locally.

Architecture and decisions: mine. Debugging sessions at odd hours: mine. AI assistance: structure, syntax, first draft. — Sachin