INTERACTIVE · CLIENT-SIDE INFERENCE · TRANSFORMERS.JS · WEBGPU / WASM

Vision Lab.

METHOD

CV INFERENCEEMBEDDINGS

The browser is the inference runtime. Real computer-vision models run with no server, no API, and no data ever leaving the device — because at this model size, they finally can.

Pick a scene or drop your own image, then run zero-shot classification with CLIP or estimate per-pixel depth. The weights download once from the Hugging Face hub and cache in your browser; every inference after that is a local matrix multiply on your own hardware.

2
models · CLIP + Depth Anything: ~173 MB
one-time download, then cached: 0
bytes of your image uploaded: 100%
on-device inference

PROBLEM

The default assumption is that machine learning needs a backend: a GPU server, an inference API, per-call cost, and user data shipped off-device. That assumption quietly decides who can ship ML and what data it is allowed to touch.

APPROACH

Take two real models, compile them to ONNX, and run them in the tab via transformers.js on ONNX Runtime Web — WebGPU where the browser exposes it, WebAssembly (CPU) otherwise. Weights fetch once from the Hugging Face hub and are cached; nothing else touches the network.

THE INSIGHT

Running the model in the browser inverts the usual privacy/infra tradeoff: no server, no API cost, no data egress. At this model size, the classic reasons you cannot ship ML to users mostly dissolve.

THE INSTRUMENT

Try it first. The first run downloads and warms the model — after that, switching tasks or re-running is instant.

01HOW IT WORKS

Both tasks share one pipeline and diverge only at the very end. The page pulls transformers.js at runtime, compiles each model to ONNX Runtime Web, and executes it on WebGPU when the browser exposes it — falling back transparently to a WebAssembly (CPU) backend otherwise. The status bar shows the target backend before the first run and the confirmed one after.

The pipeline · an image is preprocessed to a tensor, run through ONNX Runtime Web on WebGPU (WASM fallback), and post-processed into a bar chart or depth heatmap. The only thing ever fetched from the network is the model weights — once, then cached. Your image never leaves the tab.

Runtime transformers.js → ONNX Runtime Web (WebGPU / WASM)

Classify CLIP ViT-B/32 · zero-shot image ↔ text similarity · ~147 MB (q8)

Depth Depth Anything (small) · monocular relative depth · ~26 MB (q8)

Data path image → tensor → model → result, all in-tab · nothing leaves the device

Progress is a single aggregated bar weighted by byte totals across every file the model needs, so the load reads as one smooth number rather than a flickering per-file count. Sample scenes are generated procedurally on a canvas in your browser — there are no image assets to license or download.

02WHY IN-BROWSER

The tradeoff inversion is not abstract. Each of the three constraints that normally block shipping ML to users — privacy, cost, and latency — flips the moment the runtime is the client.

PRIVACY BY DEFAULT

Pixels never leave the device. For medical, legal, or personal imagery that alone can be the difference between a shippable feature and a compliance non-starter.

ZERO INFRASTRUCTURE

No inference server, no GPU fleet, no per-call cost, no cold starts. The client is the runtime — it scales to every visitor for the price of a static file host.

EDGE + OFFLINE

Once cached, the model runs with no network at all. Latency is a local matrix multiply, not a request — the difference between an interactive tool and a laggy API.

03LIMITATIONS + WHAT'S NEXT

Moving inference to the client is a genuine trade, not a free lunch. The honest edges:

Cold start is heavy

The models are not small: CLIP quantized to q8 is ~147 MB, Depth Anything (small) ~26 MB — about 173 MB the first time. That download is real; the browser caches the weights so the second task and every later visit are instant, but a first-time visitor on a slow mobile link waits.

First inference lags, then it's fast

The very first run also compiles shaders and warms the runtime, so it is slow. Subsequent runs are a local matrix multiply, fast especially on WebGPU.

WebGPU is not everywhere

On browsers without WebGPU the page falls back to the WASM (CPU) backend — correct results, just slower. GPU init can also fail on some drivers; that path degrades to WASM rather than erroring. The status bar reports the backend actually in use, so the claim is never overstated.

Depth is relative, not metric

Depth Anything predicts ordinal depth — nearer vs. farther — not meters. It is useful for masking, parallax, and compositing; it is not a measurement.

q8 trades a little accuracy for bytes

8-bit weights keep the download small and run on both backends; full-precision would score marginally better at 3–4× the size. For a demo the trade is clearly worth it.

Samples are synthetic on purpose

The built-in scenes are drawn on a canvas with deliberate depth cues — clean licensing, and honest about being generated. Upload a real photo to see both models on natural imagery.

What's next: caching the weights behind a service worker for true offline use, an fp16-on-WebGPU quality toggle for machines that can afford it, and a third head — open-vocabulary detection — that reuses the exact same pipeline up to the postprocess step.

04WHAT THIS DEMONSTRATES

Shipping ML to the edge is its own discipline: quantizing models small enough to download, negotiating heterogeneous execution providers, streaming load progress, degrading gracefully when a backend is missing, and keeping the whole thing legible as an interface. This page is a compact proof of all of that in one artifact.

The zero-shot task is also a small window into work I care about. CLIP projects images and text into a shared latent space; the bar chart is really a ranked list of cosine similarities in that space. The same idea — treating embeddings as a navigable geometry — drives my latent-space project view and my MSCD thesis. Here it's turned outward, so anyone can type their own labels and watch the geometry respond.