Many AI features in web apps send the user's audio, images, or text to a server and wait for the answer. That architecture adds network latency, moves private data off the device, and stops working when the connection disappears, which is far from ideal. Transformers.js gives developers another option: run the model in the browser for tasks such as speech recognition, background removal, embeddings, and parts of an agent workflow.
Episode summary
Mike Grabowski talks with Nico Martin, Open Source ML Engineer at Hugging Face, about where browser AI works today and where the limits still are. They explain how Transformers.js wraps model preprocessing, inference, and output handling in JavaScript APIs, how ONNX Runtime and WebGPU execute models locally, and how model size, browser storage, and device capability affect production decisions.
The discussion covers text embeddings, background removal, audio transcription, speech detection, image processing, text generation, and browser-based agents. Nico also shares what Hugging Face is exploring next, including structured output and an early WebGPU inference engine that uses an agentic system to write optimized GPU kernels.
What Transformers.js does in the browser
Transformers.js lets web developers run pretrained machine learning models in JavaScript. Its name comes from Hugging Face's Python Transformers library, but the JavaScript library supports multiple model architectures and combinations of architectures.
The library covers natural language processing, computer vision, audio, and multimodal work. The episode discusses text embeddings for similarity search, automatic speech recognition, voice activity detection, image background removal, depth estimation, and text generation. Smaller task-specific models often fit browser constraints better than a general-purpose large language model.
How the pipeline API handles model input and output
Machine learning models consume tensors and return tensors. Web applications usually work with text, images, audio, or messages. The Transformers.js pipeline API connects those layers by handling task-specific preprocessing, model inference, and postprocessing. A developer selects a task, chooses a compatible model, and passes in familiar application data. The pipeline converts that input into tensors, runs the model, and turns its output back into a usable result.
TypeScript types describe the expected input and output for each pipeline. Developers can also use lower-level parts of Transformers.js when they need a custom pipeline or an architecture the library does not yet support.
How ONNX Runtime and WebGPU fit together
Transformers.js uses the Open Neural Network Exchange format, known as ONNX, for model graphs and weights. ONNX Runtime Web executes the graph in the browser. Separating the model format from the runtime lets the same Transformers.js pipeline target different execution backends.
WebGPU gives web applications access to the device's GPU for compute. Nico recommends it when available, while production code can detect support and select a CPU fallback through the pipeline's device option. Performance still depends on the model, its operations, and the user's hardware. The library also runs in Node.js, Deno, and Bun, though Nico explains that browser inference is the main reason the project exists. A browser application can reach users through a URL without requiring a separate installation.
Model downloads, browser caching, and device limits
The first run downloads the files required by the selected model, usually from the Hugging Face Hub, and stores them in a persistent browser cache. The size depends on the task. Nico gives examples that include a voice activity detection model of a few megabytes, speech transcription models in the hundreds of megabytes, and large language models that need several gigabytes.
Browser caches are isolated by origin. If two websites use the same model, each site currently downloads and stores its own copy. Nico describes experimental work around cross-origin storage that could reduce this duplication.
Hardware creates another constraint. A recent laptop may run a larger model at an acceptable speed, while an older phone or Windows laptop may take much longer. At the time of recording, Transformers.js did not provide a cross-browser helper for checking available memory. Heavy models need an explicit download step, and on-device mode works best as an informed opt-in when file size or device performance may affect the experience.
When browser AI can beat a server request
Local inference removes the network trip. That changes the performance calculation for inputs such as audio and images, where a server implementation must upload the source data before inference and return the result afterward.
Nico's speech-recognition example processes a 30- to 32-millisecond audio sample in about one millisecond on his setup. Background removal can also avoid uploading an image and downloading a new PNG. The local model may take longer to compute than a server model and still finish the full interaction sooner because the transfer disappears. Local processing can also keep privacy-sensitive input on the device and preserve features when the application is offline.
Cloud inference remains practical for models that are too large or too slow for a user's hardware. The episode treats local and cloud execution as parts of the same architecture, selected task by task.
Can AI agents run entirely in the browser?
Nico has built offline browser-agent demos around a local large language model with tool calling. The model selects a tool, the application runs the corresponding JavaScript function, and its result returns to the conversation. This agent loop can live in the browser.
He treats fully local agents as a demo case today because model downloads and inference speed vary too much across devices for customer-critical workflows. A hybrid agent can keep its main model in the cloud while the browser handles tool execution, speech transcription, or an optional local fallback. Smaller models, faster hardware, and better browser APIs can move more of that workflow onto the device over time.
What Hugging Face is working on next
Nico describes structured output for text generation, where decoding checks each token against a required JSON, regular expression, or XML structure. This gives applications more control over locally generated responses.
He also discusses early work on a Hugging Face inference engine for WebGPU. An agentic system writes custom GPU kernels, merges operations to reduce memory reads, and creates device-specific optimizations. Nico reports early speedups in the range of 5x to 10x, while stressing that the project is still at an early stage. On constrained hardware, those gains could turn a model that responds too slowly into one that is usable.
Cross-origin model storage could prevent repeated downloads, and WebNN could provide another route for executing model graphs. Those pieces depend on browser vendors and standards work as well as the Transformers.js library.
Resources
- Nico Martin on Hugging Face, GitHub, and his website
- Mike Grabowski on X, LinkedIn, and GitHub
- Transformers.js documentation
- Transformers.js repository
- Transformers.js WebGPU guide
- Models compatible with Transformers.js
- Transformers.js examples
- ONNX Runtime Web documentation
- Nico's Gemma 4 browser-agent demo
