LLM Systems

Streaming a 26B MoE From SSD on a Jetson

A single request on an 8 GB Jetson Orin Nano, with the model's experts coming off SSD mid-inference. Scrub the run behind the blog post's GIF and watch GPU load spike as each expert read lands, or flip to the bench numbers.

moe-streamllama-cppmixture-of-expertsjetson-orinllm-inferenceedge-ai

LiveInteractive · drag, toggle, run it

LLM Systems · MoE Streaming

Streaming a 26B MoE From SSD on a Jetson

This is the exact run behind the GIF in the blog post: one streamed completion against the production gemma4-26b roster entry on a Jetson Orin Nano, with tegrastats sampling GPU load and power draw alongside every token as it arrived. Step or play through it below.

3.7s

elapsed

1/115

token

8.00

cumulative tok/s

22%

GPU load

7.55 W

power draw

GPU load

Power draw (VDD_IN)

Cumulative tok/s

token 1 / 115

Ground truth, from llama.cpp itself

The tok/s trace above is derived from token arrival times measured on this side of the HTTP connection. llama-server's own final timings block is the cross-check: 115 tokens predicted at 2.09 tok/s, prompt eval at 1.43 tok/s over 5 prompt tokens. That is the same run, measured from inside the engine instead of from the client.

Where this comes from: the architecture, the llama.cpp patch, and the full write-up are on the blog: Running a 26B MoE on an 8 GB Jetson by streaming experts from SSD.