A small language model does not fit on a Raspberry Pi. TinyLlama-1.1B needs about 4.4 GB of memory. A Raspberry Pi 4 has 4 GB, and the operating system takes some of it. So before I could ask anything about latency, I had to answer a simpler question: how do you run a model that no single device can hold? You split it across several devices and pay for it over the network. This article is about what that payment costs and how I lowered it.

The setup was five Raspberry Pi 4 Model B boards, 4 GB of RAM each, a Cortex-A72 at 1.8 GHz, on 802.11ac WiFi through a dedicated access point. One board is the controller. Four are workers. Measured round-trip time between boards was 8 to 12 ms.

Where the time goes

Before LLMs I built a testbed for ordinary DNN partitioning: AlexNet-class models on CIFAR-10, split at layer boundaries across Raspberry Pis talking to a base station over WiFi. It profiled per-layer latency, CPU and memory on every device, measured link latencies, and fed all of it to a few optimisers. It also had tc-based latency injection and INA260 power sensors. We demonstrated it at ACM MobiHoc 2024.

That testbed taught me one thing above all: you cannot reason about a partition without measured link latencies. A layer's compute profile is half the story. The other half is how long an activation takes to cross from one Pi to the next, and on WiFi that number moves.

The second lesson came from simulation. Decoder-only transformers generate one token at a time, and each token appends keys and values to a per-head KV cache. Memory grows for as long as you generate. For our WiOpt 2025 paper we built a discrete-event simulator of a decoder layer across 25 heterogeneous devices. By 1000 tokens, a layer-based EdgeShard-style placement passed 1000 seconds of latency and Galaxy sat around 400 to 600 seconds, while head-level placement with dynamic migration stayed under 200. With 3 to 5 devices, our heuristic was within 15 to 20 percent of an exact solver.

I want to be honest about those numbers. The simulator assumed 1 to 10 Gbps links. WiFi between Pis is nothing like that. The simulation said head-level partitioning was worth pursuing. It did not say what I would get on hardware.

Splitting by heads instead of layers

The standard split is layer-wise: layer 1 on device 1, layer 2 on device 2, and so on. It is simple. It also means one device computes while the others wait. And if one layer does not fit the smallest device, it does not help at all.

Attention heads give another cut. The heads in a layer are independent, and each owns its slice of the KV cache. So each head can be a unit of placement, and every device can work on the same layer at once. Head outputs are gathered at the device holding the projection, and the result is broadcast to the next layer's devices.

controller (Raspberry Pi 4)embeddings, LM head, placement,speculation decisions, token loopworker 1heads, projection, FFN4 GB, Cortex-A72worker 2heads, projection, FFN4 GB, Cortex-A72worker 3heads, projection, FFN4 GB, Cortex-A72worker 4heads, projection, FFN4 GB, Cortex-A72PyTorch RPC over 802.11ac, 8 to 12 ms RTT, retries for jitterhead outputs gathered at the projection device, result broadcast to the next layer's devices
The testbed. Five Raspberry Pi 4 boards on a dedicated 802.11ac access point. The controller owns the token loop and the placement; the workers hold attention heads, projections and FFN blocks and never load the full model.
layer-wisedevice 1device 2device 3device 4layer 1idleidleidlelayer 2idleidleidlelayer 3idleidleidlelayer 4idleidleidletimehead-leveldevice 1device 2device 3device 4FFNFFNFFNFFNtimeattention heads (layer-wise: whole layer)feed-forward network, one device per layerper-layer attention for TinyLlama: 38 ms on one device, 14 ms across four
Layer-wise versus head-level. With one layer per device, three of four devices idle at any moment. With heads spread across devices every layer's attention runs in parallel, and only the FFN remains on a single device.

On the testbed, per-layer attention for TinyLlama dropped from 38 ms on one device to 14 ms across four. End to end, head-level partitioning alone gave 1.24 to 1.31 times speedup over layer-wise.

Then the bottleneck moved. Layer-wise, attention took about 55 percent of per-layer time and the feed-forward network 35 percent. After distributing heads, attention fell to 24 percent and the FFN rose to 48 percent. The FFN still ran on one device while three waited. Parallelising the fast part made the slow part obvious.

layer-wiseattention 55%FFN 35%head-levelattention 24%FFN 48%attentionfeed-forward networkcommunication and otherparallelising attention moved the bottleneck to the FFN, which speculation then overlaps
Where the time went. Spreading heads cut attention from 55 to 24 percent of per-layer time, which made the FFN the largest share. The next step had to address the FFN.

Speculating across layers

The fix came from the residual structure. After attention and projection, a layer has an intermediate output Z. The final output is X = Z + FFN(Z). The FFN term is additive and usually small, so Z already approximates X well.

HeadSplit forwards Z to the next layer's devices as soon as it exists. They start the next attention on it while the FFN runs elsewhere. When the FFN finishes, the controller compares the FFN output norm to the norm of Z. Below a tolerance of 0.05, the speculative attention is kept. Otherwise it is re-executed with the exact input, so the output never drifts. With tolerance zero the speculative path is bit-for-bit identical to exact execution.

layer llayer l+1attentionprojectionFFN(Z) on device AZ = X_prev + proj(A) availableattention on Z (speculative)projectionFFN done: ||FFN(Z)||/||Z|| < 0.05 ?yes: keep (84% of transitions)no: re-run attention on exact Xoverlap hides 70 to 85% of the FFN timetime within one token
Inter-layer speculation inside one token. Z is available before the FFN runs. The next layer starts its attention on Z on the other devices while the FFN finishes; the controller then checks the FFN contribution against Z and re-runs attention on the exact input only when the check fails.

This is not speculative decoding. There is no draft model and no guessing of tokens. It pipelines consecutive layers inside a single token's forward pass, on compute that would otherwise sit idle.

Not every layer deserves it. We profile offline on WikiText-103 and record how often the test passes at each layer transition. A layer speculates only if its acceptance probability is at least 0.7 and the expected saving is positive. For TinyLlama, layers 1 to 20 had acceptance of at least 0.82, average 0.87. Layers 21 and 22 were below 0.60, because the final layers' FFN contributions are larger, so they run exactly. Overall acceptance was 84 percent.

The threshold is a trade-off, not a knob to tighten. At 0.02, acceptance rose to 91 percent but speedup fell to 1.48 times because of re-execution. At 0.10 it fell to 1.51 times. We kept 0.05.

The two ideas are coupled. Head partitioning creates the idle window that speculation fills. Speculation changes the placement target: a speculating layer's attention only needs to finish within the previous layer's FFN time, so heads can be spread for memory balance instead of minimum attention time. The greedy placement in our SPAWC 2026 paper places FFN and projection first, then heads, in under 5 ms. It matters: 256 tokens took 40.0 s with greedy placement, 44.8 s with round-robin and 62.4 s layer-wise. The full system reached 1.54 to 1.67 times speedup over layer-wise, and 1.42 times on GPT-2 Small.

layer-wise (baseline)62.4 sheads + speculation, random placement48.2 sheads + speculation, round-robin44.8 sheads + speculation, greedy placement40.0 sseconds for 256 tokens, lower is better; greedy is 1.56x the baseline
Placement matters. Time to generate 256 tokens with TinyLlama-1.1B on four workers. The same head-level system with speculation varies by a fifth depending on where the blocks land; the greedy placement is computed in under 5 ms.

What the speedup hides

Head-level partitioning moves four times more data than layer-wise: 52.8 MB versus 13.2 MB for 256 tokens. Per-device power rose 26 percent, from 3.8 W idle to 4.8 to 5.0 W active. Total energy still fell 6 percent because the job finished sooner, and the energy-delay product improved 47 percent. I would not have believed that without the INA260 boards on every Pi.

Memory needs active management. At 256 tokens the KV cache is about 88 MB across 22 layers. At 512 it doubles. The controller re-runs placement whenever a device crosses 85 percent of memory, about 1.4 times per 512-token run, each in under 5 ms. Without it, a static placement ran out of memory beyond roughly 400 tokens.

WiFi needs margins. We measure round-trip times at startup, add 15 percent, and retry RPCs when a link stalls. None of that is clever. All of it is necessary.

What still does not work well

The absolute numbers are slow. At 256 tokens we went from 4.1 to 6.4 tokens per second. That is useful for an on-device assistant, not a replacement for a GPU.

The approach needs at least three devices and round-trip time under about 25 ms. Beyond that, gather and broadcast cost more than the parallelism saves, and layer-wise is the better choice. Initial measurements kept at least 1.35 times speedup up to 15 ms of added RTT, but the full robustness study is still ahead of us.

Grouped-query attention is a problem. Llama-3-style models share one KV head among several query heads, leaving fewer independent units to spread. Quantised models change the compute-to-communication ratio and we have not measured them. Per-layer speculation decisions are fixed at profiling time; adapting them at runtime is open work.

What I keep coming back to is simple. On constrained hardware, the model's structure is the only free resource you have. Layers hide the parallelism inside them. Heads expose it. Residual connections let you start early and check later. Everything else is measuring where the time goes and refusing to guess.

The code is on GitHub: HeadSplit, the simulator and the DNN testbed. The papers are on the publications page.