HeadSplit: Attention Head Partitioning and Inter-Layer Speculation for Distributed Inference at the Edge
A small LLM still exceeds one edge device's memory, and layer-wise splits leave all but one device idle at any moment. HeadSplit schedules each layer's attention heads across devices so attention runs in parallel everywhere, and forwards a speculative intermediate output to the next layer before the FFN finishes, filling the idle interval it creates.
BibTeX
@inproceedings{Kafetzis2026HeadSplit,
author = {Dimitrios Kafetzis and Iordanis Koutsopoulos},
title = {HeadSplit: Attention Head Partitioning and Inter-Layer Speculation for Distributed Inference at the Edge},
booktitle = {IEEE International Workshop on Signal Processing and Artificial Intelligence for Wireless Communications (SPAWC)},
year = {2026},
publisher = {IEEE}
}