Hugging Face unlocks async continuous batching
hugging face introduced asynchronous batching to eliminate cpu/gpu wait times, achieving a 24% speedup in inference.
synchronous continuous batching forces cpus and gpus to take turns, creating idle gaps where expensive hardware sits unused. hugging face profiled an 8b model generating 8k tokens and found gpus were idle 24% of the time while waiting for cpu batch preparation. by implementing asynchronous batching, the cpu prepares batch n+1 while the gpu computes batch n, removing this bottleneck. this optimization requires no new model kernels or changes, just better hardware coordination, effectively reducing latency and improving utilization of expensive inference endpoints.