Hugging Face、非同期連続バッチングを実現
Hugging FaceはCPU/GPUの待ち時間を解消する非同期バッチングを導入し、推論を24%高速化しました。
synchronous continuous batching forces cpus and gpus to take turns, creating idle gaps where expensive hardware sits unused. hugging face profiled an 8b model generating 8k tokens and found gpus were idle 24% of the time while waiting for cpu batch preparation. by implementing asynchronous batching, the cpu prepares batch n+1 while the gpu computes batch n, removing this bottleneck. this optimization requires no new model kernels or changes, just better hardware coordination, effectively reducing latency and improving utilization of expensive inference endpoints.