Hugging Face 推出異步連續批處理
hugging face 引入了異步批處理以消除 cpu/gpu 等待時間，實現了 24% 的推理加速。
synchronous continuous batching forces cpus and gpus to take turns, creating idle gaps where expensive hardware sits unused. hugging face profiled an 8b model generating 8k tokens and found gpus were idle 24% of the time while waiting for cpu batch preparation. by implementing asynchronous batching, the cpu prepares batch n+1 while the gpu computes batch n, removing this bottleneck. this optimization requires no new model kernels or changes, just better hardware coordination, effectively reducing latency and improving utilization of expensive inference endpoints.