Hugging Face 推出异步连续批处理
hugging face 引入了异步批处理以消除 cpu/gpu 等待时间，实现了 24% 的推理加速。
synchronous continuous batching forces cpus and gpus to take turns, creating idle gaps where expensive hardware sits unused. hugging face profiled an 8b model generating 8k tokens and found gpus were idle 24% of the time while waiting for cpu batch preparation. by implementing asynchronous batching, the cpu prepares batch n+1 while the gpu computes batch n, removing this bottleneck. this optimization requires no new model kernels or changes, just better hardware coordination, effectively reducing latency and improving utilization of expensive inference endpoints.