Hugging Face, 비동기 연속 배칭 기능 공개
Hugging Face는 비동기 배칭을 도입해 CPU/GPU 대기 시간을 없애고, 추론 속도를 24% 향상시켰습니다.
synchronous continuous batching forces cpus and gpus to take turns, creating idle gaps where expensive hardware sits unused. hugging face profiled an 8b model generating 8k tokens and found gpus were idle 24% of the time while waiting for cpu batch preparation. by implementing asynchronous batching, the cpu prepares batch n+1 while the gpu computes batch n, removing this bottleneck. this optimization requires no new model kernels or changes, just better hardware coordination, effectively reducing latency and improving utilization of expensive inference endpoints.