TRL implements delta weight sync for cheaper async RL
trl merged a pr enabling delta weight sync, reducing the per-step payload for qwen3-0.6b from 1.2 gb to 20-35 mb by uploading only changed weights.
hugging face merged a pull request for trl that introduces sparse delta weight synchronization, addressing the bandwidth bottleneck in async rl. by identifying that 98-99% of bf16 weights remain bit-identical between optimizer steps, the new implementation uploads only the modified weights as a sparse safetensors file. testing on qwen3-0.6b reduced the per-step payload from 1.2 gb to 20-35 mb, enabling disaggregated training where the trainer and inference engine operate on separate infrastructure without shared clusters or rdma.