Fine-tuning NVIDIA Cosmos Predict 2.5 with LoRA/DoRA for robotics
nvidia's 2b-parameter world model cosmos predict 2.5 can now be fine-tuned using lora and dora on a single 80gb gpu to generate synthetic robot trajectories.
nvidia published a technical guide for fine-tuning its cosmos predict 2.5 world model using low-rank adaptation (lora) and weight-decomposed low-rank adaptation (dora) via the diffusers library. by freezing the base vae, text encoder, and dit weights, the technique reduces memory consumption significantly, enabling training on a single 80gb gpu or faster iteration on 8 h100s. the model uses rectified flow to train on noise prediction, generating synthetic videos for pick-and-place tasks from a dataset of 92 training videos. this enables scalable generation of robot demonstration data, mitigating the high cost and slow speed of collecting real-world physical trajectories.