Hugging Face finds FLOP estimates for fine-tuning jobs off by hours
hugging face analyzed 272 production fine-tuning jobs on nvidia h200 gpus, finding flop-based estimates explain little of observed duration, while a learned predictor achieves r² = 0.98.
hugging face researchers analyzed 272 real supervised fine-tuning jobs from their production pool of nvidia h200 gpus, totaling about 7,382 gpu-hours. they compared analytical estimates based on flops against learned predictors. the usual flop-based estimate explains little of what was observed, with r² = 0.31 and a mean absolute error of 12.41 hours. in contrast, a catboost regressor that only sees information available before job dispatch achieved r² = 0.98, with a mean absolute error of 1.71 hours and a median error of 0.51 hours in 5-fold cross-validation. this gap arises from the regressor's ability to see more training-specific signals and learn nonlinear interactions between them.