hugging face發現微調作業的flop估計值誤差數小時
hugging face分析了272個生產環境中的微調作業,發現基於flop的估計值與實際時長相差甚遠,而學習預測器可以達到r² = 0.98。
hugging face researchers analyzed 272 real supervised fine-tuning jobs from their production pool of nvidia h200 gpus, totaling about 7,382 gpu-hours. they compared analytical estimates based on flops against learned predictors. the usual flop-based estimate explains little of what was observed, with r² = 0.31 and a mean absolute error of 12.41 hours. in contrast, a catboost regressor that only sees information available before job dispatch achieved r² = 0.98, with a mean absolute error of 1.71 hours and a median error of 0.51 hours in 5-fold cross-validation. this gap arises from the regressor's ability to see more training-specific signals and learn nonlinear interactions between them.