hugging face发现微调作业的flop估计值误差数小时
hugging face分析了272个生产环境中的微调作业,发现基于flop的估计值与实际时长相差甚远,而学习预测器可以达到r² = 0.98。
hugging face researchers analyzed 272 real supervised fine-tuning jobs from their production pool of nvidia h200 gpus, totaling about 7,382 gpu-hours. they compared analytical estimates based on flops against learned predictors. the usual flop-based estimate explains little of what was observed, with r² = 0.31 and a mean absolute error of 12.41 hours. in contrast, a catboost regressor that only sees information available before job dispatch achieved r² = 0.98, with a mean absolute error of 1.71 hours and a median error of 0.51 hours in 5-fold cross-validation. this gap arises from the regressor's ability to see more training-specific signals and learn nonlinear interactions between them.