Hugging Face policy chief doubts embedded AI evaluators' effectiveness

hugging face's policy chief irene solaiman discussed the challenges of embedded ai evaluators at techcrunch disrupt. she highlighted issues with reproducibility, citing the need for open weights and access to models for evaluation. solaiman also noted that evaluations are evolving and not yet comparable. the industry's push for embedded evaluators aims to address safety concerns, but doubts remain about their effectiveness. a more uniform approach to benchmarking may be necessary for the industry to rely on evaluations.