Hugging Face adds private datasets to ASR leaderboard
hugging face added private english asr datasets from appen and dataoceanai to its open asr leaderboard to combat benchmaxxing. the public average wer remains unchanged, but users can toggle private data to view new metrics. since launching in september 2023, the leaderboard has recorded over 710k visits.
Hugging Face has introduced high-quality private English ASR datasets, developed in collaboration with Appen Inc. and DataoceanAI, to the Open ASR Leaderboard. These datasets cover scripted and conversational speech across multiple accents and remain private to prevent test-set contamination and benchmark-specific optimization (benchmaxxing).
The leaderboard's default Average Word Error Rate (WER) continues to calculate based solely on public datasets, though a new toggle allows users to view results including the private data. This update aims to provide a more robust measure of model performance by exposing gaps between saturated scripted settings and more nuanced conversational scenarios.