Hugging Face releases EVA framework for voice agents
hugging face introduced eva, a framework that evaluates voice agents on both task accuracy and conversational experience using a bot-to-bot architecture. benchmarking 20 systems revealed a consistent accuracy-experience tradeoff. the release includes an airline dataset with 50 scenarios.
Hugging Face introduced EVA, an end-to-end evaluation framework for conversational voice agents. The framework employs a bot-to-bot architecture to evaluate complete, multi-turn spoken conversations, producing two distinct scores: EVA-A (Accuracy) and EVA-X (Experience). The researchers released an initial airline dataset containing 50 scenarios and published benchmarks for 20 systems, including audio-native models and cascades. The study identified a consistent trade-off, finding that agents achieving high task completion accuracy often deliver a poor user experience, and vice versa.