Hugging Face releases VAKRA benchmark for tool use agents
hugging face released vakra, a benchmark for evaluating ai agent reasoning over 8,000+ locally hosted apis. the dataset includes 2,077 slot-bird instances and 1,597 rest-bird instances requiring 3-7 step chains. early analysis shows poor model performance on these multi-step enterprise workflows.
Hugging Face released VAKRA, a tool-grounded benchmark for evaluating AI agents in enterprise environments. The framework tests compositional reasoning by requiring agents to chain 3-7 steps across over 8,000 locally hosted APIs spanning 62 domains, supported by real databases. The dataset comprises 2,077 instances using SLOT-BIRD and SEL-BIRD collections for data manipulation and 1,597 instances using REST-BIRD collections for endpoint-style interactions. Analysis indicates current models perform poorly on these multi-step workflows.