QIMMA validates Arabic LLM benchmarks via automated and human pipelines
researchers released qimma, a new arabic llm leaderboard that uses a rigorous quality validation pipeline. tests run qwen3-235b and deepseek-v3 to filter 52,000 samples from 14 benchmarks, removing low-quality data before evaluation. the suite includes arabic-adapted humaneval+ for code assessment and focuses on 99% native content.
QIMMA introduces a quality-first approach to Arabic Large Language Model evaluation, consolidating 109 subsets from 14 benchmarks into a unified suite of over 52,000 samples. Unlike existing leaderboards, QIMMA employs a two-stage validation pipeline utilizing Qwen3-235B-A22B-Instruct and DeepSeek-V3-671B to filter data, followed by human review from native speakers to correct translation errors and cultural misalignments. The resulting benchmarks cover seven domains, including a first-of-its-kind Arabic code evaluation using adapted HumanEval+ and MBPP+ datasets.