🇨🇿 BenCzechMark - Can your LLM Understand Czech?

Source: HuggingFace Date: 2024-10-01 URL: https://huggingface.co/blog/benczechmark

Summary

Research summary and benchmark release: BenCzechMark, the first comprehensive Czech LLM evaluation suite, covering 50 tasks across 9 categories (reading comprehension, factual knowledge, Czech language understanding, math, NLI, NER, sentiment, document retrieval) with 90% native non-translated content. Uses a “Duel Win Score” via statistical significance testing rather than simple averaging. 26 open-source models evaluated: Llama-405B wins overall; Qwen-72B leads on math and IR; Gemma-2 9B outperforms larger models on Czech reading comprehension.

Implications

Model release cadence (regional models). BenCzechMark reveals that Czech-language performance doesn’t simply track English benchmark rankings — Gemma-2 9B beating much larger models on reading comprehension, and Aya-23-35B performing well on sentiment and language modeling, suggests multilingual capability is unevenly distributed in ways that English-only benchmarks completely miss.

Open-weights ecosystem health. A community-maintained Czech benchmark with a public leaderboard and open submission process is the infrastructure needed to actually track whether open-weights models improve at less-resourced European languages. The 90% native content ratio is the right design choice — translated benchmarks systematically underestimate difficulty for models that lack genuine multilingual training.