Rethinking LLM Evaluation with 3C3H: AraGen Benchmark and Leaderboard

Source: HuggingFace Date: 2024-12-04 URL: https://huggingface.co/blog/leaderboard-3c3h-aragen

Summary

Research benchmark release: AraGen, a dynamic leaderboard for Arabic LLMs using the 3C3H evaluation framework (Correctness, Completeness, Conciseness, Helpfulness, Honesty, Harmlessness). 279 human-verified questions across 4 tasks with 3-month blind testing cycles to prevent benchmark gaming. Judge selection study identified Claude-3.5-sonnet as most consistent LLM judge (lowest std dev; 100% guideline adherence). GPT-4o showed significantly higher variance across runs than Claude or Jury systems.

Implications

Open-weights ecosystem health. The rotating blind test set approach (dataset private during evaluation window, released after) is a practical solution to benchmark contamination that has not been widely adopted. If it works for AraGen, it could become the model for evaluating other rapidly-improving model families where contamination risk is high.

Model release cadence — regional models. AraGen joining Alyah as a structured Arabic LLM evaluation signal the maturation of the Arabic AI community: not just releasing models but building the evaluation infrastructure to validate them. The Claude-3.5-sonnet judge finding (outperforming GPT-4o on consistency) is also a concrete third-party data point for Claude’s reliability on structured evaluation tasks.