A Deepdive into Aya Expanse: Advancing the Frontier of Multilinguality

Source: HuggingFace Date: 2024-10-24 URL: https://huggingface.co/blog/aya-expanse

Summary

Model release and research summary: Cohere For AI’s Aya Expanse 8B and 32B — multilingual open-weights instruction models covering 23 languages, built on four algorithmic innovations: Data Arbitrage (multi-teacher selection via reward model), multilingual preference training (3-round online DPO), model merging across language family clusters, and an end-to-end unified pipeline. Aya Expanse 8B win rates of 60.4-70.6% vs Gemma 2 9B/Llama 3.1 8B/Ministral 8B; 32B outperforms Llama 3.1 70B (2x its size). Evaluated on Arena-Hard-Auto translated to 23 languages (dataset released publicly).

Implications

Open-weights ecosystem health. Aya Expanse’s 32B model beating Llama 3.1 70B on multilingual evals while being half the size is a strong efficiency claim — the model merging across language family clusters is the technique most worth watching as a path to multilingual capability without proportionally increased scale.

Model release cadence (regional/multilingual). The multilingual AI frontier is increasingly driven by academic and non-commercial labs (Cohere For AI, AI4Bharat, MBZUAI) rather than US labs. Aya Expanse releasing the translated evaluation dataset publicly raises the bar for multilingual benchmark reproducibility at a time when English-centric evals dominate the conversation.