Benchmarking Language Model Performance on 5th Gen Xeon at GCP

Source: HuggingFace Date: 2024-12-17 URL: https://huggingface.co/blog/intel-gcp-c4

Summary

Benchmark analysis: Intel 5th-gen Xeon (Emerald Rapids) with AMX acceleration on GCP C4 instances vs. 3rd-gen Xeon (Ice Lake, N2) for CPU-only AI inference. Text embedding (UAE-Large-V1): C4 delivers 10–24x higher throughput and 7–19x better TCO despite 1.3x higher hourly cost. Text generation (Llama 3.2 3B): 2.3–3.6x higher throughput, 1.7–2.9x TCO advantage; at batch sizes 1–16, 13x throughput improvement. Main argument: modern Intel Xeon CPUs can run lightweight agentic workloads end-to-end without GPUs.

Implications

Open-weights ecosystem health. A 13x throughput improvement for text generation at low batch sizes on CPU is a significant data point for the “GPU-free inference” narrative. Sub-3B models like Llama 3.2 3B become economically viable for high-throughput CPU deployment if AMX acceleration is available, which broadens the deployment surface to CPU-only cloud instances and bare-metal servers without GPU allocation.

Model release cadence — hardware thread. As CPU inference becomes more capable (AMX, SDOT, ExecuTorch), model releases targeting the 1B–3B size class will increasingly be measured against CPU deployment targets, not just GPU benchmarks. The Intel/GCP/HF collaboration pattern mirrors the Arm/ExecuTorch investments — multiple hardware vendors are competing to be the CPU inference standard for open-weights models.