Core dump epidemiology: fixing an 18-year-old bug

Source: OpenAI Date: 2026-06-30 URL: https://openai.com/index/core-dump-epidemiology-data-infrastructure-bug

Summary

An OpenAI infrastructure postmortem on tracking down rare, seemingly-random crashes across a large fleet by treating crash core dumps like an epidemiological dataset — aggregating a year’s worth of dumps and clustering them by symptom rather than debugging incidents one at a time. The population-level approach split what looked like a single mystery into two unrelated root causes: silent hardware corruption on one bad Azure host (the CPU miscomputing) and an 18-year-old race condition in GNU libunwind, a widely-used open-source stack-unwinding library.

Implications

• Dev tooling thread. The interesting move isn’t the bug itself but the method: case-by-case debugging stalled for months, and only population-scale statistical analysis of crash data surfaced the two distinct clusters. That’s a reusable pattern for any org running large fleets — treat rare failures as a distribution to characterize, not individual tickets to close.
• AI-industry economics thread. Also a quiet signal of scale: this kind of crash-clustering work only pays off once a fleet is big enough to generate a statistically meaningful population of rare-failure core dumps, which says something about the size of OpenAI’s serving infrastructure.
• Secondary note: the 18-year-old bug lived in libunwind, a dependency countless other systems share — a reminder that foundational open-source infrastructure keeps carrying latent bugs discovered only at extreme scale.