Thinking with images

Source: OpenAI Date: 2025-04-16 URL: https://openai.com/index/thinking-with-images

Summary

Summary

OpenAI published research and a blog post on “thinking with images” — describing how o-series reasoning models use visual representations (diagrams, charts, intermediate visual states) during chain-of-thought reasoning, rather than reasoning entirely in text. The April 2025 timing coincides with the o3/o4-mini launch, where multimodal reasoning was a prominent capability improvement.

Implications

Research/multimodal thread. Thinking with images represents a qualitative shift in how reasoning models interact with visual information — from treating images as inputs to be described in text, to using visual representations as working memory during reasoning. This has significant implications for tasks involving diagrams, mathematical notation, spatial reasoning, and visual problem-solving where textual encoding loses information. If the approach generalizes, it suggests a future where model reasoning happens in a mixed symbolic-visual space, rather than purely in language. This is one of the more theoretically interesting research directions from the o-series development period.