MolmoMotion: Language-guided 3D motion forecasting

Source: HuggingFace Date: 2026-06-17 URL: https://huggingface.co/blog/allenai/molmomotion

Summary

Allen AI’s MolmoMotion is a language-guided 3D motion forecasting model built on the Molmo 2 vision-language backbone. It takes a video frame, marked 3D points on an object, and a text action description, and predicts where those points will move — outputting object-attached trajectories in world space. Two variants ship: autoregressive for deterministic paths, flow-matching for uncertain multi-outcome scenarios. Trained on MolmoMotion-1M (1.16M annotated videos, 736 motion types), it improves robotic pick-and-place success from 56% to 76.3% in simulation and improves motion quality in video generation.

Implications

This feeds the native-tooling speed thread and the agent-form/proactivity thread:

• Language as the robot API. The key move here is using natural language action descriptions as the conditioning signal for physical trajectory prediction. That closes a gap between instruction-following agents and physical-world execution — the same language interface that drives text agents can now parameterize robotic motion.
• Open weights, world-space grounding. MolmoMotion ships open (Allen AI), on top of an open backbone (Molmo 2), with an open benchmark (PointMotionBench). That puts this capability class directly on the open-weight track of the capability clock — closed-lab robotics teams are not the only ones with path-from-video-to-motion.
• Downstream: video generation and simulation. The improvement to video generation motion quality is a side-effect that matters for agentic pipelines that use video synthesis as a planning or simulation substrate.