Battle Bots: Six AI Chatbots Face Off to Debug an Issue Building in Bolt.New

Source: Nate’s Newsletter Date: 2025-01-26 URL: https://natesnewsletter.substack.com/p/battle-bots-six-ai-chatbots-face

Summary

Nate tests six LLMs on a single OAuth debugging scenario in Bolt.new, arguing that AI app-building platforms like Bolt.new, Lovable.dev, and Cursor overpromise simplicity but fail when non-engineers hit real technical obstacles. “Doom loops” emerge from limited context windows, outdated training data, and insufficient technical depth — the marketing claim that building is “as easy as a command” doesn’t hold under pressure.

Implications

Vendor positioning thread. The gap between “build an app with a prompt” marketing and actual debugging capability is a credibility problem that will surface as non-technical users hit real obstacles at scale. Vendors that set accurate expectations will retain users; those that oversell will churn them.

Agent product strategy thread. Context window limits and outdated training data are the specific technical failure modes Nate identifies — these are architectural constraints, not UX issues. Agent products targeting non-technical builders need to address these at the architecture level, not just through better error messages.

Labor displacement thread. “False confidence among non-engineers” is the displacement risk — tools that make non-technical users believe they can replace developers without actually delivering on that promise create organizational risk when the limits are discovered.

Watch: Whether vibe-coding tool quality improves enough by 2026 to actually deliver on the “anyone can build” promise, or whether the non-technical builder segment remains underserved.