New Research: The Harder the Problem, the Dumber the Model

Apple researchers found that AI's latest "thinking" models—like OpenAI's o1 and Claude's reasoning variants—collapse completely when problems cross certain complexity thresholds. More surprising: these models actually think less as problems get harder, despite having computational power to spare.

The study tested models like DeepSeek-R1, Claude 3.7 Sonnet Thinking, and OpenAI's o3-mini on controlled puzzles instead of traditional math tests. Unlike established benchmarks that suffer from data contamination, these puzzle environments let researchers manipulate complexity precisely while tracking each step of the AI's reasoning process.

The results reveal troubling gaps in current AI reasoning capabilities. These models, trained with reinforcement learning to generate detailed thought processes before answering, still fail to develop true problem-solving skills. When complexity crosses a threshold, accuracy drops to zero across all tested models.

Three zones of performance

The researchers identified three distinct performance zones that challenge assumptions about AI reasoning capabilities.

In simple problems, standard language models often outperform their "thinking" counterparts. The extra reasoning steps waste computational resources without improving results. For moderate complexity, thinking models gain an edge by working through problems step-by-step. But at high complexity, both model types fail completely.

This pattern breaks conventional wisdom about scaling AI capabilities. More thinking doesn't always mean better results—sometimes it makes things worse.

The algorithm paradox

Even when researchers gave complete solution algorithms, the models still failed at identical complexity thresholds. This suggests the problem isn't finding solutions but executing logical steps consistently.

Consider the Tower of Hanoi puzzle. When given the exact recursive algorithm to solve it, models performed no better than when working from scratch. They couldn't follow their own prescribed steps, revealing gaps in logical execution rather than creative problem-solving.

Reasoning effort drops when needed most

The most counterintuitive finding: models reduce their reasoning tokens as problems become more complex, exactly when more thinking should help. This happens well before hitting computational limits.

For simple Tower of Hanoi puzzles, Claude 3.7 Sonnet Thinking might use 20,000 tokens. But for harder versions requiring more moves, it drops to 5,000 tokens despite having 64,000 available. The model seems to give up rather than persist through difficulty.

The overthinking trap

Analysis of the models' internal reasoning reveals another problem: inefficient exploration patterns. On simple problems, models often find correct solutions early but continue exploring wrong paths—a phenomenon researchers call "overthinking."

For medium complexity problems, the pattern reverses. Models explore many incorrect solutions before stumbling on correct ones late in their reasoning process. Beyond a certain threshold, they find no correct solutions at all.

This suggests these models lack effective self-correction mechanisms. They can't distinguish promising reasoning paths from dead ends, wasting computational resources on fruitless exploration.

Real-world implications

These limitations have serious implications for deploying reasoning models in critical applications. If models consistently fail beyond certain complexity thresholds and can't follow explicit algorithms reliably, they're unsuitable for tasks requiring guaranteed logical consistency.

The findings also question whether current training approaches can produce genuinely generalizable reasoning. Despite sophisticated reinforcement learning techniques, these models appear to rely on pattern matching rather than developing robust logical reasoning capabilities.

The contamination problem

Traditional AI benchmarks suffer from data contamination—models perform well because they've seen similar problems during training. The researchers' puzzle approach sidesteps this issue by using controllable environments with systematic complexity scaling.

Mathematical benchmarks like AIME show suspicious patterns. Models perform worse on AIME 2025 than AIME 2024, despite humans finding the newer test easier. This suggests training data leakage rather than genuine reasoning improvement.

Beyond current limits

The study exposes gaps between current AI capabilities and human-like reasoning. While these models excel at pattern recognition and can handle moderate complexity tasks, they lack the persistent, systematic thinking required for complex problem-solving.

Current approaches may have hit architectural limits. Simply scaling model size or training data won't solve the logical consistency problems revealed by this research. New training methods or architectures may be needed to achieve reliable reasoning at scale.

The models' inability to benefit from explicit algorithms particularly concerns researchers. If an AI can't follow step-by-step instructions reliably, it cannot serve as a dependable reasoning partner for complex tasks.

Why this matters:

• AI reasoning models hit hard limits that more compute can't solve—they think less, not more, when problems get complex
• These failures happen even when given explicit solution steps, revealing gaps in logical execution rather than creative problem-solving

Read on, my dear:

Apple Research: The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity