Artificial Analysis released a revised Intelligence Index on Tuesday. Models with downloadable weights trail the proprietary leader by 26 points on the revised suite. The results do not establish that individual models have deteriorated.

Evaluations with private datasets or withheld answers now account for 60% of the index’s weighting. The independent benchmarking organization’s new methodology combines nine evaluations, with knowledge work, coding, science and general capabilities each receiving a quarter of the total weight. In that 60% of the index, the evaluations keep private test questions or held-out answers from developers before testing.

What Changed

AI-generated summary, reviewed by an editor. More on our AI guidelines.

Scores on the revised suite

In Artificial Analysis’s October 27 release results, Claude Opus 5.5 leads with an index score of 58, followed by GPT-6 Astra at 57 and GPT-6.1 Sol at 55. All three scores use maximum reasoning effort, the setting controlling how much work a model can spend on a response.

Kimi K3.1 Pro scores 46, putting it 12 points behind the leader, compared with a three-point gap under version 4.3. Its weights have not yet been released. Changes to tests and weights make Intelligence Index scores unsuitable for long-term comparisons across versions.

Among models whose weights are actually available, DeepSeek V4.1 Flash 1017 and GLM-5.3 tie at 32 at maximum effort, leaving a 26-point gap to the leader on the revised index. Open weights means downloadable model parameters; it does not necessarily mean the accompanying software is open source.

These are Artificial Analysis’s measurements. The index primarily tests English-language text tasks and does not measure suitability for every application.

Coding changes must survive review

The new FrontierCode 2.0 benchmark contains 132 private coding tasks drawn from real software repositories. It grades whether a maintainer would accept the change, and failing any blocking check earns zero. Artificial Analysis runs each task three times in its Stirrup testing system. In its October 27 results, Opus 5.5 at maximum effort scores 46% on FrontierCode 2.0.

A published walkthrough from Cognition, the coding-agent company that created the benchmark, shows a failure by the overall index leader. Opus 5.5 fixes a reported bug in Mock Service Worker, a library used to simulate network requests, but discards the old generator without running its finally-block cleanup.

The immediate bug disappears, while the skipped cleanup can leave resources unreleased or the application in an inconsistent state. In Cognition’s example, Opus fails the task at every reasoning setting. That walkthrough is the benchmark creator’s demonstration, separate from Artificial Analysis’s composite results.

Know someone who'd find this useful? ✉️ Email it to a friend in one click, or they can subscribe free here.

The other addition, Terminal-Bench-Science, tests 70 expert-curated research workflows and requires every test for a task to pass. In Artificial Analysis’s release setup, Astra at maximum effort completes 63%, ahead of Opus at 59% and Sol at 58%.

Cost and comparisons over time

The October 27 cost figures show Sol at maximum effort costing $1.27 per Intelligence Index task, against $10.04 for Opus at maximum effort. Sol uses about one-eighth the spending per Intelligence Index task, with three fewer index points. Those prices measure spending on benchmark tasks, rather than the cost of successfully completing a business assignment.

The revised suite also replaces Humanity’s Last Exam with HLE-Diamond and updates its knowledge and physics evaluations. HLE-Diamond contains 1,000 questions, split evenly between reasoning and knowledge.

Artificial Analysis launched a separate Progress Index alongside the revision. It estimates model capability and task difficulty together, linking models through tests they share so that systems released years apart can be compared on one continuous scale.

Frequently Asked Questions

What does Intelligence Index v5 measure?

It combines nine evaluations, giving equal weight to knowledge work, coding, science and general capabilities. The suite primarily tests English-language text tasks.

Which models lead the revised index?

The October 27 release results put Claude Opus 5.5 at 58, GPT-6 Astra at 57 and GPT-6.1 Sol at 55, all at maximum reasoning effort.

Which leading models have downloadable weights?

DeepSeek V4.1 Flash 1017 and GLM-5.3 tie at 32 at maximum effort. Kimi K3.1 Pro scores 46, but its weights had not been released.

Does the wider gap mean open models got worse?

The revised tests and weights changed the comparison. The wider measured gap does not establish that individual models deteriorated, and scores across versions are unsuitable for long-term comparisons.

What do the reported task costs mean?

At maximum effort, GPT-6.1 Sol costs $1.27 per Intelligence Index task and Claude Opus 5.5 costs $10.04. These are benchmark-task costs, rather than the cost of successfully completing a business assignment.

AI-generated summary, reviewed by an editor. More on our AI guidelines.

Meta Releases Muse Spark Open, an 816B Model That Trails Chinese Rivals at Coding
Meta released the weights for Muse Spark Open on Thursday, a model it calls the most capable U.S. open-weight model to date. On Terminal-Bench 2.1, a coding test in Meta's own table, it scores 76.4 ag
The Implicator
Mistral Previews 1-Trillion-Parameter Large 4, Pitching Open Weights Against Lock-In
Earlier this year, Hugging Face was breached by rogue OpenAI agents. It turned to leading American closed models to help defend itself, only to find their guardrails blocking the work. The guardrails
The Implicator
Z.ai Delays GLM-5.3 Weights Two Weeks After Cyber Score Beats Mythos 5
Beijing-based Z.ai released GLM-5.3 on Friday while holding back the model's downloadable weights and gating its most sensitive cybersecurity functions. In launch tests, GLM-5.3 scored 84.5% on CyberG
The Implicator
AI News

San Francisco

Editor-in-Chief and founder of Implicator.ai. Former ARD correspondent and senior broadcast journalist with 10+ years covering tech. Writes daily briefings on policy and market developments. Based in San Francisco. E-mail: editor@implicator.ai