Artificial Analysis updates Decision Index v2 with confidence-aware scoring

• Confidence now affects the headline score, and private datasets account for 45% of the index rather than 25%, according to Artificial Analysis's launch thread.
• Document-level reliability trails question accuracy: the AA-Paperclip results give Drex Max v1.1 97% accuracy on individual answers but a 63% whole-document pass rate.
• Matching rounded scores conceal very different bills: Drex Max v1.1 and GLiDE v1.2 both score 73, but cost $102 and $780 per million tasks respectively in the cost comparison.

Language models now return probability distributions in structured JSON, with malformed replies retried once. The new symmetry test gives credit for confidently sticking with “I don't know.”

Each answer scores 50 × (1 + c − (p − c)²), where p is the probability assigned to the chosen answer and c is 1 for correct or 0 for incorrect. Averaged across an evaluation, this equals 50 × (1 + Accuracy − Brier score).

Artificial Analysis describes the score as the average payoff across confidence thresholds used to route uncertain answers to a person or another model. It therefore avoids selecting one particular routing cutoff.

Calibration changes the results. Clef passes 36% of AA-Paperclip documents versus 24% for Celeris-1.1 Decision, yet Celeris matches it on Decision Score, with a Brier score of 0.15 versus Clef's 0.27.

The November 8 update combines five Decision Scores with one Symmetry Score, using the following benchmark weights:

• AA-Paperclip, 30%: new private document decisions and extraction test.
• AA-Symmetry, 15%: consistency across equivalent questions.
• EvalSafe-O*NET-AA, 15%: new workplace-document judgment evaluation.
• AA-Omniscience-Decision v1.1, 15%, down from 25%: ten alternative phrasings of the abstention option.
• HLE-MultiChoice v1.1, 15%, down from 25%: adds an option to decline, using those ten phrasings.
• When2Call, 10%, down from 25%: NVIDIA's tool-calling evaluation.

WildGuardTest was removed because of high saturation. Drex Max v1.1 narrowly leads GLiDE v1.2, both rounded to 73, with Mammoth 27B at 72.

Small changes to index weights can reshuffle models, one critic argued in a separate benchmark discussion. Public benchmarks belong at the start of model selection, with private evals testing product fit, the same commenter argued.

AA-Paperclip is the standout addition: each of its 1,500 private documents requires five decisions and ten fact extractions, and all 15 answers must be correct to pass.

Whole-document pass rates are:

• Claude Haiku 5.5 (Max): 92%.
• GPT-6 Luna (Max): 86%.
• Drex Max v1.1: 63%.
• GLiDE v1.2: 62%.
• Decider v1.2: 55%.

Haiku's individual-answer accuracy is 99%, only two points above Drex's 97%. Drex gets all five decisions right on 71% of documents and all ten extractions right on 88%, putting more of its failures on decision-making.

The joint-confidence calculation multiplies the probabilities of all 15 chosen answers, explicitly treating them as independent. Small per-answer calibration errors consequently compound at document level.

AA-Symmetry tests 120 base questions across 440 variations, changing:

• Option order.
• Option labels.
• Yes/no phrasing through negation.

Changing the top answer earns zero; keeping the answer while moving its probability also reduces the score.

Mammoth 27B scores 96 and Elephant 9B scores 93, driven by frequent, confident abstention, according to the symmetry results. Mammoth keeps the same answer across variations on 99% of questions, compared with 91% for Drex Max v1.1 and 88% for GLiDE v1.2.

GLiDE v1.2 and Decider Flash v1.2 evaluate questions in separate passes and bill the input repeatedly. Artificial Analysis reports about 63,000 input tokens per AA-Paperclip task for those models, versus about 9,000 for Drex Max v1.1.

The index-wide cost figures expose two pricing reversals:

• GLiDE v1.2: $780 per million tasks versus Drex Max v1.1's $102, nearly eight times the bill despite only 33% higher list prices.
• Decider Flash v1.2: $196 per million tasks versus Decider v1.2's $68, nearly three times the bill despite half the token price.

Across decision models, costs range from $14 per million tasks for Celeris-1.1 Decision to $859 for Clef.

Higher-scoring language models take seconds rather than milliseconds in Artificial Analysis's timing results:

• Drex Max v1.1: index score 73; 275 milliseconds per task; $102 per million tasks.
• GPT-6 Luna (Max): score 76; 11.34 seconds; $1,530 per million tasks.
• Claude Haiku 5.5 (Max): score 77; 32.65 seconds; $6,240 per million tasks.

Celeris-1.1 Decision records the lowest time per task at 56 milliseconds. Mammoth 27B sits four index points below GPT-6 Luna (Max) while running 118 times faster.

EvalSafe-O*NET-AA uses synthetic workplace documents and reference-model judgments rather than ground-truth answers. The methodology identifies GPT-6 Astra and Claude Fable 5.1 as the reference models.

Artificial Analysis evaluates 6,582 of the original 7,500 questions where both models agree on the most likely answer. It excludes the remaining 918 because of disagreement, ties or missing answers.