Theo finds 23 wrong claims in Gemini 4 Carbon code analysis

• Carbon made 23 wrong claims out of 79 (29.1%) in an architecture evaluation by Theo, the developer behind T3 Code.
• Its eight-PR review score improved from Argon's 12.5 to 26.5, below Sonnet 5.5's 35, in his code-review results.
• Errors included invented line numbers, he clarified; his accompanying video reports an 18% false-claim rate across his full suite.
• Carbon finished ninth among 11 models in his five-test comparison, which measures relative standing across workloads.

Google claims Carbon leads CursorBench 4.0 and recommends it for architectural design and code auditing. Some earlier Gemini hallucinations also involved a T3 Code integration bug that withheld line numbers, Theo acknowledges in his video.

Theo asked models to review the architecture of his TypeScript-to-Rust compiler port. The results separate wrong claims from the overall architecture score, runtime, cost, and output volume.

Converting the wrong-claim counts to percentages:

• Gemini 4 Argon: 24 of 36, 66.7%.
• Gemini 4 Carbon: 23 of 79, 29.1%, a 37.6-percentage-point decline.
• Opus 5.5: 1 of 34, 2.9%.
• GPT-6.1 Sol: 0 of 38.
• GPT-6 Astra: 0 of 33.

The error pile barely shrank: Carbon produced 43 additional assertions and only one fewer false claim. It completed the review in 29 minutes, compared with 44 for Argon and 50 for Opus.

Theo's video reports two additional comparisons:

• T3 Code analysis: Argon made 47 false claims out of 92 (51.1%); Carbon made 23 out of 86 (26.7%).
• Across all his evals: the reported false-claim rate fell from 36% to 18%, a 50% relative reduction.

Both the 18% aggregate and the 29.1% architecture result are below the figure in his “30%+” tweet. These percentages count factual assertions in generated analysis, with different numbers of assertions produced by each model.

Some earlier Gemini hallucinations came from T3 Code's old ACP bindings, Theo says in his video. The integration failed to pass information, including line numbers, that the model would have received through the official Antigravity CLI.

DeepMind's team audited the bindings and fixed the integration, according to his account. He also describes models inventing database column names and making unsupported claims about application architecture.

The code-review test uses eight real T3 Code pull requests and separates valid, plausible, and wrong findings. Its table also tracks recall, novelty, runtime, cost, and output.

Carbon's 14-point gain over Argon put it ahead of all three GPT entries in this test. It finished fifth of nine models, behind Sonnet 5.5, Fable 5.2, Muse Wonder Full, and Opus 5.5.

Theo's broader comparison spans five workloads:

1. TS-Rust architecture.
2. T3 Code analysis.
3. Lakebed.
4. Review.
5. D1.

Carbon's score of 46 is its normalized mean standing, where 100 means best on every benchmark a model ran. Sonnet 5.5 scored 92, Opus 5.5 scored 91, and Carbon ranked ninth of 11.

Carbon's dollar-cost cell is blank in the architecture table. Google's developer guide lists introductory prices of $2 per million input tokens and $10 per million output tokens through November 30, rising to $4 and $20 on December 1.

Theo also reports a separate Fable allowance capped at 50% of the weekly limit on the $200 Claude Ultra plan. His chart values one account's logged traffic at API list prices, using 1.1 million proxied requests from August 11 through November 9.

API-list-price equivalents of usage on one $200 Claude Ultra account

Roughly 70% of T3 Code feature requests are “prompt skill issues,” Theo estimated in a separate thread. Responding to a comment about education, he called that his problem too.