T3 Code usage comparisons draw criticism over unmatched tasks

• T3 Code’s usage comparisons pool unmatched real-world work, Theo clarified in a reply.
• Model-efficiency conclusions lack key context: pvncher asked for task matching, effort settings, delegation details and harness instructions.
• Gemini’s 51% false-claim rate came from a separate repository-understanding test with unequal effort settings, shown in theo’s chart.

An orchestration wait timeout in T3 Code leaves the delegated agent running. One daily user reported that T3’s injected instructions clashed with his personal rule against subagents.

Theo said the usage figures aggregate real-world activity in T3 Code across different tasks.

pvncher asked for basic benchmark plumbing in his critique:

• Comparable tasks for both models.
• The effort level used for each model.
• Whether each model received the prompt directly or worked as another model’s subagent.
• Differences in skills and AGENTS.md instructions between harnesses.

T3’s README calls the product an “agent harness control surface.” Theo defended a distinction between the runtime an LLM connects through and tools added to that runtime.

pvncher argued that T3’s orchestrator, custom wait tooling and instructions steer model behavior beyond what a UI alone would change.

The official orchestration specification documents several app-owned operations:

• delegate_task creates a T3 child thread, optionally using a different provider or model.
• task_status reads delegated-task state; task_cancel stops delegated work.
• t3_thread_launch starts a top-level thread with a workspace bound before the agent starts.

pvncher accepted the architectural distinction and suggested new terminology for the opinionated layer above the core harness, arguing that its prompts and tools can substantially change behavior on the same task.

Cache handling was another objection: pvncher said T3’s subagents do not use cache-safe thread forking, which could inflate usage depending on the prompt.

T3’s delegation documentation says a child receives the supplied task prompt and an optional role instruction. The parent’s conversation history is not copied into it.

Prompt-cache behavior also depends on the surrounding tool and instruction configuration. Anthropic’s cache documentation specifies a tools → system → messages hierarchy: changing tool definitions invalidates all three levels, while changing the system prompt invalidates the system and message caches.

Theo said his team put “a disgusting amount of effort” into a separate comparison of models’ understanding of T3 Code’s architecture.

He later characterized Gemini’s false claims as “lying” and said ACP bugs were separate issues that traditional benchmarks had not measured.

The integration also had a version constraint: Theo reported staying on 1.4.1 because versions 1.4.2 onward automatically wrote a skill file.

The wider harness debate produced competing positions:

• Native harnesses: Models should run in the harness they were trained for, with behavior controlled through prompts, mattshumer_ argued.
• Custom harnesses: Hermes consistently beats vendors’ harnesses using their own models and performs better on open models, Teknium claimed, arguing that turning Hermes into a meta-harness would reduce its quality.

VS Code demonstrated six comparison sessions in its Agent Bake Off video, with controls that expose several of the variables disputed in the T3 discussion:

• A shared prompt runs across selected models in parallel.
• Effort levels and harnesses are selectable, including Claude, Codex and Copilot.
• A configurable judge compares the attempts, including timing, token usage and results.
• A synthesis step can combine findings into a final result.