Independent DeepSWE test reports Jev Router nearly 5x slower than GPT-6 Astra low

• A 20-task DeepSWE run tied Jev Router with GPT-6 Astra low at 16 of 20 tasks, but its $1.50 per-task cost and 19.9-minute mean runtime exceeded Astra low's $1.46 and 4.6 minutes in theo's $1,000 benchmark.
• OpenRouter's Jev Router uses a decision model to choose model and reasoning effort on each turn, while preserving a working model through a session, as OpenRouter's launch thread describes.
• A Pi SDK support-agent trial by omarsar0 found the reverse cost and latency result on eight cases: both systems were correct, while the router cost $0.008 versus $0.018 and took 1.5 versus 1.9 seconds at the median.
• Request-level routing lacks repository, tool, and task context, kunchenguid argued, while cache loss can erase savings from switching models mid-session.
• A Jev timeout or invalid decision fails the request under OpenRouter's stated failure handling, rather than falling back to another router.

The official model listing calls the endpoint OpenAI-compatible, with the model slug as the only required swap for most SDKs. Its comparison page lists a free endpoint and a 1 million-token context window. An independent launch report noted that OpenRouter had not yet published endpoint usage data or comparative routing results.

The run used the public-leaderboard harness, 20 tasks, and one attempt per task. Its result is a tie on solved tasks, followed by a dramatic spread in elapsed time.

• Jev Router: 16/20, or 80%; $1.50 per task; 19.9 minutes per task.
• GPT-6 Astra low: 16/20, or 80%; $1.46 per task; 4.6 minutes per task.
• GPT-6 Astra xhigh: 17/20, or 85%; $4.42 per task; 12.8 minutes per task.

The benchmark establishes result parity with Astra low in this harness, without a cost or runtime gain. theo expanded the critique in a follow-up, then agreed in a reply when pvncher argued that clear, isolated benchmark tasks omit hour-long agent runs with compaction, ambiguous prompts, repository navigation, and user steers in the same discussion.

OpenRouter's design starts with a real constraint: moving a conversation to another model can force that model to reread the cached chat. The router uses Jev before each turn to select both a model and reasoning effort.

• Jev scores task difficulty and precision, then estimates whether a larger model, more effort, a cheaper model, or a model change would help, according to OpenRouter's routing description.
• It retains a model for the session, can change effort without changing models, and switches only when expected gain exceeds the cost of the lost cache, per OpenRouter's session policy.
• Jev reads conversation text solely for routing under zero-data-retention terms; attachments are not sent to it, OpenRouter's privacy note says.
• A timeout or invalid Jev response ends the request, with no fallback router, in OpenRouter's failure behavior.

A small support-agent evaluation produced a sharply different result from DeepSWE. omarsar0 used eight cases against a fixed GPT-6 Sol baseline, for 32 real calls total.

Both systems got every case right. The router's median response time was 1.5 seconds versus 1.9 seconds, and its cost was less than half of the baseline's.

A separate custom workflow test from kevinkern ran into its 30-minute limit with Jev in the loop: the Team + Jev run reached 12 of 14 checks at $6.72, while Astra solo completed all 14 at $6.98 in 21.3 minutes.

omarsar0 said in response that no shared number deserves blanket trust, a useful qualification for two experiments with different agent harnesses and task shapes.

The central critique is that task difficulty is usually not present in the user message. kunchenguid's example puts the identical prompt, “how does this work,” in a one-file HTML project and in the Linux kernel, where the required investigation is radically different.

That argument places routing at task level: a capable agent investigates the codebase, then delegates substantial subtasks to models with suitable capability. It also treats cache economics as a reason a gateway cannot swap models freely throughout one continuous session.

The same context problem drives theo's critique: a request to port code to Rust does not reveal whether the system is a 100-line TypeScript file or a million-line application. theo further said Jev has a 32K text limit, no vision input, and cannot request the missing context in a follow-up.

A snapshot from theo's run showed a router heavily concentrated on one budget model rather than spreading traffic evenly across the available set.

• deepseek/deepseek-v4.1-flash: 59% of calls.
• openai/gpt-6-sol: 24%.
• google/gemini-3.8-flash: 18%.
• x-ai/grok-4.7: 0%.

OpenRouter exposes per-turn routing metadata and a Chat insights panel with the selected model plus task, difficulty, precision, and larger-model-benefit scores, according to its UI description.

TanStack AI shipped subagents that can be defined once and passed into chat(). Its integration uses Jev to route each message to the appropriate agents, serially or in parallel, while each agent streams a separate UI card.

AlemTuzlak described the implementation as a utility that prepares routing input for Jev, then reads its response to create the workflow routing.