LangSmith adds Jev as a production trace judge

• LangSmith has added Jev as a production evaluator, so trace and thread data can become structured feedback keys that drive charts, alerts, and automations, according to LangChain's rollout.
• The integration evaluates a shared trace state with typed Noul, Choice, and Score questions, rather than parsing a judge's prose, as LangChain's guide describes.
• LangChain's small fixed-trace experiment reported 100% agreement with one human oracle across 500 repeated pass/fail calls, 0.44-second average latency, and $0.34 total cost for Jev, per LangChain's benchmark results.
• Judge quality changes sharply by task: a separate Braintrust benchmark put Jev at 80.2% on groundedness but 78.1% on answer correctness, behind reasoning models on the latter.
• Calibration and high-risk gates remain task-specific: an r/MachineLearning calibration analysis found larger gaps to human labels than several LLMs, while hrishioa's agent-run analysis called Jev dangerous for harmful-command detection.

The published methodology did not record the Jev service version, despite replaying fixed runs across judges. A separate DocJev field test reported 40 of 40 document classifications at 139 ms against 794 ms for GPT-5.6 Luna, then found a closer split on document boundaries, 7 of 8 versus 8 of 8.

LangSmith now lists TypeSafe as a model provider. In a tracing project, the workflow creates an LLM-as-a-Judge evaluator, selects jev-latest, maps run or thread fields into state, and defines named questions, according to the official evaluator guide.

The state holds the trace context. The questions hold the grading criteria. Each question becomes its own feedback key after the evaluation runs.

The three supported shapes, documented in TypeSafe's primitives reference, are:

• Noul: a yes-or-no probability, useful for a condition such as PII leakage.
• Choice: one named option, its probability distribution, and confidence.
• Score: one level on an ordered rubric, its probability distribution, and confidence.

LangSmith says questions against the same state run together, so PII leakage, user intent, and frustration can be separate feedback keys in one request. Those keys can trigger a webhook, including for safety or security flags; hwchase17's post characterized the pattern as cheap, fast semantic verification for online evals.

A separate production integration report from kylejeong claimed a 4.3x speed improvement, though it did not publish its taskset or baseline configuration.

The headline benchmark is narrow. LangChain's evaluation announcement describes a Deep Agents weather agent whose five captured responses were stored as fixed LangSmith examples; one human reviewer labeled them on the same rubric.

Each judge scored two signals, a continuous quality score and binary does_pass, over 100 repetitions per captured run. The benchmark therefore measures repeatability on unchanged traces as well as agreement with that single reviewer's labels.

For the binary decision, LangChain reported:

• Jev: 500 of 500 agreements with the human oracle, 100.0%.
• GPT-5.6 Terra: 99.8%.
• GPT-5.6 Luna: 96.4%.
• Claude Sonnet 4.6: 80.0%.

Its mean per-case quality-score variance was 0.0000149 for Jev. LangChain reported Luna at 433 times higher variance, Terra at 913 times, and Claude at 92 times.

The same run averaged 0.44 seconds and $0.00035 per Jev call, compared with 2.16 to 2.83 seconds for the LLM judges. The full set cost $0.34 with Jev, $0.39 with Luna, $2.90 with Terra, and $28.17 with Claude. The methodology says providers' default temperature, top-p, seed, and token settings applied to the LLMs.

Braintrust tested a different judge profile and found Jev particularly competitive for groundedness. Its chart put Jev at 80.2% groundedness accuracy across 1,086 claims, ahead of DeepSeek V4 Flash without reasoning at 79.2%.

The same evaluation put Jev at 78.1% answer correctness across 616 independent pairs. GPT-5.6 Luna with reasoning reached 88.6%, DeepSeek V4 Flash with reasoning 86.4%, and the no-reasoning DeepSeek configuration 85.9%.

Braintrust said the gap emerged on tasks involving math and code, where reasoning-capable judges did better. Its result lines up with the boundary LangChain describes for the integration: narrow, typed judgments produce a different operating profile from open-ended judging with written reasoning.

A r/MachineLearning post compared calibration gaps against human labels, where lower is better. It reported gaps of 5.0 for Jev versus 3.8 for Gemini Flash on yes/no, 9.8 versus 4.1 for DeepSeek V4 on pick-one, and 19.7 versus 12.9 for GLM-5 on rubric scoring.

The poster also reported that Jev could act alone on 86% of yes/no decisions while holding 95% accuracy. That is a coverage result, not evidence that a probability carries unchanged calibration into a new trace distribution.

Cost is also a moving baseline. zeeg's critique questioned whether some Jev savings come from replacing unnecessarily expensive judges, rather than from a previously unavailable evaluation capability. The production launch changes the economics of running many typed checks, while the useful question still depends on the task and the judge it replaces.

After analyzing several thousand agentic runs, hrishioa's report said Jev was the best option they had tested for measuring progress and estimating task completion. The same report said it was dangerous for detecting harmful commands and weak at catching lazy agent behavior.

theo's warning offered the broader version of that concern, saying people were already using Jev incorrectly. These reports concern operational judgments that lie outside LangChain's five weather traces and Braintrust's groundedness and answer-correctness sets.

One operational caveat sits in the LangSmith setup instructions: TypeSafe does not currently offer zero data retention, so prompts and outputs sent for evaluation may be retained by the provider.

The configuration requires a TYPESAFE_API_KEY stored in LangSmith Provider Secrets, then uses TypeSafe with the jev-latest model slug. That makes the retained material potentially include whichever run or thread fields were mapped into evaluator state.