Developers test Jev as a low-cost judge for agent evaluations

• Jev is being tested as a low-cost semantic judge for repeated, bounded rubric checks, a use case hwchase17 framed as cheap, fast online evaluation.
• A controlled LangChain experiment found Jev matched its human oracle on all 500 repeated pass/fail decisions from five frozen weather-agent runs, according to LangChain's evaluation announcement.
• Reinforcement-learning graders are a natural volume target: Vtrivedy10's rollout diagram puts a classifier check at every non-programmatic rubric criterion for every rollout.
• A trace replay from hrishioa's field analysis improved time and spend forecasts, while its security test found harmful commands could bypass the classifier.

A Langfuse integration guide sends a task, tool calls, and final output to one request, then receives a review flag, severity score, and failure mode. A Southbridge replay found a written 0-5 progress rubric far steadier than asking for a raw completion percentage.

Jev takes a state, text or JSON, plus questions whose answer space is fixed before the call. The Langfuse guide describes three evaluator primitives:

• Noul: the probability that a binary proposition is true.
• Choice: a selected label, per-label probabilities, and confidence.
• Score: a probability-weighted position on an ordered rubric, plus its distribution and confidence.

Questions run in parallel and in isolation over the same state. The useful unit is an atomic test, not an open-ended critique.

In LangChain's controlled experiment, judges saw five captured weather-agent runs, each assessed for a continuous quality score and a binary pass/fail verdict. A human reviewer labeled those fixed outputs, then Jev, GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 each ran 100 times per case.

Jev matched the human pass/fail labels on 500 of 500 repeated decisions. Terra reached 99.8%, Luna 96.4%, and Claude 80.0%; Jev's mean per-case quality-score variance was 0.0000149, versus 92x to 913x higher for the three chat-model judges.

Jev averaged 0.44 seconds and $0.00035 per call, $0.34 across the run, compared with $28.17 for Claude. The result covers five fixed traces rather than 500 independent agent episodes, and LangChain did not set temperature, top-p, seed, or max tokens for the LLM judges; the Jev service version was unavailable in its metadata.

In Vtrivedy10's proposal, an RL task supplies the prompt, tools, sandbox, initial state, and a rubric with M criteria. The grader then applies those criteria to each of N independent rollouts.

The diagram separates programmatic checks, such as tests passing, from semantic checks, such as whether an explanation is grounded in the diff. Each pass/fail judgment feeds rewards and advantages back into the policy update.

Although omarsar0's prototype originally described a completion verifier after every turn, a follow-up says its current trigger is when the agent claims it is done; intermediate checks remain an experiment. The same thread says a benchmark for that long-horizon use case does not yet exist omarsar0's reply.

Southbridge tested a different role: an observer that reads an agent trace rather than choosing its next action. Its field report replayed 13,549 observations across 500 completed runs in 12 task families, asking eight questions at each point.

On held-out runs, a predictor using runtime counters alone missed remaining time by 3.81 minutes on average; adding Jev readings reduced that to 3.13 minutes. Remaining-spend error fell from 3.72 cents to 3.01 cents, and the full replay recorded $2.30 in Jev usage.

The team abandoned a direct 1-100 completion question after identical requests returned 100, 100, and 1. A six-level phase description, from barely begun to delivered and verified, produced steadier readings. The authors also report that Jev overdiagnosed agent laziness.

The probability distribution belongs to more than the model. _avichawla's note says every probability is conditional on the candidate set, so adding one label changes all the others even with the same model and prompt; the deployable unit includes the model, prompt, tokenizer, labels, and thresholds.

HamelHusain's graphic frames validation around human labels split into train, development, and test sets. It also favors true-positive and true-negative rates over a single accuracy number when classes are imbalanced.

Southbridge's security pass scored 220,297 real tool calls, including 127,995 shell calls, and flagged 3,814 as dangerous. About 1% of decisions reversed on retry, and the report says hex encoding and Python wrappers could get harmful commands through.

The same test marked 82 of 120 sampled messages as unjustified shortcuts, compared with 10, two, and two for DeepSeek, Gemini, and GLM under the shared 0.50 threshold. The Southbridge analysis characterizes Jev as an extra check against honest model mistakes, not a defense against hostile intent.