Benchmark author says GLM-5.3-Flash rerun improves scores after routing error

• A configuration-controlled rerun lifted GLM-5.3-Flash's vision result: skalskip92's rerun protocol used pinned native FP8 endpoints and a self-hosted checkpoint across 12 runs.
• The original score probably included quantized endpoints, according to skalskip92's routing explanation, because the evaluation did not set a precision filter.
• The corrected run still leaves concrete object-detection weaknesses: skalskip92's follow-up cites crowded scenes, imprecise boxes, and one general box where several are required.
• The review now extends beyond GLM-5.3-Flash, with skalskip92's reevaluation update saying prior OpenRouter-tested models may also have been scored too low.

OpenRouter's provider-routing guide exposes quantizations as a provider filter, while its cost-routing explainer notes that the cheapest endpoint can be quantized. Roboflow's Object Detection evaluation requires a model to return both a class label and a bounding box for every object. Z.AI's official BF16 model card describes GLM-5.3-Flash as a 320B-parameter multimodal MoE with 18B active parameters.

Variation appeared before the postmortem. Testing GLM-5.3-Flash through the OpenRouter API at different thinking levels, skalskip92 found it worse than an earlier run and asked whether it was being served at lower precision in skalskip92's API test.

The later diagnosis was that, without a provider quantization filter, the router could use available providers, skip those with recent outages, then prefer cheaper endpoints. Those cheaper endpoints were often quantized, as skalskip92's routing explanation put it.

Four three-run cells made up the replacement sample, according to skalskip92's rerun protocol:

• OpenRouter pinned to native FP8, low effort.
• OpenRouter pinned to native FP8, max effort.
• Self-hosted official checkpoint, low effort.
• Self-hosted official checkpoint, max effort.

The local setup ran the roughly 306 GB official checkpoint on two B200s with vLLM, using FP8 experts and BF16 for the remaining weights and vision tower skalskip92's self-hosted setup. The reported result selects the best prediction for each image from all 12 runs in the rerun protocol.

The corrected low-effort mAP@50 bar rose from 33.1% to 36.3%, a 3.2-point change, according to skalskip92's correction note. Gemini 3.7 Flash remained at 69.4% in the same chart.

The evaluator's remaining failure pattern is named here:

A box can look plausible and still fail the metric when its overlap with the ground-truth region is too small, skalskip92's mAP clarification added.

Agent Arena measures a different workload, real-world long-horizon agentic tasks traced causally to model outcomes, as arena's methodology note explains.

Its per-signal breakout listed by arena's signal update was:

• Confirmed Success: +15.3%.
• Praise vs. Complaint: +4.9%.
• Steerability: +2.4%.
• Bash Recovery: -0.7%.
• Tool Hallucination: no issues reported.

Skalskip92 said the issue might have lowered scores for multiple models previously evaluated through OpenRouter and had started rerunning all of them in skalskip92's reevaluation update.

The correction note specifically named Glimmer 30B and Qwen 3.8 27B alongside GLM-5.3-Flash in skalskip92's correction note. In a later reply, skalskip92 put the potential scope at six of 34 models skalskip92's scope estimate.