Reason-ModernColBERT ranks 87.59 on BrowseComp-Plus

• LightOn’s benchmark thread says its 150M-parameter Reason-ModernColBERT, paired with GPT-5, reached 87.59 accuracy on BrowseComp-Plus, a 7.59-point gain over the previous best.
• The same benchmark thread reports wins on recall and calibration too: 83.52% recall versus 80.29%, and 7.46 calibration error versus 7.92, while using fewer search calls.
• Practitioner reaction centered on model size efficiency: follow-up thread notes the strongest prior retriever baseline was Qwen3-8B, about 54 times larger than the 150M ColBERT model.
• The result also points to a workflow change, because scaffold details says a simple get_document(id) fetch tool improved performance over the official top-5-snippet-only scaffold, suggesting retrieval quality is driving deep-research performance more than bigger generators alone.

According to the primary results thread, Reason-ModernColBERT topped BrowseComp-Plus with GPT-5 at 87.59 accuracy, improving on the prior best by 7.59 points while also setting the best reported recall and calibration error in the same run. That matters because earlier bests on those metrics came from different runs, so this is not just a single-metric win.

The size delta is unusually large for a retriever result. In a companion reaction, the baseline being beaten is described as Qwen3-8B, making the winning retriever roughly 54 times smaller at 150M parameters. LightOn’s own thread adds that even its generic GTE-ModernColBERT base model outperformed Qwen3-8B, which suggests the gain is not only from reasoning fine-tuning but from the retrieval architecture itself base-vs-reasoning comparison.

The most concrete implementation detail in the primary thread is the scaffold difference. The official BrowseComp-Plus setup exposes search returning top-5 documents plus the first 512 tokens of each, while LightOn also tested a minimal variant that adds get_document(id) so the LLM can pull full documents on demand. The post says that simple change both boosted performance and reduced search calls, which implies the retriever is surfacing high-signal candidates early enough that the model can spend fewer tool invocations.

That interpretation is echoed in community discussion. In a direct reply, Jo Kristian Bergum argues the result shows “deep research is a retrieval problem, not a reasoning problem” and says the score is approaching oracle-level accuracy. A separate reaction from the late-interaction thread makes the sharper claim that dense single-vector retrievers are the real bottleneck on quality and generalization, not that late interaction is merely incrementally better.

This was not posted as a closed benchmark stunt. The main thread says the models, training code, and data are open, with links to the Hugging Face checkpoints Reason model and GTE model, plus PyLate training examples GTE training code and Reason training code. It also says the models were trained in about four hours and that PyLate keeps the workflow close to Sentence Transformers, so existing dense-retrieval pipelines can be adapted rather than rebuilt PyLate docs.

There is also some immediate systems-level optimization work around this model family. In the Sentence Transformers note, Tom Aarsen says he is adding input flattening to remove padding tokens with Flash Attention 2 and has seen about “+50% training and inference speed” in tests. Combined with the benchmark thread showing fewer search calls at higher quality, the engineering takeaway is that late-interaction retrieval is getting both algorithmic and runtime wins at the same time.

Antoine Chaffin also said in a follow-up reply that BrowseComp-Plus may be “almost solved” and that harder datasets may be needed next, which is a sign this result is pushing beyond leaderboard churn into benchmark-saturation territory.