LightOn releases mDenseOn and mLateOn retrieval models for 8 languages

• LightOn shipped two open 307M retrievers for multilingual, long-context, and code retrieval, as the launch post frames mDenseOn and mLateOn.
• The data story is scale: the translate-train step says every curated English pair was translated into eight languages, yielding 2.8B multilingual query-document pairs.
• Late interaction is the useful result: mLateOn scores 67.04 on full MIRACL after scoring 65.61 on target languages, while the dense-model result shows mDenseOn dropping from 59.61 to 58.02.
• The biggest benchmark gap is long documents, where the MLDR result reports 87.69 on target languages and a 20-plus point same-data gap over mDenseOn.
• The release is reproducible in a real sense: models, 2.8B training pairs, 16.3M fine-tuning samples, and training code are included, according to the openness note.

The Hugging Face launch post gives the accounting: a 665M-pair English seed, about 1.9B translated monolingual pairs, and 220M document-swap cross-lingual pairs. The GitHub repo is the operator-facing part, with pretraining and fine-tuning scripts for multilingual dense, multilingual late-interaction, and English-only variants. The arXiv paper ties the new 307M mmBERT models back to the earlier DenseOn and LateOn English recipe.

LightOn took the English retrieval recipe behind DenseOn and LateOn, then translated it into French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, and Arabic. The launch post says the pretraining corpus combines:

• 665M curated English query-document pairs.
• About 1.9B translated monolingual pairs.
• 220M cross-lingual pairs, created by swapping the document into a different language.

Fine-tuning adds translated hard-negative data plus organic multilingual data from MIRACL, long-context data from MLDR, and code data from LateOn-Code, according to the fine-tuning mix. The official post says the full fine-tuning set is about 16.3M contrastive samples across nine natural languages and code.

The translation prompt translated the query and document together and asked the model to reformulate idioms rather than transliterate them. That is the right kind of boring detail for retrieval data, because paired translation can preserve the local meaning that separate query/document translation can break.

mDenseOn and mLateOn share the same mmBERT-base backbone at 307M parameters, but they expose different retrieval tradeoffs:

• mDenseOn: one embedding per input, CLS pooling, cosine similarity, query: and document: prefixes.
• mLateOn: one embedding per token, no pooling, MeanMaxSim late interaction, [Q] and [D] prefixes, and a 768 to 1536 to 768 to 128 projection head.

The official blog describes the pair as matched on backbone, corpus, cross-encoder teacher, mined negatives, batch size, and training duration. That setup makes the late-interaction gaps easier to read as a paradigm effect, not a data or backbone mismatch.

The multilingual models did not pay an obvious English tax. On BEIR, the BEIR result reports 56.70 for mDenseOn and 57.56 for mLateOn, with mDenseOn above the earlier English-only DenseOn and mLateOn as the best evaluated model in that comparison.

The prior English release used 665M curated pretraining pairs and 1.88M hard-negative fine-tuning pairs, according to the English recipe recap. The arXiv paper reports the earlier 149M DenseOn and LateOn at 56.20 and 57.22 average nDCG@10 on BEIR.

The dense model shows the normal translate-train failure mode. The MIRACL dense result puts mDenseOn at 59.61 on target languages and 58.02 on the full benchmark, with languages outside retrieval training treated as second-class citizens.

mLateOn flips that pattern. The MIRACL late-interaction result puts mLateOn at 65.61 on target languages and 67.04 on all languages, so adding unseen languages improves the average.

The per-language chart is the money shot:

• MIRACL mean lift over mDenseOn: +6.0 on target languages, +10.2 on unseen languages.
• MLDR mean lift over mDenseOn: +22.7 on target languages, +30.6 on unseen languages.

LightOn trained mostly on Romance and Germanic languages plus Arabic, but the script-transfer note says the model stayed strong on Japanese, Cyrillic, and other typologically distant scripts. Benjamin Clavié, a retrieval researcher and former LightOn collaborator, gave the outside read: his reaction said the result should update priors on late interaction for out-of-distribution retrieval.

MLDR is where the dense-vs-late split becomes lopsided. The MLDR result reports mLateOn at 87.69 on target languages, about 9 points over the next best model.

The caveat is in the same post: LightOn fine-tuned on the MLDR training split. That makes the matched comparison more useful than the leaderboard position, and the same MLDR post says mLateOn still beats mDenseOn by more than 20 points under the same data and backbone.

The official launch post adds one training detail: supervised fine-tuning and evaluation use sequences up to 8,192 tokens, while late-interaction contrastive pretraining truncates documents much more aggressively at 300 tokens. LightOn reads that as evidence that most long-document capability is acquired in supervised fine-tuning.

Code retrieval came from fine-tuning, not code-specific pretraining. The code result reports 71.5 for mDenseOn and 73.5 for mLateOn on MTEB Code after training on LateOn-Code data.

The launch chart puts mLateOn on the code Pareto frontier at 307M parameters and 73.48 nDCG@10. It sits below voyage-4-nano at 76.43 and above jina-v5-text-nano at 70.75 in the same chart.

This release ships the parts retrieval papers often leave implicit: models, the 2.8B training pairs, the 16.3M fine-tuning samples, and training code. The repo README describes Python scripts for multilingual and English dense pretraining, multilingual and English late-interaction pretraining, and the corresponding fine-tuning paths.

The training stack also matters. The launch post says mLateOn was trained with PyLate, mDenseOn with Sentence Transformers, and late-interaction evaluation used FastPLAID for approximate multi-vector search.

Open data is already the community-facing bet. In a follow-up post, antoine_chaffin framed the release as a path from random weights to production-ready models, after ModernBERT, Ettin, PyLate, LateOn-Code, ColBERT-Zero, DenseOn, and LateOn.

The release leaves room for obvious experiments. In a backbone reply, antoine_chaffin said the team did not ablate the backbone much, was biased toward ModernBERT-based backbones, and would like to see runs with EuroBERT or recent Liquid AI backbones.

A separate reply says the unsupervised post-pretraining checkpoints are also released. For a Wolof question, the checkpoint reply said adding a bit of Wolof during fine-tuning should help, and that other intermediate pretraining checkpoints are less useful because the team did not use a WSD scheduler.

The next branch is multimodal. The multimodal reply says PyLate is already multimodal-ready on a branch, with data cleaning and scaling underway.