Sentence Transformers 6.0 adds MultiVectorEncoder for late-interaction retrieval

• Late-interaction retrieval is now a fourth first-class Sentence Transformers model family, with tomaarsen's release post introducing MultiVectorEncoder beside dense, sparse, and reranker models.
• In a same-backbone comparison, multi-vector LateOn led DenseOn on 9 of 13 NanoBEIR datasets and averaged 0.6868 versus 0.6764 nDCG@10, per tomaarsen's benchmark result.
• The index expansion is severe: tomaarsen's Natural Questions example produced 608,414 token vectors and a 311.5 MB index from 4,874 passages, about 16 times a simple 1024d dense index.
• CrossEncoder ranking can change dramatically under bfloat16 because sigmoid logits saturate; tomaarsen's precision-fix post reports nDCG@10 rising from 0.1849 to 0.6795 after float32 upcasting.
• The released package is sentence-transformers==6.0.0, linked by tomaarsen's install post alongside its blog post and full release notes.

The official walkthrough sends page images through the same two calls used for text retrieval. The GitHub release notes spend serious space on bfloat16 ranking ties and changed chat-message batching semantics too.

MultiVectorEncoder retains a sequence of vectors rather than pooling an input to one embedding. MaxSim takes each query token's best match among document tokens, then adds those maxima, making the final rank decomposable into token-level matches.

The official walkthrough says the loader accepts PyLate and Stanford ColBERT checkpoints, plus ColPali-style vision-language models and bare transformer backbones. A bare backbone receives a fresh projection layer, so it requires training; saved configurations retain prefixes, query expansion, and punctuation skip lists.

HierarchicalTokenPooling clusters a document's token vectors with Ward linkage and keeps roughly 1 / pool_factor of them. At pool_factor=2, the release example cuts 608,414 vectors to 305,438 and 311.5 MB to 156.4 MB, while reporting 100.6% of the unpooled BEIR result.

The same interface can encode a page as image patches, preserving charts and tables without an OCR step. ColQwen-Omni extends that pattern to text, images, audio, and video; tomaarsen's multimodal retrieval example describes a text query for “nausea” finding audio that says “carsickness” without transcription.

The integration includes four late-interaction losses, including cached and distillation variants, five evaluators, and a trainer that retains familiar arguments. Starting from ModernBERT-base, tomaarsen on MultiVectorEncoder training reports NanoBEIR mean nDCG@10 moving from 0.1338 to 0.4831 in about 25 minutes on one RTX 3090.

Multi-column losses now merge columns into one forward pass, which tomaarsen on training speed puts at roughly 1.25x faster for hard-negative and triplet training with identical loss trajectories. The same post calls fp16 plus FlashAttention the fastest GPU configuration, at 3.87x over fp32.

Version 6 raises the dependency floors to transformers v5 and PyTorch 2.2. It also changes similarity and similarity_pairwise from properties to methods, requires trust_remote_code=True for custom module classes including local paths, and treats a bare list of chat messages as one conversation rather than a batch.