EmbeddingGemma 2 maps text, code, images, video and audio into one embedding space
A technical breakdown describes EmbeddingGemma 2’s shared embeddings for text, code, images, video and audio. It details staged distillation from Gemini Embedding 2, Matryoshka dimensions and int4 training.

TL;DR
- One embedding space covers text, code, images, video and audio, including cross-modal search, as gabriberton explains.
- A small Gemma 4-derived text stack sits behind optional media encoders, with the architecture outlined in gabriberton's breakdown.
- Training progresses from text-only distillation to multimodal distillation, then contrastive learning; the training summary also describes Matryoshka embeddings down to 128 dimensions.
A 130M-parameter shared transformer does the cross-modal heavy lifting, alongside a 141M-parameter text embedder. Google's model card buries an annoying deployment gotcha: float16 can return NaNs or silently degraded embeddings.
Shared backbone and optional encoders
Google's October 6 developer guide introduced the model under Apache 2.0. Beyond comparing separate media items, EmbeddingGemma 2 can turn an interleaved sequence of text and media into a single vector, according to the technical report.
The architecture described in gabriberton's breakdown separates into three paths:
- Text and code: A 271M-parameter stack initialized from Gemma 4, using bidirectional attention and tuned for embeddings.
- Images and video: A 167M-parameter vision transformer, co-trained with the text stack. The paper reports 439M parameters for the vision-language configuration.
- Audio: A 305M-parameter USM-based Conformer, kept frozen during training, as noted by gabriberton. The paper reports 577M parameters for the audio-language configuration.
Google rounds the complete model to 740M parameters in its developer documentation; the paper reports 744M. Unused modality encoders can be omitted from memory.
Media encoders produce soft tokens that are projected into the transformer's 512-dimensional hidden space. The output sequence is mean-pooled, projected to 768 dimensions and L2-normalized.
Three training stages
Gemini Embedding 2 supplies the teacher embeddings for the first two stages. Google separates modality expansion from the final change in learning objective:
- Text-only distillation: Learn from the teacher using text inputs.
- Multimodal distillation: Expand training to images, video, audio and documents.
- Contrastive finetuning: Drop distillation and combine contrastive learning with a spread-out regularizer that encourages embeddings to occupy the space more evenly.
The paper adds same-tower negatives to the contrastive objective to prevent queries and targets from clustering into separate subspaces. It also masks duplicate examples that would otherwise become false negatives.
Matryoshka dimensions
Matryoshka training enters during the final stage in gabriberton's account. It makes leading slices of the 768-dimensional vector usable as shorter embeddings, with L2-normalization applied again after truncation.
Google's model card reports the following full-precision results:
| Dimensions | Vector storage relative to 768d | MMEB-v2 overall score |
| --- | --- | --- |
| 768 | 100% | 59.01 |
| 512 | 67% | 58.38 |
| 256 | 33% | 56.24 |
| 128 | 17% | 45.65 |
The storage ratios describe vector payloads at the same numeric precision, excluding index overhead. Google characterizes 128-dimensional embeddings as primarily suited to text-only workloads.
Checkpoint averaging and int4
Google finishes with two additional operations:
- Checkpoint averaging: Take an unweighted average of the final finetuning checkpoint and a significantly earlier checkpoint from the same run. The authors report reduced degradation on tasks that overfit, alongside further gains on tasks already benefiting from finetuning.
- Int4 quantization-aware training: Produce low-precision versions through additional training, rather than only quantizing the finished weights afterward.
Shared token budget
All modalities consume the same 8,192-token context window. The model card specifies these default costs:
- Images: 280 soft tokens per image.
- Video: 140 soft tokens per sampled frame, with sampling defaulting to one frame per second.
- Audio: 25 soft tokens per second, with input supplied as mono audio at 16 kHz.
The supported vision budgets are 70, 140, 280, 560 and 1,120 tokens per image or frame. The paper used a 32-frame video budget during training and evaluation, and can interleave frames with soundtrack segments and textual timestamps.