FineDNA releases 514B training tokens and 300,000 biology-model tasks

• FineDNA opens a three-stage training pipeline: the release announcement covers 514.30B mid-training tokens, 203,923 SFT conversations and 98,433 RL prompts.
• Code-built variant explanations reached about 67% accuracy versus below 60% for self-distilled traces in the authors' run comparison, which also varied other mixture details.
• Carbon-X, derived from Qwen3-14B, reached a 66.7 five-benchmark average in the launch results, versus 40.0 for its starting model and 67.5 for GPT-6.1 Sol.

The team mined DNA-bearing papers and web pages with a filter that rejects nucleotide matches inside LaTeX braces. One rewarded answer put BRCA2 on the wrong chromosome; the final RL recipe includes a penalty for ungrounded rationales.

FineDNA's dataset card separates the available corpus from the weighted training run:

1. Mid-training: 23 mid_* source configurations containing 514.30B tokens. Carbon-X drew 302B training tokens with weighted sampling, including repeated passes over some sources.
2. SFT: 203,923 conversations across eight domains. sft_mix contains the trainer's exact mixture without additional weighting.
3. RL: 98,433 rows covering 80,406 distinct prompts. rl_mix preserves training order and deliberately repeats some slower-learning task groups.

The authors published the complete recipe in huggingface/carbon. FineDNA is Apache 2.0, while its third-party mid-training sources retain their own licenses and terms.

The team tested three frontier teachers on 1,000 benign-or-pathogenic variant classifications at high or maximum reasoning effort.

In one worked example, Opus's visible explanation admitted it could not identify the gene or recall a ClinVar record, then supplied the wrong classification.

The self-distillation attempt ran RL with verifiable rewards, retained correct rollouts and used those explanations as SFT targets.

A correct label was a lousy proxy for a grounded explanation.

The team assembled variant-effect training explanations from two inputs:

• Facts computed in code: the changed sequence block, base substitution, local sequence context and CpG changes.
• The mid-trained model's likelihoods: how unusual the alternate base looked relative to the reference, converted into verbal categories.

On the 1,000 held-out variants used for this comparison, grounded-trace SFT reached 66.7%; the self-distilled run finished at 59.3%. Other mixture details also differed, limiting causal attribution of the gap to trace construction alone.

The authors chose the DNA-English balance through mixture ablations. The 302B-token run used these sampling weights:

• 55% raw DNA
• 20% English
• 15% biology text
• 10% instruction data

Those weights differ sharply from the available corpus, which contains 310.22B English tokens and 181.96B DNA tokens.

The mid-training recipe used a cosine learning-rate schedule from 5e-5 to 5e-6. Lower-rate ablations preserved English but left DNA performance at chance; higher rates learned DNA while forgetting English.

The team sampled 32 rollouts for each of 74,484 variant prompts, selecting a learnable bucket with 2 to 30 successful completions.

After label balancing, the usable pools contained 952 and 7,200 prompts, respectively. Under the same RL recipe, the starting Qwen3-14B stayed around chance on FrontierVEP while the mid-trained model continued improving.

Carbon-X's training-stage results average NextBase, FrontierVEP, GenomeQA, BioReason Coding and BioReason Non-SNV:

• Mid-training plus SFT: 40.0 → 59.3, +19.3 points.
• RL: 59.3 → 66.7, +7.4 points.
• Final comparison: Carbon-X's 66.7 trails GPT-6.1 Sol's 67.5 by 0.8 points.

For BioReason, the team removed a label leak and gene names from prompts. Training data underwent DNA-span overlap, text n-gram and sequence-homology checks, according to the evaluation notes.

Comparison models used their best-average reasoning setting; the team also reported sharply effort-dependent refusals on NextBase.

For broader biology requests, the authors favored GPT-6.1 Sol, citing its lower tendency to refuse legitimate prompts in their testing.

Carbon-X's 14.8B-parameter weights fit on one 80 GB GPU in bfloat16, according to its model card. The team still describes specialized DNA models as useful for cheap whole-genome scoring.

• Tokenizer: text BPE and non-overlapping DNA 6-mers share one decoder. A 6-mer containing an ambiguous character such as N becomes one unknown token.
• Loading: Transformers v5 or later; the custom tokenizer requires trust_remote_code=True.
• Context: 40,960 tokens accepted, versus 8,192-token mid-training sequences and post-training examples shorter than 16k tokens. Much longer inputs remain unevaluated.
• Errors: base counts, coordinates and sequence edits can still be wrong. GenomeQA accuracy was 29.8% on splice-site questions and 34.6% on histone marks.
• Clinical status: the model has not been clinically validated.