Will DePue reports Blackwell fingerprints in six Chinese model weights

• Six models from four Chinese labs carry Blackwell-compatible initialization patterns, according to the announcement by Will DePue, founder of independent research group Unemployment Labs.
• Physical Blackwell tests reproduced the patterns, and DePue reports in the original thread that his method matched disclosed hardware for all 24 reference models.
• The fingerprints establish initialization patterns, leaving chip location and export violations undetermined, as WesRoth notes.
• DeepSeek-V4.1-Flash remains unresolved: DePue described an unsuccessful search before suggesting he might have found a NVIDIA trace.

DeepSeek’s recovered seed is 953049583. PyTorch users had already documented GPU-dependent random-number layouts in September. DePue’s new replay code runs on a laptop.

DePue groups the six matches by streaming-multiprocessor count, or SM count, in his published results:

• DeepSeek-V4-Flash: 148 SMs, compatible with B200/B300.
• DeepSeek-V4-Pro: 152 SMs, compatible with GB200/GB300.
• Kimi K3: 152 SMs, compatible with GB200/GB300.
• Kimi K3.1-Pro: 152 SMs, compatible with GB200/GB300.
• MiniMax-M3: 148 SMs, compatible with B200/B300.
• Ling-3.1-flash: 152 SMs, compatible with GB200/GB300.

Earlier checkpoints from each lab read as Hopper. Kimi K3.1-Pro shares K3’s seed and may be a further post-trained checkpoint, according to the full report, so six matching releases need not represent six separate pretraining runs.

PyTorch’s initialization kernels distribute pseudorandom numbers across GPU threads. The seed determines the numbers; the launch geometry determines where they land in the tensor.

DePue’s method works backward through that arrangement:

1. Recover surviving initialization values. Embedding rows for tokens never encountered during training can preserve their starting values. Trained weights sometimes retain weaker correlations with their initialization.
2. Reconstruct the thread layout. In the configurations examined, sufficiently large initializations use 2,048 threads per SM. Hopper’s 132 SMs yield 270,336 threads; the two Blackwell layouts yield 303,104 and 311,296.
3. Search seeds and layouts together. The search spans all approximately 4.3 billion 32-bit seeds across candidate layouts.
4. Predict held-out weights. A recovered seed and layout must also match weights excluded from the search.

The cross-device difference appears once a tensor exceeds one wave of the launch grid, as the September PyTorch issue documents. Small tensors can agree across GPUs even when larger tensors diverge.

DePue reports these physical-hardware checks in the original results:

• DeepSeek-V4-Flash on B300: untouched starting weights reproduced bit for bit using unmodified PyTorch.
• The same model on H100: no matching untouched rows.
• Kimi K3 on GB200: correlation of approximately 0.999995.
• Seed incremented by one: the matches disappeared.

Bit-for-bit hardware replay is the report’s strongest evidence. The 24 disclosed-hardware comparisons also include DeepSeek-V3 and Kimi K2 on H800s, and NVIDIA’s Nemotron models on GB200s.

DePue says senior researchers at OpenAI and Anthropic checked the work privately; he subsequently sought experienced ML researchers willing to validate it publicly.

The report explicitly allows several paths after initialization:

• Weights can move to another cluster before training.
• An existing checkpoint can continue training on different hardware.
• A mixed NVIDIA and Huawei workflow can preserve an NVIDIA initialization pattern without exposing Huawei’s participation.

DePue argues that normal training jobs initialize on the GPUs already assigned to the run. For Kimi K3, he points to at least 64 separately seeded processes with the same Blackwell-compatible layout, plus the need to transfer a checkpoint exceeding five terabytes under the separate-initialization hypothesis.

The report leaves ownership, geographic location, procurement route and export-control violations undetermined. A fingerprint also cannot enumerate every chip used later in training.

A two-way-partitioned AMD MI300X can expose 152 compute units, matching the layout associated here with GB200/GB300. DePue’s published caveats also identify specially configured AMD hardware as an alternative for the 148-unit layout.

He says additional tests exclude AMD for DeepSeek-V4-Flash, while calling it very unlikely for the other five models. That alternative remains part of the hardware attribution for those five.

Even within NVIDIA’s lineup, these signatures identify groups: H100 SXM, H800 and H200 share the 132-SM layout. B200/B300 and GB200/GB300 likewise remain paired identifications.

Across roughly six trillion seed-and-layout combinations, DePue estimates the strongest accidental match would reach about 8 sigma. His reported strongest results for the six models range from 49 to 1,385 sigma; the weakest individual test reaches 20.

Only DeepSeek-V4-Pro rests entirely on traces in trained weights, he clarified in a response to statistical criticism. He rejected a simulation that put shared row structure in both vectors, arguing that his candidate Philox vector is independently generated.

Those scores address accidental numerical matches under his statistical model. Training location and subsequent hardware changes remain separate questions.

DeepSeek-V4.1-Flash still contained hundreds of apparently untouched rows, but DePue’s exhaustive stock-PyTorch searches found no matching seed across GPU layouts from 16 to 512 SMs or the CPU generator.

Ascend speculation appeared in one response, but DePue explained in a clarification that an unreadable signature leaves the hardware undetermined. A changed generator or seeding scheme could also defeat the search on NVIDIA hardware.

His later update was tentative: he wrote that he might have cracked it and that it “might be NVIDIA.” That post supplied no recovered seed or layout.

DePue revised the report’s compute estimates on November 12. The updated figures replace the announcement’s estimate of 2,000 to 3,000 Blackwells for the largest runs:

• DeepSeek-V4-Pro: best estimate of 3,000 GPUs, with a range of 750 to 6,500.
• Kimi K3: best estimate of 4,500 GPUs, with a range of 1,000 to 9,500.
• Five distinct runs combined: approximately 12,000 GPUs, with a range of 7,000 to 22,000.

The revised largest-cluster cost estimate is $200 million to $300 million, replacing $150 million to $200 million. The revision uses lower utilization assumptions than NVIDIA’s benchmarks, roughly 20% of peak for most runs and 25% for DeepSeek.

These calculations assume training stayed on the initialization hardware. Token counts, utilization and run duration also enter the estimate; halving the assumed duration doubles the required GPU count.

The policy interpretation drew a separate objection: teortaxesTex argued in a reply that continued capability gaps could be consistent with export restrictions having an effect.

Blackwell’s attraction may extend beyond pretraining. The commenter teortaxesTex argued in a follow-up that modern agentic reinforcement learning places stronger demands on the hardware than pretraining alone.

A later discussion tied that claim to per-chip throughput and rollout staleness: long agent trajectories can lag an advancing trainer, producing increasingly off-policy experience.

By November 13, DePue had published the reproduction repository, following his earlier promise to release code.

The package contains:

• 7 MB of extracted embedding rows, checked against hashes in models.json.
• replay.py, which reconstructs rows across SM counts from 16 to 256, reports the best layout, and compares the next-best layout and seed-plus-one control.
• gpu_check.py, which delegates generation to stock PyTorch through torch.cuda.manual_seed and normal_.

The small replay covers four Blackwell-positive models plus DeepSeek-V3 as a Hopper control. DeepSeek-V4-Pro’s trained-weight-only analysis falls outside that row-replay package.

The README’s CPU replay invocation is:

REFETCH=1 python replay.py downloads the rows again from pinned Hugging Face revisions and checks their hashes.