Stripe reports Shield scaling improves fraud recall by more than 8%

• Shield’s fraud recall improved more than 8%, according to the announcement; a follow-up clarified that precision stayed fixed.
• Training epochs fell from 24 to 4.8 hours without shrinking the dataset, according to the training breakdown.
• Agents now execute experiments from data exploration through evaluation, using documented operational knowledge, as described in the experiment workflow.

The gem in Stripe’s engineering write-up is a production rollback after late fraud reports overturned an apparent model improvement. Shield also grew to roughly 30 million parameters, triple its previous size, without increasing serving latency.

Shield is Radar’s core transaction-fraud model, as a reply clarified. Its earlier training pipeline kept every known fraudulent payment but randomly sampled the remaining payments, leaving roughly 800 million transactions.

Removing that downsampling expanded a nine-month training window to 11 billion transactions. The larger model surpassed the baseline by its third epoch and initially looked better in a production experiment.

Then the labels caught up:

• Cardholders can take weeks to discover and report fraudulent payments. Recent transactions therefore have incomplete fraud labels.
• Removing downsampling added about 20 times more fraudulent payments labeled as legitimate, while the number of known fraud examples stayed unchanged.
• The newest training weeks contained the most immature labels, teaching the model that some late-reported fraud patterns were legitimate.

As reports arrived, measured performance fell below the earlier model’s. Stripe rolled back to a model trained with the earlier sampling strategy.

Stripe kept the 11-billion-example dataset fixed while cutting epoch time by 80%. Its optimization sequence exposed successive bottlenecks:

1. 24 → 12 hours: Stream data from S3 into memory instead of downloading it to disk for data loaders to read. Disk throughput had been starving the GPUs.
2. 12 → 7 hours: Compile tensor operations with torch.compile, use lower-precision arithmetic, execute Shield’s parallel ResNeXt branches concurrently, and improve CPU-to-GPU transfers.
3. 7 → 4.8 hours: Reduce peak training memory from 1.3 TB to 500 GB, enabling more data-loading workers and prefetching. Compress communicated gradients to BF16, tune communication buckets, and use asynchronous CPU-to-GPU transfers.

Stripe calls its internal tool-and-instruction set Agentic MLE. Agents operate inside existing ML infrastructure across six parts of the experiment loop:

• Explore data.
• Implement features.
• Backfill historical values.
• Train models.
• Debug failed jobs.
• Evaluate against a baseline.

Reusable platform procedures and model-specific documentation supply the operational knowledge. Historical backfills must reconstruct what was knowable at payment time; agents also need procedures for diagnosing distributed training failures before retrying them.

Feature experiments use Shepherd, which computes historical training features and serves online equivalents from the same definitions.

Engineers review proposed experiments and results at defined checkpoints, retaining three responsibilities:

• Set priorities.
• Review evidence.
• Decide what ships.

After a fraud attack slipped past Shield, agents used model explanations to identify features pushing fraudulent payments toward low risk scores, as described in the account of the attack. Humans helped prioritize 38 experimental ideas.

Several agents proposed changing stochastic weight averaging, or SWA, which combines parameter snapshots from a training run into one model. Shield already used SWA, starting halfway through training; falling training loss alongside rising validation loss prompted experiments with earlier snapshots.

The experiments split on the metric that mattered:

• Early stopping alone: Improved overall ranking but reduced recall within Stripe’s narrow false-positive range.
• SWA starting at epoch 0, with an eight-epoch run: Produced the best results in the controlled comparison.

That configuration delivered the reported recall gain above 8%, with precision held fixed according to the clarification. Stripe estimates that the change prevents tens of millions of dollars a year in additional fraud.