Study attributes 60.33% of sampled GPU library failures to build or runtime environments

• Build and runtime problems accounted for 181 of 300 sampled GPU-library issues (60.33%), according to rohanpaul_ai's summary.
• Software incompatibility was the largest single root cause: 76 reports (25.33%) in the reported breakdown.
• Supported operations can still produce incorrect outputs when specialization assumptions fail, according to the linked paper.

The sample-selection process excluded ambiguous diagnoses and replaced them with better-documented reports. One CUTLASS case came down to sparse-matrix metadata arriving in the wrong memory layout.

Researchers at Microsoft Research Asia and Tsinghua University manually examined issue descriptions, discussions, diagnostic evidence and available fixes. Their repository-stratified sample contained:

• FlashInfer: 37 reports.
• FlashAttention: 28 reports.
• Triton: 77 reports.
• CUTLASS: 56 reports.
• DeepGEMM: 4 reports.
• xFormers: 19 reports.
• bitsandbytes: 79 reports.

The initial collection covered 9,099 issues created between December 26, 2015 and July 10, 2026. Screening produced 3,181 candidate closed reports, with each repository's final allocation proportional to its candidate pool.

Reports containing only symptoms, speculation or competing explanations were replaced with a report from the same repository whose creation date was closest. The percentages therefore describe selected, diagnosable closed reports; unresolved failures were outside the final sample.

The researchers assigned root causes to three dimensions in Table III:

• Kernel Library: 119 reports, 39.67%.
• Runtime Environment: 100 reports, 33.33%.
• Compiler and Build System: 81 reports, 27.00%.

Software incompatibility covered mismatches among drivers, CUDA, frameworks and packages. Hardware incompatibility accounted for the remaining 24 reports in the runtime-environment dimension.

Observable symptoms received separate labels from root causes. The symptom table breaks down the same 300 reports as follows:

• Deployment failure: 82 reports, 27.33%. Installation, dependency, linking or loading problems.
• Build failure: 73 reports, 24.33%. Source or generated code failed to become an executable artifact.
• Crash: 63 reports, 21.00%.
• Incorrect functionality: 53 reports, 17.67%.
• Performance degradation: 23 reports, 7.67%.
• Hang: 6 reports, 2.00%.

Deployment and build failures together accounted for 155 reports (51.67%), preventing the software from becoming ready for device execution. That symptom total measures a different classification from the headline's build-and-runtime root-cause total.

Specialized kernels impose implicit correctness constraints on four things, according to the abstract shared in the thread:

• Tensor shapes.
• Metadata layouts.
• Memory aliasing.
• Hardware resources.

The CUTLASS sparse-GEMM kernel required interleaved metadata in Gemm::LayoutE, but its caller supplied row-major metadata. The fix converted that metadata into a separate tensor with the required layout before passing it to the kernel.

The authors report that violations can produce incorrect outputs or launch failures even when the operation itself is supported. Operator support alone leaves these specialization-specific contracts unspecified.

The testing analysis found gaps in diagnosing low-precision numerical errors and hardware-specific execution failures. Two FlashAttention examples expose different traps:

1. A correct forward pass concealed broken gradients. A particularly nasty block-sparse attention bug matched the sparse reference in its forward pass, then returned finite but severely inaccurate gradients. Backward execution had lost the enabled-block information and computed gradients as though attention were dense. The repair saved the mask rule and backward block list during forward execution and passed both to the backward kernel, restoring agreement with an FP32 reference.
2. The reference introduced the discrepancy. In a separate report, gradients agreed with an FP32 reference but differed from an FP16 reference. The authors attributed the mismatch to rounding in the FP16 reference rather than a kernel fault.