OpenAI launches Codex investigations from performance alerts
OpenAI's Performance Factory launches Codex investigations from performance alerts using a shared forensics skill. It tracked over 1,700 investigations and found root causes in more than 600 cases.

TL;DR
- Performance alerts now launch cloud Codex investigations, with OpenAIDevs reporting more than 1,700 investigations tracked and 600 root causes identified in just over three months.
- A shared
perf-forensicsskill supplies the investigation playbook, while OpenAI's task workflow preserves findings between agent runs and human reviews. - Diagnosis still outruns remediation: rohanpaul_ai highlighted that only about 15 Codex-authored fixes shipped unchanged.
- Deployment triggers another investigation step: OpenAI's recovery workflow checks production performance before closing the task or reopening the investigation.
One adversarial reviewer rejected an image-download diagnosis because the blamed service had already recovered while failures continued. The nicest engineering detail is in the agent handoffs: task records preserve reproducible queries rather than raw profiling dumps.
perf-forensics
OpenAI's performance team accumulated more than 30 specialist tools before encoding its investigation process in perf-forensics. The skill guides Codex through three questions:
- When did performance deteriorate? The slowdown can begin hours before a metric crosses the alert threshold.
- Which requests changed? Comparable requests getting slower and a shift toward inherently slower requests point to different causes.
- Which changes reached those requests? Codex checks deployment and feature-flag timing against the affected traffic, then tests explanations against code and telemetry.
The team refined the skill against historical regressions. For the latency and availability problems it targeted, Codex could then return a diagnosis from an alert without an engineer steering each step.
Four-part orchestrator
The Performance Factory architecture separates workflow state from agent execution:
- Task service. Engineers define states, allowed transitions, and steps requiring human involvement in configuration. The example pairs
config.tomlwith Markdown prompts such aswork/implementing.mdandwork/reviewing.md. - Dispatch loop. A periodic check finds ready tasks and starts a Codex turn with instructions for the current state. The agent records progress and advances the task when its step is complete.
- Cloud execution. An internal API invokes Codex inside OpenAI's network with repository and observability access. A service identity scopes permissions and makes performance-agent actions attributable.
- Subscriptions. Review comments, failed CI checks, and deployments can wake waiting tasks. No agent needs to remain running through a human review.
Query-based handoffs
Agents leave free-form notes alongside structured metadata, including when a regression began and when its cause was identified, OpenAI's performance team told VentureBeat.
The task retains relevant metrics, queries, and high-level findings instead of copying large profiling datasets. A successor agent receives that record and can rerun the queries to check the evidence.
Adversarial review
Early investigators sometimes stopped at the first plausible explanation. OpenAI added a separate Codex session instructed to disprove the diagnosis, using counterfactual checks such as:
- If a rollout caused the slowdown, did comparable requests outside the rollout stay healthy?
- If the suspected change was rolled back, did the affected metric recover?
In the manual assessment described in OpenAI's report, 70% of 100 root-cause suggestions were accurate. The remaining 30% needed refinement or had not fully established causality.
Separately, the performance team told VentureBeat that adversarial reviewers currently return about 25% of diagnoses for more investigation. Reviewer feedback had already helped the team improve the original investigator's instructions.
Fifteen shipped Codex PRs
OpenAI supplied a shipping breakdown in comments to VentureBeat that its launch post omitted:
- More than 600 investigations reached a root cause.
- Around 400 progressed to a proposed fix.
- Approximately 50 shipped fixes were influenced by or matched agent suggestions.
- Around 15 shipped fixes were the exact pull requests authored by Codex.
Agent proposals sometimes introduced bugs or unnecessary complexity. OpenAI added adversarial review to proposed patches as well as diagnoses; architectural tradeoffs, such as a cache improving latency while introducing stale data, still require engineering judgment.
OpenAI also reported 79% autonomous success for monitoring training runs versus 25% for writing inference and serving code in a separate research-acceleration study highlighted by TheRealAdamG.
Production recovery
After an engineer deploys a fix, another agent compares performance with its earlier baseline. The branches in OpenAI's recovery process are explicit:
- Full recovery: mark the task resolved.
- Partial recovery: resolve the original regression and open a new task for the remaining slowdown, which may have another cause.
- No improvement: record the outcome and return the regression to investigation.
Impact-ranked reports
Agents could generate investigations faster than engineers could read them. OpenAI changed the handoff described in its engineering account to:
- Link related incidents across alerts and channels.
- Group findings by relevant metrics and rank estimated impact.
- Summarize the regression, its impact, the root cause, and the proposed fix.
- Include metric time series, CPU flame graphs, and traces so reviewers can inspect the supporting evidence.
Engineers can edit a proposed PR directly or use the diagnosis to write their own fix. PR comments can wake another agent to address feedback.
Historical evals
OpenAI tests investigation quality separately from summary usefulness in its evaluation process:
- Fresh investigators: each historical case starts with the original signal, stripped of answer-revealing details. Agents cannot read incident writeups, saved investigation memories, or previous runs' results.
- Evidence boundaries: metrics and traces can expire, while telemetry collected after recovery can reveal the answer. The team inspects transcripts for information that spoiled the test.
- Separate scores: onset, impact, and affected requests are evaluated independently from causal reasoning. Correct timing does not establish that the blamed deployment caused the regression.
- Summary backtests: saved task snapshots let a separate grader assess readability, retained charts, and whether the proposed next step follows from the evidence, without depending on expiring telemetry.
Daily self-repair
An agent runs every day to scan recent messages and logs for tool failures and access errors, according to OpenAI's engineering account. When the same tool error blocks multiple investigations, it creates a repair task inside the Performance Factory.
If the cause is clear, an agent proposes a fix for an engineer to review and merge. Post-merge evals check whether investigators can now get past the step that previously blocked them.