OpenAI automates ChatGPT performance regression workflows with Codex

OpenAI has described the Performance Factory, a cloud-based system that uses Codex to investigate and help resolve ChatGPT performance regressions. Its first production workflow, the Regression Workflow, tracked more than 1,700 investigations and root-caused more than 600 cases in just over three months—an average of 17 automated investigations a day.
The workflow starts when a configured performance alert fires. An agent investigates the slowdown, saves findings to a persistent task, and proposes a fix for an engineer to review. After deployment, another agent checks whether performance has returned to its previous baseline. OpenAI says the workflow automates the path from investigation through fix authoring and recovery verification, while engineers retain authority over what ships.
A task system for agentic performance work
The Performance Factory combines four components: a task service with configurable workflows, a dispatch loop that assigns ready work, an internal API that runs Codex in the cloud, and subscriptions that react to events such as GitHub review comments or completed deployments.
Engineers define task states, permitted transitions and human-required steps in configuration files, with Markdown prompts describing each agent step. Tasks keep their status and findings independently of a particular agent session. This allows work to pause for code review or a deployment, then resume with the relevant context available to an agent or engineer.
Codex runs in an environment inside OpenAI’s network with access to code repositories and observability tooling, including metrics, logs, traces, CPU profiles and memory snapshots. The agents operate under service identities, enabling permission scoping and action tracing. For broader context on how Codex-based agents are changing engineering practice, Codex agents reshape engineering workflows outlines the shift toward workflows where people focus on judgment and oversight.
Diagnosis is checked before a fix reaches review
For an alert, the investigating agent identifies which metric deteriorated, when the change began and which requests were affected. It then examines deployments, feature-flag changes, code, logs and traces to test possible explanations. OpenAI notes that an alert can occur hours after a regression actually began, making this temporal analysis important.
OpenAI added an adversarial review because early agents sometimes accepted the first plausible explanation. A separate Codex session attempts to disprove the diagnosis by checking assumptions and counterfactuals. In a manual review of 100 root-cause suggestions, OpenAI graded 70% as accurate; the remaining 30% needed refinement or did not fully establish causality.
Once a diagnosis clears review, Codex can implement a proposed fix and open a pull request. Engineers review the change and decide whether to merge it. OpenAI also applies adversarial review to proposed fixes, since a change may be unnecessarily complex, introduce issues or require architectural trade-offs such as accepting stale data when adding a cache.
Business implication: automate the repeatable evidence work
OpenAI evaluates investigation skills against historical incidents and assesses summaries separately, while a daily agent looks for tool and access failures in recent runs. The practical implication for service teams is to make recurring incident work explicit and stateful: automate evidence collection, diagnosis checks and post-release verification, but reserve engineering attention for risk, trade-offs and deployment decisions.

