New telemetry service takes down OpenAI’s API, ChatGPT and Sora for hours
A new telemetry service caused every node in OpenAI’s clusters to make heavy Kubernetes API calls, which took down the control plane and, with it, all OpenAI services.
- Occurred
- Disclosed
- Published
- Organization
- OpenAI
- System
- OpenAI API, ChatGPT and Sora
- Impact
- All OpenAI services degraded or unavailable from 3:16 PM to 7:38 PM PST
- Root cause
- Configuration of a new telemetry service overwhelmed the Kubernetes control plane
Timeline
- 3:16 PM PST. Services begin to degrade.
- 5:36 PM PST. The API substantially recovers.
- 5:45 PM PST. ChatGPT substantially recovers.
- 7:01 PM PST. ChatGPT and Sora fully recover.
- 7:38 PM PST. The API fully recovers across all models.
Trigger or entry point
The rollout of a new telemetry service to OpenAI’s Kubernetes clusters.
Failure mechanism
According to OpenAI’s write-up, the service’s configuration made every node in each cluster run resource-intensive Kubernetes API operations. This overwhelmed the API servers and took down the control plane in most large clusters. Service discovery relied on DNS, which in turn depended on the control plane, so services could no longer find each other. Cached DNS records hid the problem for about 20 minutes, until they expired.
Impact
All OpenAI services were affected for a little over four hours.
Response
The fix was to remove the new service, but that required access to the control plane, which the load prevented. OpenAI describes remediation as very slow because of this lock-out effect. Some clusters needed manual intervention.
Mitigations
OpenAI lists five measures:
- phased rollouts with monitoring of control-plane health,
- fault injection tests, including tests of bad rollouts,
- emergency access to the Kubernetes API servers,
- decoupling the data plane from the control plane,
- faster recovery through better caching and rate limiting.
Sources
- API, ChatGPT & Sora Facing Issues (incident write-up)primary · 2024-12-11
- OpenAI blames its massive ChatGPT outage on a ‘new telemetry service’secondary · 2024-12-13
Corrections
No corrections have been made to this entry.
Report a correction