OpenAI scales Habitat storage past 70 million requests per second
OpenAI says its Habitat online storage platform now handles more than 70 million requests per second for products used by more than 1 billion people each week. The platform serves more than 500 petabytes of data across almost 40 geographic regions and underpins data access for ChatGPT, the API, Codex and internal services.
Habitat started in mid-2024 as a Python client-side library connected to Azure Cosmos DB. It was designed to hide operational details from product teams, including schema lookup, routing, authorization, encryption, serialization, request shaping and connection pooling. As OpenAI expanded, however, the library model made coordinated protocol changes across dozens of services increasingly difficult.
From client library to central storage service
OpenAI moved Habitat into a standalone service by the middle of 2025. The shift created a central control point for deployments, observability and platform improvements, while allowing access controls, audit logging and limits on access to underlying storage resources to be enforced in one place.
The company describes a regional Azure Cosmos DB routing change that required feature flags and coordinated client rollouts as an example of the old model’s fragility. Deployments, shadowing work and fixes took days, and an unrelated rollback restored a buggy client version. A service reduced that operational fan-out and made changes available centrally.
Making Python work at high request rates
OpenAI initially retained Python to establish its APIs and infrastructure quickly, despite its overhead in a high-throughput service. Habitat performs I/O proxying as well as routing, compression, encryption, checksumming, health checks, request shadowing and hedging. The company found that asyncio scheduling delays could dominate tail latency when CPU-heavy work delayed response-processing coroutines.
To manage this, OpenAI monitored event-loop delay alongside conventional resource metrics, kept concurrency per process low and scaled out worker processes. CPU profiling also identified synchronized JSON parsing of Statsig feature-flag configurations as a source of stalls. The team deployed smaller targeted configurations, extended the refresh interval and added jitter to background tasks.
Connection pooling created a separate load-distribution problem. OpenAI found that aiohttp TCPConnector’s default LIFO reuse could feed more traffic to slower, overloaded processes after a burst. Changing reuse to FIFO broke that feedback loop. The company now relies largely on Istio and Envoy for pooling and server-load-aware balancing, while Envoy upgrades Python HTTP/1 connections to HTTP/2, pools connections, and provides rate limits and circuit breakers.
Constrained requests and a Rust migration
Habitat limits its online interface to a simple NoSQL API rather than arbitrary SQL. OpenAI says this keeps requests predictable and avoids unbounded operations that can overload the platform. Complex querying is handled through an offline secondary view in Rockset, fed through change data capture, so analytical and search workloads remain isolated from online storage.
At its Python peak, Habitat served more than 20 million requests per second. In Q2 2026, OpenAI says two engineers, Codex and GPT-5.5 rewrote the service in Rust. Rust now handles 95% of production requests, and OpenAI reports 6x better CPU efficiency, 15x better memory efficiency, and lower average and tail latencies than the Python version.
For businesses building high-volume data services, Habitat illustrates the practical value of centralizing control, constraining expensive request patterns, measuring scheduling delays and treating connection behavior as part of reliability engineering.