OpenAI details how Habitat scaled Azure Cosmos DB for global demand
OpenAI has described how its Habitat online storage platform scaled Azure Cosmos DB to support more than 1 billion users across 40 geographic regions. Habitat provides fast data access for ChatGPT, the API, Codex and internal services, and OpenAI says the platform now serves more than 500 petabytes of data.
The second part of OpenAI’s engineering account focuses on the storage layer beneath Habitat. The company says its largest Azure Cosmos DB account stores tens of petabytes and handles more than 20 million queries per second. Its work with Azure led to a shared, multi-tenant container design intended to retain workload isolation while reducing capacity-planning work for product teams.
From partition limits to shared tenancy
Azure Cosmos DB containers provision throughput in request units per second, while data is sharded by logical partition key. OpenAI initially relied on a stated 10,000 RU/s limit per logical partition key in a default shared container. In practice, logical keys shared the limits of their underlying physical partition, allowing busy workloads to affect unrelated keys.
Storage growth introduced another constraint. When physical partitions split as data expanded, a fixed amount of provisioned container throughput was distributed across more partitions. For storage-heavy workloads, that could lower the effective throughput available to an active partition even when request rates remained stable.
OpenAI then created dedicated containers for critical services and major features. That limited the blast radius, but shifted capacity planning to individual product teams. A launch could move from internal testing traffic to 1 million QPS without the aggregate headroom available in a shared container, creating last-minute scaling work.
Azure changes enable a return to multi-tenancy
OpenAI and Azure addressed the earlier issues with hot-key throttling, which throttles the hottest keys before they consume a physical partition’s capacity, and request-tiering quality of service. Lower-priority bulk inserts and extracts can be throttled ahead of interactive traffic.
An improved Azure Cosmos DB serverless mode also allowed each physical partition to scale to 10,000 RU/s and added splits based on RU use as well as storage use. OpenAI says these changes let Habitat return to one shared container by default and eliminated provisioning and noisy-neighbor incidents for months while product development accelerated.
The companies also changed how they operate the service. A late-2024 regional outage showed that failover for OpenAI’s largest accounts could take several hours at that scale. A later failback triggered a connection stampede that overwhelmed an Azure Cosmos DB gateway. OpenAI and Azure now jointly monitor Habitat and physical storage layers, test against a production-sized non-production account, and collaborate on SDKs, gateway protection, metadata caching and connection reuse.
Lower-latency reads through placement, caching and Rust
Read requests account for more than 90% of Habitat traffic. OpenAI selectively replicates small, latency-sensitive datasets such as authentication metadata, while using a distributed Valkey lookaside cache close to request-processing infrastructure for other data. Azure Cosmos DB change feed propagates cache invalidations globally within seconds, helping maintain high cache hit rates with lower staleness.
OpenAI also rewrote the Habitat service in Rust to remove overhead from its original Python implementation. Combined with replication and caching changes, the company says these steps cut global read p50 latency from roughly 120 milliseconds to roughly 10 milliseconds.
For businesses building global data platforms, the practical implication is that isolation, capacity management, cache invalidation and regional placement must be designed together, with operational testing that reflects the scale of the production workload.