How we optimized Azure Cosmos DB in our application storage platform, Habitat, to manage unprecedented growth

By Jon Lee, Hrachya Hakobyan, and Chaomin Yu, Members of Technical Staff

Share Making multi-tenancy reliable at scale Making multi-tenancy reliable at scale A simplified primer on Azure Cosmos DB containers Phase I: Relying on Azure Cosmos DB primitives Logical partition key limits Noisy neighbors Weak partitions Phase II: Trying workload isolation and single-tenancy Lack of stable aggregate growth Capacity planning Phase III: Using multi-tenancy by default, with data movement capability Hot key throttling Quality of Service (QoS) Serverless mode Better reliability as co-managed services Improving performance at a global scale Store data near users when capacity allows Broadly replicate only the smallest and most latency-sensitive datasets Rely on integrated caching for everything else Reduce proxy overhead with Rust Scaling Change Data Capture Looking ahead Making multi-tenancy reliable at scale A simplified primer on Azure Cosmos DB containers Phase I: Relying on Azure Cosmos DB primitives Logical partition key limits Noisy neighbors Weak partitions Phase II: Trying workload isolation and single-tenancy Lack of stable aggregate growth Capacity planning Phase III: Using multi-tenancy by default, with data movement capability Hot key throttling Quality of Service (QoS) Serverless mode Better reliability as co-managed services Improving performance at a global scale Store data near users when capacity allows Broadly replicate only the smallest and most latency-sensitive datasets Rely on integrated caching for everything else Reduce proxy overhead with Rust Scaling Change Data Capture Looking ahead All OpenAI products require fast, reliable access to data, provided by our storage platform, Habitat. It has to respond in milliseconds to over 1 billion users across 40 geographic regions. Our infrastructure faced an extraordinary challenge: supporting explosive growth and new use cases while continuing to improve reliability, performance, and product developer velocity. Solving that challenge required rethinking how our online storage platform was designed and operated. We constantly evaluated tradeoffs, following the pattern of: first become reliable enough for mission-critical product traffic, then fast enough for global users, and, finally, to deftly operate at massive scale