OpenAI Habitat Shows Why Storage APIs Should Be Boring

OpenAI published the first part of an engineering write-up on Habitat, the online storage platform behind products such as ChatGPT and Codex. The headline numbers are large: more than 70 million requests per second, over 1 billion weekly users, nearly 40 regions, and more than 500 PB of data.

The more useful part is not the scale by itself. It is the shape of the choices OpenAI describes: move a fragile client library into a service, keep the online API deliberately constrained, measure Python event-loop delay directly, use Envoy to reduce connection pressure, and only then move the service from Python to Rust when the platform had stabilized.

What Changed

Habitat started as a Python client-side library for product teams that needed simple storage access without managing database details directly. It handled schema lookup, routing, authorization, encryption, serialization, request shaping, and connection pooling while using Azure Cosmos DB underneath.

By mid-2025, that client-side approach had become operationally brittle. A change to regional routing for critical data required coordinated library rollout across dozens of services, shadowing logic, bug fixes, and feature-flag sequencing. One unrelated rollback to an older client could put the platform back into a state the migration had already tried to avoid.

OpenAI moved Habitat into a standalone service so storage logic, deployment, observability, access control, audit logging, and platform improvements had a central control point. That service was still written in Python at first. OpenAI says the Python version eventually served more than 20 million requests per second, but it was expensive in CPU, memory, and tail latency. In Q2 2026, two engineers used Codex and GPT-5.5 to rewrite the service in Rust. The Rust version now handles 95% of production requests and is reported as 6x more CPU efficient and 15x more memory efficient than the Python version.

Why We’re Paying Attention

Most small teams will never operate at OpenAI’s request rate. The lesson still travels well because the failure modes are familiar at normal scale.

Shared client libraries feel fast early on. They let teams move without standing up another service. But once the library owns policy, routing, telemetry, caching, and migration behavior, every important platform change becomes a distributed deployment problem. If dozens of callers must upgrade before the platform can behave differently, the library is no longer just a helper. It is operational coupling.

The other lesson is API shape. Habitat deliberately avoids arbitrary online queries. Instead of letting clients issue expressive SQL that can hide table scans, joins, or unpredictable fanout, it exposes a constrained NoSQL-style API around predefined object and edge types. OpenAI’s argument is practical: simple, predictable, constant-work requests are easier to isolate, load-balance, and scale. Complex read patterns get an escape hatch through change data capture into separate Rockset instances, keeping analytical and search workloads away from the hot online path.

That is a good design instinct for AI products too. Agents, chat products, RAG systems, and workflow tools all accumulate small reads. If each user action fans out into hundreds of storage calls, the storage layer has to make bad calls hard to express.

How It Works

The Habitat pattern is a storage facade with a narrow online contract:

Product service
  -> Habitat service
    -> auth, routing, encryption, serialization, request shaping
    -> cache or online store
    -> change data capture for offline/query-heavy views

The important boundary is that the online path optimizes for predictable work. Product teams can store and fetch defined objects and direct edges, but Habitat does not try to be a general graph traversal engine. If a feature needs deeper traversal, analytics, or search, that workload moves to a secondary view built from CDC.

OpenAI also calls out a Python-specific operational point that is easy to miss: asyncio concurrency is not CPU parallelism. Habitat handled I/O-heavy proxying, but it also did CPU-heavy work such as routing, compression, encryption, checksumming, downstream health checking, request shadowing, and hedging. At high utilization, ready coroutines could sit behind CPU work and create tail latency even when the downstream storage call had already completed.

A Small Useful Test

This post does not need a sample repo. The useful example is a small measurement teams can add to an existing Python service that relies on asyncio.

import asyncio
import time


async def watch_event_loop_delay(interval_seconds=0.1):
    expected = time.perf_counter() + interval_seconds

    while True:
        await asyncio.sleep(interval_seconds)
        now = time.perf_counter()
        delay_ms = max(0.0, now - expected) * 1000
        expected = now + interval_seconds

        # Replace this print with your metrics client.
        print(f"event_loop_delay_ms={delay_ms:.2f}")

That metric will not explain every latency problem, but it answers one important question: is the service slow because the downstream is slow, or because the event loop is too busy to resume work on time?

Pair it with a short review of the online storage API:

Hot-path storage review

Can one request trigger unbounded reads?
Can clients express joins, scans, or graph traversal on the online path?
Are query-heavy views separated from write-serving storage?
Can routing and authorization change centrally, or does every caller need a library rollout?
Do background refresh tasks have jitter and bounded config size?
Are connection pools load-aware, or can reuse concentrate traffic on slow workers?

For most teams, this review is more valuable than copying Habitat’s exact architecture. The goal is to find places where cheap code paths can create expensive production behavior.

Cost And Operational Notes

Habitat is internal OpenAI infrastructure, not a product to adopt. There is no sample repo or new service to sign up for here.

The cost lesson is architectural. A client library may be cheaper to build, but a service can be cheaper to operate once policy and routing need to change quickly. A powerful query API may be nicer for developers, but a constrained API can keep hot-path costs predictable. Python may be the right first implementation, but only if the team measures its event loop, CPU cost, connection behavior, and tail latency honestly.

The Rust rewrite is also worth reading carefully. OpenAI did not present it as the first move. They accepted Python’s short-term inefficiency to stabilize the platform, then rewrote once the API and operating model were clearer. That sequencing is the useful part for smaller teams: do not prematurely rewrite a moving target, but do not confuse temporary technical debt with a permanent operating model.

What We’d Watch Next

OpenAI says a second post will cover multi-tenancy reliability, layered read-performance optimization, and its Azure Cosmos DB partnership. That is the part to watch if you operate shared storage for many internal teams, because the hard questions are usually isolation questions: which tenant gets slowed down, which workloads are allowed to burst, and how the platform proves one feature cannot starve another.

The short version: Habitat is a reminder that good storage platforms are often boring on purpose. They make the cheap path obvious, the expensive path explicit, and the operational control plane central enough to change under pressure.