Nowadays, most of us interact with frontier AI models in a structured way, even if we do not always think about it that way. Most systems already sit on top of a protocol, often created by the model providers themselves. OpenAI and Anthropic, for example, each provide APIs to communicate with their inference systems.

In practice, one approach is to use an OpenAI-compatible API that can serve a multi-turn agent. We can point the official OpenAI SDK at a compatible inference endpoint, hold a conversation with the agent, and use the same interface across turns.

These systems are built in a specific, clever way that lets them scale. When traffic grows or subagents spawn, these systems need to be scaled so that we, as engineers, can make necessary adjustments without users noticing. For example, we may scale a single-replica system to three replicas behind a load balancer when we need additional traffic support, while still retaining the proper conversation context and previous turns.

“The microservices principles everyone stopped talking about five years ago turn out to be exactly what agent traffic needs.”

Many of the questions we need to ask ourselves before developing these systems were already addressed by the 2010s-era microservices literature. We will show how to rebuild an agent-friendly API from those principles, backing it with Oracle AI Database Free, and measure what changes.

Key insight: The load shape is different, and that difference is exactly what a stateful design struggles to absorb.

Point 1 is the one that people usually understand: a browser session survives on one server thanks to stickiness and human think time.

Source link