Respan released an open-source AI gateway written in Rust on Tuesday, after spending two years building more than 100,000 lines of wrapper code around LiteLLM.

The new gateway replaces that system with a small core and plugins that handle model providers, routing, MCP tools, realtime voice, and security controls.

The breaking point came in January 2026, when a routine LiteLLM upgrade caused intermittent connection failures. Respan rolled back the update but never figured out the problem.

“We still haven’t figured out the cause,” the company said in its Hacker News launch post.

Since then, its engineers have stopped upgrading LiteLLM directly, instead pulling in upstream fixes one at a time.

To avoid another maintenance headache, the company borrowed an idea from Nginx, keeping the gateway’s core small and moving features like routing, model providers, and security into plugins. Developers can build their own using the same interfaces, without having to change the underlying code.

Developers can build their own using the same interfaces, without having to change the underlying code.

Some of those integrations are surprisingly small. Respan says its Mistral provider takes just 22 lines of code, while quirks like Groq reporting token usage in a different field can be handled separately. This makes it easier when updating one provider while avoiding the need to change how the rest of the gateway works.

When a request comes in over HTTP or WebSocket, the gateway runs it through the plugins in order. They can decide which model handles it, change the response, make additional calls, or stop the request before it reaches a provider.

Source link