Serving a model from more than one AWS Region is a well-established pattern for highly available inference: when one Region runs short of GPU capacity, requests move to another. The pattern is simple to describe and hard to operate. Until now, if you served models on Amazon SageMaker AI managed endpoints, cross-Region routing was a piece you still had to build yourself.
Today, we’re introducing cross-Region inference routing in Amazon SageMaker AI. It uses a new resource, the inference router, that gives your application one invocation path across the endpoints and inference components you already run, in one or more AWS Regions and AWS accounts. You register the destinations that serve a model, and your application invokes the router instead of a specific endpoint. The router keeps requests in its own Region while the local destination is healthy, shifts new requests to another registered destination when it isn’t, and returns traffic gradually after recovery.
In this post, we explain how cross-Region inference routing decides where each request runs, walk through an example with one model in two Regions, and cover monitoring, limits, and what you pay. For a cross-account deployment that serves several models, see How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI.
A SageMaker AI endpoint runs in one AWS Region and one account. When autoscaling can’t add instances of the type your endpoint needs, requests throttle and time out, even when the same model has spare capacity in another Region.
Source link







