Thanks to Amit Gupta, Senior Director- AI/ML and Mark Wang, Software Engineering PMTS from Salesforce for your contributions in the vision and partnership of cross-Region inference routing on SageMaker AI.
Salesforce’s Agentforce Foundation team serves a wide variety of models for products across Salesforce, including Agentforce and the Salesforce Trust Layer, in 30+ global Regions. These include both models trained in house and open source models. Hosting models separately in each Region splits capacity and makes traffic spikes harder to absorb, which turns a capacity shortfall into an availability problem.
This post shows how Salesforce set up the cross-Region inference routing in SageMaker AI and how it lets the team add capacity securely while keeping its data residency commitments. It also walks through how Salesforce sees where each request was served and why. The cross-Region inference routing gives an application one invocation path across SageMaker AI endpoints in several Regions and accounts, and sends each request to an endpoint that can serve it. Those endpoints are Salesforce’s own, already deployed. The post starts with the capacity problem Salesforce faced, explains how the router works and how Salesforce integrated it into its Internal Gateway, walks through an example setup, and then describes what the team observed.
Salesforce’s Agentforce Foundation team serves customers from the Region their requests come from, across 30+ global Regions and hundreds of AWS accounts, and they provision capacity based on each customer’s traffic. Every customer has its own peak window, and those windows often overlap. Provisioning for the overlapping peak would be the safe choice, but it costs too much.
Source link







