Production AI on Microsoft Foundry: a private reference setup for agents

Most AI pilots on Azure start with a key copied into an app and a model endpoint open to the internet. That works for a demo and fails the first security review. This is the setup I use when an agent has to go live on Microsoft Foundry: Entra ID instead of keys, everything behind private endpoints, guardrails on every step of the agent, evaluations before each release and traces in Application Insights. Every claim links to the official Microsoft pages, so you can check it.

Dominik Heinrich

November 7, 202611 min read

1. When Azure is the right home for your AI, and what changed in the names

Azure is the natural choice when your company already lives in Microsoft: identities in Entra ID, documents in SharePoint, people in Teams, workloads in Azure subscriptions with landing zones and policies. Then your AI platform inherits the governance you already have instead of building a second one. The model choice is broad: Microsoft describes access to more than 10,000 models from Microsoft, OpenAI, Anthropic, Meta and others in the Foundry Models catalog.

The naming has moved twice, which confuses a lot of search results. Microsoft's own overview page lists the history: Azure AI Studio, then Azure AI Foundry, and now Microsoft Foundry. The agent API changed too. The same page maps the old Assistants API (Agents v0.5/v1) to the Responses API (Agents v2), and the old terms threads, messages and runs to conversations, items and responses. If a tutorial still creates threads and runs, it describes the classic experience.

For the rest of this article I follow the current Microsoft Learn documentation for Foundry and the Azure Architecture Center's baseline Microsoft Foundry chat reference architecture, which is the closest official blueprint for an enterprise agent.

2. Resource, project and identity: no keys in your code

Foundry has two levels: a Foundry resource, which holds networking, policies and model deployments, and projects inside it, where teams build agents. Create the resource with a system-assigned managed identity and project management enabled. Microsoft's quickstart is explicit that project creation fails without the managed identity, and that the Azure CLI needs version 2.80.0 or later for the project commands.

From day one, let applications authenticate with Microsoft Entra ID: the web app uses its managed identity, people get roles through groups, and nobody copies an API key into a configuration file. The baseline architecture does exactly this: the web application calls the agent over a private endpoint and authenticates to Foundry with its managed identity. Keys are a fallback for tools that can't do Entra ID, and then they belong in Key Vault with rotation, not in code.

Keep at least two projects, or better two resources in separate subscriptions, for non-production and production. Model deployments, quotas and guardrails differ between them, and you want to test a change before your customers meet it.

bash

# Azure CLI 2.80.0+ (project commands). Names are examples.
az group create --name my-foundry-rg --location eastus

# Foundry resource with a managed identity and project management
az cognitiveservices account create \
  --name my-foundry-resource \
  --resource-group my-foundry-rg \
  --kind AIServices \
  --sku S0 \
  --location eastus \
  --custom-domain my-foundry-resource \
  --assign-identity \
  --allow-project-management true

# A project for the team that builds the agent
az cognitiveservices account project create \
  --name my-foundry-resource \
  --resource-group my-foundry-rg \
  --project-name my-foundry-project \
  --location eastus

Tip from practice

If project creation fails with a message that a managed identity must be enabled, the resource was created without --assign-identity. Fix it there instead of falling back to keys.

3. Private by default: the standard agent setup in your own network

Foundry Agent Service offers a standard setup with private networking. In this configuration you bring your own virtual network and your own Azure resources to store chat and agent state: Azure Storage for files, Azure Cosmos DB for conversations and Azure AI Search for vector data. All of them sit behind private endpoints, and public network access on the Foundry resource is disabled. The baseline reference architecture adds the edges around it: users come in through Azure Application Gateway with a web application firewall, the chat UI runs on App Service across three availability zones, and outbound traffic from the agents strictly routes through Azure Firewall, which enforces egress rules.

That last point matters more than it looks. An agent with a web search tool or an MCP connection is software that makes outbound calls on behalf of a prompt. With an explicit egress allowlist on the firewall, a prompt injection can't quietly send data to an arbitrary host.

Plan the subnets and private DNS zones before you create anything. The architecture uses separate subnets for App Service integration, private endpoints, the agent integration, Azure Bastion, build agents and the firewall. Retrofitting private networking onto a running agent environment is much harder than starting with it.

Tip from practice

Start from Microsoft's Foundry Agent Service reference implementation linked from the baseline architecture. It deploys the network, private endpoints and firewall with infrastructure as code, which saves days of DNS debugging.

Reference setup on Microsoft Foundry, based on the Azure Architecture Center's baseline Foundry chat architecture: everything private, egress through the firewall, observability in Application Insights.

4. Choose and deploy the model: standard, batch, priority or provisioned

Pick the model by the job, not by the headline: a small, fast model for routing and extraction, a stronger one for reasoning-heavy answers, and an embedding model for search. Then choose the deployment type. Microsoft compares four: standard deployments share inference capacity across customers, so throughput can vary with demand; batch suits large offline jobs; priority processing buys lower latency; and provisioned throughput holds a fixed amount of processing capacity exclusively for your deployment, whether or not requests are being made.

That last sentence is the cost trap and the benefit at the same time. Provisioned throughput gives predictable latency for a busy, steady workload, but you pay for it at 3 a.m. too. Start on a standard deployment, measure tokens per minute and latency for a few weeks, and only then decide whether the steady base load justifies provisioned capacity.

Before you deploy, list which model versions and SKUs your region actually offers. The quickstart shows a command for that, and a deployment fails with DeploymentModelNotSupported if the combination doesn't exist in your region.

bash

# Which versions and SKUs of a model exist in your region?
az cognitiveservices model list \
  --location eastus \
  --query "[?model.name=='gpt-5-mini'].{version:model.version,skus:join(',',model.skus[].name)}" \
  --output table

# Standard (shared capacity) deployment to start with
az cognitiveservices account deployment create \
  --name my-foundry-resource \
  --resource-group my-foundry-rg \
  --deployment-name gpt-5-mini \
  --model-name gpt-5-mini \
  --model-version "2025-08-07" \
  --model-format OpenAI \
  --sku-capacity 10 \
  --sku-name GlobalStandard

5. Build the agent: prompt agent, hosted agent and a curated toolbox

Foundry Agent Service meets you anywhere between declarative and full code. A prompt agent is defined by instructions, a model and tools, and Foundry runs it. A hosted agent packages your own code, for example a framework you already use, and Foundry hosts and scales it. Or you call the Responses API from an agent you run elsewhere. The agent runtime manages conversations, tool calls and the agent lifecycle.

Tools are where agents become useful and where they become risky. Foundry's toolboxes let you curate a set of tools once, such as web search, file search, code interpreter, MCP servers and custom functions, and share them across agents. Treat a toolbox like an API you publish: each tool has an owner, a clear description, least-privilege access and a test. For grounding on your own documents, the baseline architecture connects the agent to Azure AI Search over a private endpoint, so retrieval stays inside your network.

Start with one prompt agent and two or three tools. Most of the value in a first release comes from good instructions and reliable retrieval, not from a large tool list. Add tools only when an evaluation shows the agent needs them.

Foundry Agent Service + Microsoft Agent Framework Explained, Microsoft Mechanics (YouTube). Loads from YouTube only after you click.

6. Guardrails on every step, not just on the chat box

Classic content filtering looks at two places: what the user types and what the model answers. Agents have two more. Microsoft Foundry's guardrails are named collections of controls, and each control defines a risk, the intervention points to scan and the action to take. Four intervention points are supported: user input, tool call, tool response and output. Tool call and tool response are agent-only and in preview, as are agent guardrails in general.

The tool response point is the one most teams forget. Indirect prompt injection arrives through content the agent reads, such as a web page or a document, not through the user. Scanning tool responses is how you catch instructions hidden in that content before the model acts on them. The control list also includes protected material, personally identifiable information and task adherence in preview, so an agent that drifts away from its job can be flagged.

One date to note: Microsoft announces that groundedness detection (preview) in Azure OpenAI retires on April 30, 2027. If you rely on it, plan the replacement now and check your groundedness with evaluations as well.

Tip from practice

Keep a small file of attack prompts for your domain, including instructions hidden inside a document the agent retrieves, and run it against every guardrail change. A guardrail you never tested is a guess.

7. Evaluate before every release: quality, safety and agent behaviour

An agent changes behaviour every time you edit an instruction, swap a model version or add a tool. Without evaluations, you find out from customers. Foundry's evaluators cover general quality, retrieval-specific metrics such as groundedness and relevance, safety and security, and agent-specific metrics. The agent evaluators are the interesting ones: task adherence checks whether the agent's actions follow its assigned rules and policy constraints, intent resolution checks whether it understood what the user wanted, and tool call accuracy checks whether it made the right tool calls with the right parameters.

Build a test set of 50 to 200 real questions with expected outcomes, including the awkward ones from support tickets. Run it in your pipeline on every change to instructions, tools or model version, and block the release when a score drops below the last accepted run. You can add custom evaluators for domain rules, for example that an answer about a contract always cites the clause.

Microsoft also describes an agent optimizer (preview) that evaluates agent behaviour and suggests better instructions, tool descriptions and model choices. Use suggestions like that as a pull request that still has to pass your evaluation, never as an automatic change in production.

8. Observe and control cost in production, then compare the clouds

Foundry's tracing is built on OpenTelemetry and integrated with Azure Monitor Application Insights. A trace shows the execution flow of a request: model calls, tool invocations and agent decisions, which is exactly what you need when an agent gave a strange answer on Tuesday. On top, Microsoft describes production dashboards for token consumption, latency, error rates and quality scores, with alerts when outputs fail quality thresholds or produce harmful content.

Cost control is mostly discipline: a budget with forecast alerts on the subscription, token usage per agent and per project in the dashboards, the smallest model that passes your evaluation for each step, and a regular look at whether a steady load has grown large enough for provisioned throughput. Review the Azure Well-Architected guidance for AI workloads once before go-live and once a quarter.

Pitfalls I see most: keys in app settings, public network access left on after the pilot, no egress control for agents with tools, evaluations run once by hand instead of in the pipeline, and a model version upgraded silently. How the same setup looks on AWS is in /blog/aws-bedrock, and on Google Cloud in /blog/google-vertex-ai. The building blocks are similar; the names, the network model and the governance tools differ.

Tip from practice

Pin model versions in production and upgrade them like any other dependency: in non-production first, through the evaluation gate, with a rollback plan.

Takeaway
On Azure, a production agent is mostly the platform you already run: Entra ID, private endpoints, a firewall and Application Insights, with Foundry's guardrails and evaluations as the gates every change has to pass.

Real projects behind this

Try it yourself

Production readiness

Want this in your company? Let's talk mail@dominikheinrich.ch

Sources and official guides

Production AI on Microsoft Foundry: a private reference setup for agents

1. When Azure is the right home for your AI, and what changed in the names

2. Resource, project and identity: no keys in your code

3. Private by default: the standard agent setup in your own network

4. Choose and deploy the model: standard, batch, priority or provisioned

5. Build the agent: prompt agent, hosted agent and a curated toolbox

6. Guardrails on every step, not just on the chat box

7. Evaluate before every release: quality, safety and agent behaviour

8. Observe and control cost in production, then compare the clouds

Get in touch