Building Reliable AI Agents - Implementing Advanced Evaluation with Azure AI SDK and Custom APIM Integration

The evaluate() function returned 401 Unauthorized for the fourth time that afternoon, and I was starting to take it personally.

I had the right endpoint. The right API version. The environment variables were set. Every smoke test I ran against the Azure API Management gateway directly came back fine. But the moment I handed that same gateway URL to the Azure AI Evaluation SDK, it acted like I’d never authenticated at all.

I wanted automated checks on our research agent: groundedness, relevance, faithfulness, fluency, and the other metrics we needed — the kind of eval harness any serious AI deployment needs. Instead I spent three hours staring at HTTP 401s while the SDK stripped the headers APIM required.

Here’s what was actually happening, and how I eventually got it working.

Where the request went wrong

The Azure AI Evaluation SDK’s built-in evaluators work when the model sits behind a plain Azure OpenAI endpoint. But most enterprise setups do not look like that. They route through Azure API Management instead. APIM is the gateway in front of the AI service; it handles authentication, rate limiting (capping how many requests a caller can make in a window), IP filtering, and request transformation before the model sees a request.

The SDK’s evaluators do not know about that gateway. They construct their own openai.AzureOpenAI client, read AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY from the environment, and offer no hook for extra headers or a bearer token — a short-lived credential that proves your identity to Azure. APIM required custom headers such as a username claim, a subscription key, and a tenant-routing field. Those headers never arrived, so APIM rejected the request before the model saw it.

Our research agent needed five dimensions: groundedness (are the claims supported by sources?), answer relevancy, contextual precision and recall, faithfulness to provided context, and fluency/coherence. None of that helped until the evaluators could reach the model.

graph TD
    Client[Evaluation Client] -->|Standard Path| Direct[Direct Azure OpenAI]
    Client -->|Enterprise Path| APIM[Azure API Management]
    APIM -->|Custom Auth Required| Direct
    
    style Enterprise Path stroke:#f66,stroke-width:2px
    style APIM fill:#f66,stroke:#333,stroke-width:2px

This standard path never worked for us:

# Standard approach - fails with APIM
evaluators = {
    "groundedness": GroundednessEvaluator(),
    "relevance": RelevanceEvaluator()
}

results = evaluate(
    evaluators=evaluators,
    target_function=run_research_agent,
    test_cases=test_scenarios,
    # Even when providing correct APIM URL, it fails with auth errors
    azure_endpoint=os.getenv("AZURE_APIM_URI")
)

401 Unauthorized. 403 Forbidden. Every time.

Give the evaluators the right client

I had to own the HTTP client instead of letting the SDK create it under the hood.

I wrote a base class for the evaluators. It gets an Azure AD token through Azure’s identity chain (environment variables → managed identity → CLI credentials, tried in order), passes that token as the api_key to AzureOpenAI, and adds the required APIM headers to every request with with_additional_headers(), a method on the OpenAI Python SDK’s transport layer.

class APIMEvaluatorBase:
    """Base class for custom APIM-compatible evaluators."""
    
    def __init__(self, metric_name, apim_config):
        self._metric_name = metric_name
        self.config = apim_config
        
        # Get OpenAI client with proper auth for APIM
        self.client = self._create_apim_client()
    
    def _create_apim_client(self):
        # Get Azure AD token
        credential = DefaultAzureCredential()
        token = credential.get_token("https://cognitiveservices.azure.com/.default")
        
        # Create client with custom headers for APIM
        client = AzureOpenAI(
            azure_endpoint=self.config.endpoint,
            api_version=self.config.api_version,
            api_key=token.token,  # Using token as API key
        )
        
        # Add required APIM headers
        headers = {
            "mkl-User-name": self.config.username,
            "username": self.config.username
        }
        
        # Apply headers to all requests
        client = client.with_additional_headers(headers)
        return client
        
    def __call__(self, response, context=None, query=None):
        """Evaluate with our APIM-aware client."""
        raise NotImplementedError()

Each metric got a subclass — APIMGroundednessEvaluator, APIMRelevanceEvaluator, APIMContextualPrecisionEvaluator, and so on — using the same client factory.

from azure.identity import DefaultAzureCredential
from openai import AzureOpenAI

credential = DefaultAzureCredential()
token = credential.get_token("https://cognitiveservices.azure.com/.default")

client = AzureOpenAI(
    azure_endpoint="https://<your-apim-gateway>.azure-api.net/openai",
    api_version="2024-02-01",
    api_key=token.token,
).with_additional_headers({
    "Ocp-Apim-Subscription-Key": "<your-subscription-key>",  # if required
    "x-custom-username": "<service-account-name>",
})

# Smoke-test: if this returns a completion, your auth+headers are correct.
resp = client.chat.completions.create(
    model="<your-deployment-name>",
    messages=[{"role": "user", "content": "ping"}],
    temperature=0.0,
)
print(resp.choices[0].message.content)

The SDK still stopped short

The custom evaluators worked one at a time. Each returned a score and reasoning. Passing them into evaluate() — the SDK function that aggregates all metrics and test cases — was different.

The framework’s aggregation logic assumes evaluator behaviors our APIM-aware classes could not satisfy. I spent a full morning in stack traces before accepting that evaluate() was a dead end for this setup.

I wrote the aggregation loop myself.

def run_manual_evaluation(test_scenarios, apim_config):
    """Run evaluation manually with custom APIM-aware evaluators."""
    
    evaluators = {
        "groundedness": APIMGroundednessEvaluator(apim_config),
        "relevance": APIMRelevanceEvaluator(apim_config),
        "contextual_precision": APIMContextualPrecisionEvaluator(apim_config),
        "faithfulness": APIMFaithfulnessEvaluator(apim_config),
        "fluency": APIMFluencyEvaluator(apim_config)
    }
    
    thresholds = {
        "groundedness": 3.5,
        "relevance": 3.5,
        "contextual_precision": 3.0,
        "faithfulness": 3.5,
        "fluency": 3.0
    }
    
    all_results = []
    
    for scenario in test_scenarios:
        agent_response = run_research_agent(
            query=scenario["query"],
            additional_context=scenario.get("additional_context", "")
        )
        
        scenario_results = {
            "query": scenario["query"],
            "response": agent_response,
            "metrics": {}
        }
        
        for metric_name, evaluator in evaluators.items():
            if metric_name not in scenario.get("evaluation_metrics", list(evaluators.keys())):
                continue
                
            result = evaluator(
                response=agent_response,
                context=scenario.get("context", ""),
                query=scenario["query"]
            )
            
            scenario_results["metrics"][metric_name] = {
                "score": result["score"],
                "reasoning": result["reasoning"],
                "threshold": thresholds[metric_name],
                "pass": result["score"] >= thresholds[metric_name]
            }
        
        all_results.append(scenario_results)
    
    summary = calculate_evaluation_summary(all_results, thresholds)
    save_evaluation_results(all_results, summary)
    
    return all_results, summary

Leaving the SDK’s orchestration layer gave me visibility into every evaluation call and made debugging authentication failures trivial. When auth failed, I could print() one evaluator’s response instead of having the framework swallow it.

What the metrics actually looked like

Here’s a groundedness evaluation in practice. The agent’s response scores a 5.0 — every claim maps back to the provided context:

scenario = {
    "query": "What are the main environmental risks for offshore drilling?",
    "context": "Offshore drilling poses several environmental risks including: 
               oil spills that can damage marine ecosystems, 
               disruption of marine habitats during drilling, 
               underwater noise pollution affecting marine mammals, 
               and greenhouse gas emissions from flaring operations.",
    "evaluation_metrics": ["groundedness", "relevance"]
}

response = "Offshore drilling presents significant environmental concerns. 
            The primary risks include catastrophic oil spills that damage marine ecosystems,
            physical disruption of seafloor habitats during drilling operations,
            noise pollution that can disorient and harm marine mammals like whales and dolphins,
            and substantial greenhouse gas emissions from gas flaring."

result = {
    "score": 5.0,
    "reasoning": "The response accurately reflects all environmental risks mentioned 
                 in the context without adding unsupported claims."
}

And here’s one that scored a 2.0 on relevance — the model answered about environmental permits when the question was about safety regulations:

scenario = {
    "query": "What regulations govern offshore drilling safety?",
    "context": "Offshore drilling is regulated by several frameworks including 
               the BSEE regulations in the US, which mandate safety equipment like blowout preventers. 
               The SEMS rule requires safety management systems. 
               International operations often follow IMO guidelines and regional regulations."
}

response = "Offshore drilling has strict regulations. Companies must follow 
            environmental protection standards and obtain permits before drilling. 
            Environmental impact assessments are required in many jurisdictions."

result = {
    "score": 2.0,
    "reasoning": "The response discusses regulations but focuses on environmental permits
                 rather than the safety regulations mentioned in the context. 
                 It doesn't address BSEE, SEMS, or IMO guidelines specifically asked about in the query."
}

Three small things that mattered

TypedDict for return types. The SDK expects specific shapes from evaluator __call__ methods. Python’s TypedDict kept those signatures compatible and caught mismatches before runtime:

from typing import TypedDict, List, Dict

class GroundednessResult(TypedDict):
    score: float
    reasoning: str

class APIMGroundednessEvaluator(APIMEvaluatorBase):
    def __call__(self, response, context=None, query=None) -> GroundednessResult:
        # Implementation

Retry logic. APIM calls timed out and hit transient errors — not often, but often enough to break a long evaluation. An exponential backoff wrapper — wait 2^n seconds between retries, up to 3 attempts — turned those intermittent failures into passing runs:

def safe_api_call(client, *args, max_retries=3, **kwargs):
    """Make API call with retry logic."""
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(*args, **kwargs)
        except (APITimeoutError, ServiceUnavailableError) as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)  # Exponential backoff

Result caching. Re-running the same prompts against the same responses only burns tokens. An MD5-keyed cache, hashing query + context + response, saved cost and latency on re-runs:

class CachingEvaluator:
    """Wrapper for evaluators that caches results."""
    
    def __init__(self, evaluator, cache_file=None):
        self.evaluator = evaluator
        self.cache_file = cache_file or f"{type(evaluator).__name__}_cache.json"
        self.cache = self._load_cache()
    
    def _load_cache(self):
        if os.path.exists(self.cache_file):
            with open(self.cache_file, 'r') as f:
                return json.load(f)
        return {}
    
    def _save_cache(self):
        with open(self.cache_file, 'w') as f:
            json.dump(self.cache, f)
    
    def __call__(self, response, context=None, query=None):
        key = hashlib.md5(f"{query}|{context}|{response}".encode()).hexdigest()
        
        if key in self.cache:
            return self.cache[key]
        
        result = self.evaluator(response, context, query)
        self.cache[key] = result
        self._save_cache()
        
        return result

What we got

After running the full eval battery across our research agent’s test scenarios, the numbers looked like this:

{
  "summary": {
    "groundedness": {
      "average_score": 4.7,
      "pass_rate": 0.95,
      "threshold": 3.5
    },
    "relevance": {
      "average_score": 4.5,
      "pass_rate": 0.92,
      "threshold": 3.5
    },
    "contextual_precision": {
      "average_score": 4.2,
      "pass_rate": 0.89,
      "threshold": 3.0
    },
    "faithfulness": {
      "average_score": 4.6,
      "pass_rate": 0.94,
      "threshold": 3.5
    },
    "fluency": {
      "average_score": 4.8,
      "pass_rate": 0.98,
      "threshold": 3.0
    },
    "overall_pass_rate": 0.91
  }
}

91% overall pass rate. Contextual precision was the weakest link — the agent sometimes pulled in tangentially relevant information when the question demanded laser focus. That became the next thing to fix.

If I started again

Test one evaluator against the gateway before orchestrating five. One 401 is a five-minute fix. Five evaluators throwing 401 inside a framework you do not control is an afternoon.

Log authentication failures verbosely. The SDK’s default error messages won’t tell you which header is missing — you’ll need to inspect the raw HTTP response to see what APIM is actually rejecting.

If the SDK’s orchestration does not fit, walk away from it. The manual loop here is under a hundred lines, easy to debug, and does not fight you over headers.

For the Azure AI Evaluation team, three changes would help: documentation for APIM-protected endpoints, SDK support for Azure AD token auth with custom headers, and extension points for custom evaluators so developers do not have to bypass orchestration.

The quality of an AI system is not just about what the model can do. It is about whether you can reliably measure what it actually does. That measurement has to survive the architecture the service actually runs behind.