The Federal Reserve’s SR 11-7 supervisory letter sets out three core principles for model risk management: governance, conceptual soundness, and ongoing monitoring. These remain foundational, but their application must be reinterpreted for agentic systems, which are dynamic, multi-step, and tool-enabled.

Model definition expands beyond the foundation model to include fine-tuning datasets, RAG corpora, prompt templates, tool integrations, and plugins.
Conceptual soundness requires anticipating variability and testing systems under ambiguous or adversarial conditions.

Validation must go beyond accuracy to cover behavioral, security, and resilience testing

The manual provides structured frameworks to create an SR 11-7 Model Validation document for an AI agent that can survive examination by a federal financial regulator:

Key MRM Questions that risk teams should ask (e.g., about governance, inventory completeness, fairness).

Sample Tests that specify what to test, how to test it, and what evidence to collect.
Testing approaches that institutions can adapt to their own systems.

This ensures institutions produce documentation and processes aligned with their own risk profiles, rather than relying on generic templates.

The manual specifies evidence requirements for each control area, including:

Governance drills: escalation logs, decision rationale, post-mortems.

Inventory checks: annotated comparison reports, dependency diagrams, change tickets.

• Validation: hallucination rate metrics, injection prompt transcripts, incident reports.
  Monitoring: assertion lists, compliance scan results, accuracy trend graphs.
• Fairness tests: demographic parity analysis, consistency results, groundedness scores.
• Audit trail: version control logs, decision path reconstructions.
• Vendor oversight: SLA compliance records, penetration test results, exit plans.

The manual stresses ongoing monitoring that combines human oversight with automated checks to counteract vigilance fatigue and automation bias:

• Assertion-based compliance scanning against predefined rules.
• Factual accuracy checks comparing outputs to source-of-truth databases.
• Latency and load monitoring to maintain reliability.
• Update regression testing before and after vendor model updates.

Monitoring results should feed into dashboards with escalation triggers, and monitoring rules should evolve alongside regulatory and product changes.

The manual explains several options for ongoing fairness evaluations of AI Agents including:

• Demographic Parity Simulation: Comparing outputs across simulated profiles differing only by protected attributes.
• Consistency Testing Across Variants: Checking if re-phrased queries or demographic changes yield consistent results.
• RAG Groundedness Tests: Ensuring answers are tied to verifiable retrieved sources.

Evidence includes test cases, statistical analysis (e.g., adverse impact ratios), and remediation documentation.

A champion–challenger setup means running a parallel model (“challenger”) alongside the production model (“champion”). The challenger is actively monitored and can replace the champion immediately if performance falls below defined thresholds.

This approach ensures continuity and mitigates the temptation to leave a failing system online. The manual encourages fallback systems to be tested in advance and tied to objective disablement triggers (e.g., hallucination rate thresholds).

The manual emphasizes that institutions remain responsible for vendor actions:

• Vendor assessment against SR 11-7 principles.
• SLAs for notice of updates, disclosure of documentation, and timely incident reporting.
• Independent guardrail testing, including penetration tests to probe bypass risks.
• Exit strategies to transition away from vendors that no longer meet requirements.

Vendors can use the manual to anticipate and satisfy bank reviewers’ expectations:

• Evidence preparation: inventories, validation reports, groundedness checks.
• Conceptual soundness: documenting architecture decisions and risk trade-offs.
• Auditability: maintaining version control and decision path reconstruction.
• Regulatory readiness: demonstrating governance and monitoring aligned with SR 11-7 and examiner expectations.