Skip to content

AI governance: frameworks, controls and monitoring

  • What the main governance frameworks require, with their current versions and dates as of November 2026.
  • How to turn them into an operating model: an AI inventory, risk tiers, required controls per tier, and owners.
  • How to make agent behavior observable with OpenTelemetry’s GenAI conventions, with a tested example.
  • Where evaluation gates, data rules and incident response fit.
  • How governance fails in practice, and how to keep it proportionate.

This page is about the organization and operations side. The technical risks of agents and the controls inside the application are on Risks of agentic AI.

AI governance is the set of decisions, owners and controls that keep AI systems lawful, safe and useful after launch. Four references cover most of what an engineering organization needs: the EU AI Act (law, for systems used in the EU), the NIST AI Risk Management Framework (a voluntary method for managing AI risk), ISO/IEC 42001 (a certifiable management system standard) and the OWASP Top 10 lists (security risks for LLM applications and agents).

In practice, governance comes down to five habits: know which AI systems you run (an inventory), classify each by risk, require controls in proportion to that risk, observe what the systems actually do, and have a named owner who can change or stop each one.

LLM applications differ from ordinary software in ways that break existing controls:

  • The same input can produce different outputs, so a passing test is evidence, not proof.
  • Behavior changes without a code change. A provider updates or retires a model, a prompt is edited, a document is added to a retrieval index.
  • Agents act. A tool-using agent can send messages, change records and spend money, so a quality problem becomes an operational incident.
  • Regulation now applies. The EU AI Act has binding obligations in force, with more arriving in 2026 to 2028, and customers increasingly ask for ISO/IEC 42001 certification or NIST-aligned risk assessments in procurement.

Governance that only reviews a system before launch misses most of this. It needs to continue in production.

Framework Type Current version and status Where it binds
EU AI Act Law: Regulation (EU) 2024/1689, amended by the Digital Omnibus on AI, Regulation (EU) 2026/1744 In force since August 1, 2024; amendments in force since July 27, 2026; obligations phased in to 2028 AI systems placed on the market or used in the EU, including by non-EU providers
NIST AI RMF Voluntary framework AI RMF 1.0 (NIST AI 100-1, January 2023), plus the Generative AI Profile (NIST AI 600-1, July 2024); NIST states that a revision of 1.0 is in progress Voluntary; widely used in US procurement and as a common vocabulary
ISO/IEC 42001 Certifiable management system standard ISO/IEC 42001:2023, first edition; adopted unchanged in Europe as EN ISO/IEC 42001:2026 Voluntary; certification by accredited bodies
OWASP Top 10 for LLM Applications Security awareness list 2026 edition, published August 4, 2026 Voluntary; a common basis for threat models and tests
OWASP Top 10 for Agentic Applications Security awareness list First edition, December 2025 Voluntary

The Act classifies AI systems by risk and attaches obligations to providers (who build or place a system on the market) and deployers (who use it professionally).

  • Prohibited practices (for example social scoring and certain manipulative or biometric uses) and the AI literacy duty have applied since February 2, 2025. The 2026 amendments softened the literacy duty from “ensure” a sufficient level of literacy to taking measures to promote it.
  • General-purpose AI model obligations (technical documentation, copyright policy, training-data summary, and extra duties for models with systemic risk) have applied since August 2, 2025. These fall mainly on model providers such as the labs, not on teams that call their APIs.
  • Transparency obligations in Article 50 apply from August 2, 2026: people must be told when they interact with an AI system, and AI-generated content must be marked. Published analyses of Regulation 2026/1744 report a grace period until December 2, 2026 for machine-readable marking by generative systems already on the market before August 2, 2026; check the consolidated text for your case.
  • High-risk systems (Annex III uses such as employment, credit scoring, education and essential services) were due on August 2, 2026; the Digital Omnibus moved them to December 2, 2027, and high-risk AI embedded in products covered by Annex I legislation to August 2, 2028. High-risk obligations include a risk management system, data governance, technical documentation, logging, human oversight, accuracy and robustness, and registration.

For most teams building on hosted models, the practical questions are: does any use case fall into Annex III, do users know they are talking to an AI system, and is generated content labeled? This page is not legal advice; involve counsel for classification decisions.

The AI RMF organizes risk management into four functions: Govern (policies, roles, accountability, culture), Map (context, intended use, affected people, risks), Measure (test and track risks with metrics) and Manage (prioritize, treat, monitor and respond). The Generative AI Profile, NIST AI 600-1, lists twelve risks specific or amplified by generative AI, among them confabulation, information security, data privacy, harmful bias and value chain integration, with more than 200 suggested actions mapped to the four functions. Its value for engineers is a shared checklist that security, legal and product teams already recognize.

ISO/IEC 42001:2023 specifies an AI management system, in the same family and structure as ISO/IEC 27001 for information security: policy, roles, risk assessment, impact assessment, controls (listed in its Annex A), internal audit, management review and continual improvement. An organization that already runs ISO 27001 can extend its existing processes. Certification shows customers that the process exists; it does not certify that any individual model or system is safe.

The OWASP lists are the most directly usable by engineers: each entry describes a risk, examples, mitigations and attack scenarios. Use the 2026 LLM list and the agentic list as the starting point for threat models and red-team scenarios; Risks of agentic AI maps them to concrete controls.

Keep one register of AI systems. For each, record: owner, purpose, users and affected people, models and providers (with model IDs), data categories sent to the model, tools and permissions, autonomy level, EU AI Act classification, and the date of the last review. Without the inventory, every other control is guesswork. Include AI features inside bought software, not only what you build.

Tier Typical OrdersPortal example Required before launch Required in production
Low Internal summary of support tickets Owner, inventory entry, data check Usage and cost monitoring
Medium Customer-facing reply drafts sent after staff review Evaluation set and threshold, AI disclosure to users, prompt-injection tests Tracing, quality sampling, incident route
High Agent that issues refunds without review All of the above, plus threat model, approval gates in code, red-team, sign-off by risk owner Per-run traces with audit retention, alerts on actions, kill switch, quarterly review
Restricted Screening job applicants or setting credit limits Legal classification (likely EU AI Act high-risk), impact assessment, human decision-maker Bias monitoring across groups, logging per the Act, registration where required

Tiers keep governance proportionate: low-risk tools get a light process, and effort goes where harm is possible.

Decide per data category what may be sent to which provider and under what terms: training on your data, retention period, region of processing, and zero-data-retention options. Some providers’ newest models require minimum retention or exclude certain options (see Anthropic Claude models for an example), so check data terms per model, not only per provider. Where data may not leave your environment, run open-weight models yourself; see Local LLMs and AI on Docker and Kubernetes.

Every change to a prompt, model, tool or retrieval index runs an evaluation set before release, with thresholds agreed by the owner. A model upgrade is a change like any other. The harness engineering section covers automated evaluation and regression testing and red teaming.

You cannot govern what you cannot see. For each run record: the model and version, prompt version, inputs and outputs (subject to data rules), every tool call with arguments and results, policy decisions (allowed, approved, denied), token usage and cost, latency, and the business outcome. Link each run to the business record (the ticket or order) so a complaint can be traced back to the exact run.

Worked example: tracing an agent run with OpenTelemetry

Section titled “Worked example: tracing an agent run with OpenTelemetry”

OpenTelemetry defines GenAI semantic conventions: standard span names and attributes for model calls, tool executions and agent invocations, such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.tool.name. Using them means any OpenTelemetry-compatible backend can show token use and tool calls without custom mapping. The conventions now live in their own repository and are marked Development status, so attribute names can still change; pin the version you implement.

This sample records one OrdersPortal returns-agent run as nested spans. Token counts are fixed numbers standing in for the usage values a real API response returns, and a small exporter prints spans instead of sending them over OTLP. Run with Python 3.10 or later (tested with 3.11.18) after pip install opentelemetry-sdk (1.45.2 used here); save as trace_run.py and run python trace_run.py.

# trace_run.py - record one agent run as OpenTelemetry spans using GenAI attribute names
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, SpanExporter, SpanExportResult
class PrintExporter(SpanExporter):
"""Prints one line per finished span; production code exports via OTLP instead."""
def export(self, spans):
for s in spans:
a = dict(s.attributes)
extra = {k: v for k, v in a.items() if k.startswith(("gen_ai.usage", "ordersportal"))}
print(f"{s.name:<32} parent={'yes' if s.parent else 'no ':<3} {extra}")
return SpanExportResult.SUCCESS
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(PrintExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("ordersportal.support-agent", "1.4.0")
def model_call(step: int, input_tokens: int, output_tokens: int) -> None:
with tracer.start_as_current_span("chat claude-opus-5-5") as span:
span.set_attribute("gen_ai.operation.name", "chat")
span.set_attribute("gen_ai.provider.name", "anthropic")
span.set_attribute("gen_ai.request.model", "claude-opus-5-5")
span.set_attribute("gen_ai.usage.input_tokens", input_tokens) # from the API response
span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
span.set_attribute("ordersportal.step", step)
def tool_call(name: str, decision: str) -> None:
with tracer.start_as_current_span(f"execute_tool {name}") as span:
span.set_attribute("gen_ai.operation.name", "execute_tool")
span.set_attribute("gen_ai.tool.name", name)
span.set_attribute("ordersportal.policy_decision", decision) # allow / approve / deny
with tracer.start_as_current_span("invoke_agent returns-agent") as run:
run.set_attribute("gen_ai.operation.name", "invoke_agent")
run.set_attribute("gen_ai.agent.name", "returns-agent")
run.set_attribute("ordersportal.ticket_id", "T-48213")
model_call(1, 2100, 180)
tool_call("get_order", "allow")
model_call(2, 2450, 210)
tool_call("issue_refund", "approve")
model_call(3, 2700, 95)
run.set_attribute("ordersportal.outcome", "refund_pending_approval")

Output (child spans finish before the parent, so the agent span prints last):

chat claude-opus-5-5 parent=yes {'gen_ai.usage.input_tokens': 2100, 'gen_ai.usage.output_tokens': 180, 'ordersportal.step': 1}
execute_tool get_order parent=yes {'ordersportal.policy_decision': 'allow'}
chat claude-opus-5-5 parent=yes {'gen_ai.usage.input_tokens': 2450, 'gen_ai.usage.output_tokens': 210, 'ordersportal.step': 2}
execute_tool issue_refund parent=yes {'ordersportal.policy_decision': 'approve'}
chat claude-opus-5-5 parent=yes {'gen_ai.usage.input_tokens': 2700, 'gen_ai.usage.output_tokens': 95, 'ordersportal.step': 3}
invoke_agent returns-agent parent=no {'ordersportal.ticket_id': 'T-48213', 'ordersportal.outcome': 'refund_pending_approval'}

What makes this useful for governance:

  • Span names follow the conventions (chat {model}, execute_tool {tool}, invoke_agent {agent}), so dashboards and cost reports work across providers.
  • Business attributes sit beside standard ones. The ticket ID and the policy decision let an auditor answer “who approved this refund, and what did the agent see?”
  • Message content is deliberately absent. The conventions treat prompt and completion content as opt-in because it can contain personal data; capture it only where your data rules allow, with matching retention.

Many SDKs and observability products (provider SDKs, LangSmith, Arize Phoenix, Langfuse and others) emit or accept these spans; check each product’s documentation for the version of the conventions it supports.

Approach Strength Weakness Fits
Lightweight policy plus inventory Fast, low overhead Weak evidence for regulators and customers Small teams, low-risk internal uses
NIST AI RMF alignment Flexible, widely understood vocabulary Not certifiable; easy to adopt in name only US-facing organizations, procurement questionnaires
ISO/IEC 42001 certification Third-party evidence; fits existing ISO 27001 programs Audit cost; certifies process, not outcomes B2B vendors whose customers ask for it
EU AI Act compliance program Required where it applies Legal interpretation and documentation effort Any system in scope, especially high-risk uses

These are layers, not alternatives: an inventory and risk tiers underpin all of them, and one set of controls can provide evidence for several frameworks at once.

  • Governance theater. Policies exist but are not wired into delivery: no evaluation gate in CI, no approval in code, nobody reads the dashboards.
  • Shadow AI. Teams adopt AI features in SaaS tools or call model APIs with personal keys, outside the inventory. Make the approved path the easiest one.
  • Review once, drift forever. A provider retires a model, a prompt is edited, and the system that was reviewed no longer exists. Re-review on model, prompt, tool or data changes.
  • Logging everything. Full prompt and output logs create a new store of personal data. Log what you need, mask what you can, and apply retention.
  • No kill switch. Nobody knows how to disable an agent at 2 a.m. Name the owner, document the switch and rehearse it.
  • Uniform heavy process. Requiring a full impact assessment for every internal summarizer teaches people to route around governance.
  • For low-risk internal tools, an inventory entry, a named owner and cost monitoring are usually enough.
  • When a control produces no decision, such as a review that never blocks anything, remove or redesign it.
  • When existing processes already cover the risk (security review, privacy review, change management), extend them with AI-specific questions rather than creating a parallel track.

1. What is the difference between the NIST AI RMF and ISO/IEC 42001? The AI RMF is a voluntary framework that describes outcomes and suggested actions across four functions (Govern, Map, Measure, Manage); there is no certification. ISO/IEC 42001 is a management system standard with requirements that an accredited body can audit and certify, structured like ISO 27001. Many organizations use the RMF’s vocabulary for risk work and 42001 for the auditable management system.

2. A team builds a support chatbot on a hosted model for EU customers. Which EU AI Act obligations are most likely to apply? Probably not the high-risk obligations, unless the use falls into an Annex III category. The Article 50 transparency duties apply from August 2, 2026: users must be told they are interacting with an AI system, and generated content must be marked where the Act requires it. The general-purpose AI obligations fall mainly on the model provider. The classification should be documented and checked with counsel.

3. What should a trace of an agent run contain to support an incident investigation? The model and version, prompt version, the sequence of model calls with token usage, every tool call with arguments, results and policy decisions, approvals with the approver, the final outcome, and a link to the business record. Content capture is governed by data rules, so traces should identify what was sent even when the content itself is masked.

4. Senior scenario: your organization has 40 AI features across teams, no inventory and an EU customer base. You have one quarter. What do you do? Week one: build the inventory from API keys, invoices, procurement records and a short survey, with an owner per system. Then classify each against a simple tier model and the EU AI Act categories, and get legal review for anything near Annex III. Fix the highest tiers first: approvals in code for agents that act, tracing, evaluation gates, a kill switch, and AI disclosure for customer-facing features. Publish one approved path (approved providers, data rules, a shared tracing library) so new work arrives governed by default. Report progress per tier, not per policy written.

5. Senior scenario: a provider announces the retirement of the model behind your highest-risk agent in 60 days. How does governance handle it? The inventory shows every system using that model. The owner schedules a migration as a normal change: run the evaluation set and red-team scenarios against candidate models, compare quality, cost and behavior, review data terms for the new model, and get risk-owner sign-off for the high-risk agent. Roll out gradually with tracing to compare runs, and keep the old model available as a rollback until the retirement date.

  • Governance is continuous: inventory, risk tiers, proportionate controls, observation and named owners.
  • The EU AI Act binds in phases; after Regulation 2026/1744, high-risk obligations start December 2, 2027 (Annex III) and August 2, 2028 (Annex I).
  • NIST AI RMF gives the vocabulary, ISO/IEC 42001 the auditable system, OWASP the concrete security risks.
  • Trace every agent run with standard GenAI attributes and link it to the business record.
  • Re-review on every model, prompt, tool or data change, and rehearse the kill switch.