Govern: Who Decides If Your AI Is Safe to Deploy?
Govern is the first function in NIST's AI RMF, and it's the one most companies skip.
Most companies can’t answer this question: if an AI system makes a bad decision tomorrow, who’s accountable for it? Not “who built it.” Not “which vendor sold it to us.” Who owns the risk? If the answer takes more than a sentence, you don’t have AI governance. You have AI usage with no one steering it. Govern is the first function in NIST’s AI RMF, and it’s the one most companies skip. They jump straight to “let’s test the model for bias” or “let’s document our data flows” without first establishing who has the authority to make those calls, who reviews them, and what happens when something goes wrong. That’s backwards. Testing and mapping without governance just produces findings nobody’s authorized to act on.
What Govern Actually Requires
Strip away the framework language and Govern comes down to four things:
Legal awareness. You know which laws and regulations apply to your AI use, and you’ve documented that understanding. Not a general sense of “AI regulation is coming.” Specific knowledge of what applies to your industry, your data, and your use cases right now.
Defined roles. Specific people or teams are accountable for specific decisions. Who approves an AI system for deployment? Who can pull it back if something goes wrong? Who reviews the risk before either happens?
Risk tolerance. You’ve decided, in writing, how much risk your organization will accept for different types of AI decisions. A chatbot answering FAQ questions carries different risk than an AI system recommending loan denials. Your policies should reflect that difference explicitly, not leave it to whoever’s in the room that day.
Oversight structure. Someone independent from the people building or deploying the AI system reviews it before and after launch. If the same team that built the model is also the only one checking its work, you don’t have oversight. You have a formality.
Why Most Governance Efforts Fail Before They Start
Companies that struggle with Govern usually make one of these mistakes:
They treat it as a document, not a structure. Writing an “AI governance policy” and putting it in a shared drive isn’t governance. Governance is the actual roles, actual review processes, and actual decision rights that operate day to day. NIST’s suggested actions repeatedly emphasize separating AI development from AI testing and oversight functions. If the same person builds it and blesses it, you have a conflict of interest baked into your process.
They borrow structure from traditional IT governance and stop there. Traditional model risk management, the kind banks have used for decades, is a reasonable starting point. But AI governance needs to also account for things traditional IT risk frameworks don’t: bias testing protocols, explainability requirements, and human-in-the-loop design for high-stakes decisions. If your AI governance policy is just your existing change management process with “AI” swapped in, it’s missing half the picture.
They forget third-party AI. This is the gap that catches almost everyone. Your organization's AI risk isn't limited to the models you built. It includes every vendor tool that uses AI to touch your data or your decisions: the CRM that scores your leads, the recruiting platform that ranks your candidates, the cloud provider that flags anomalies, and the shadow AI employees quietly use every day. NIST's Govern function specifically calls for policies covering third-party AI systems, including how you'll get transparency into training data and assumptions, how you'll test vendor systems before relying on them, and what happens if a vendor's AI fails or gets it wrong. Most companies have never inventoried this. If a regulator asked "what AI systems touch customer data across your full vendor stack," most companies couldn't answer today.
The Practical Starting Point
You don’t need a fifty page governance charter to start. You need three things, in this order:
First, name the accountable owner. For every AI system in production or under development, one person or role is accountable if it fails. Not a committee. A name. Committees diffuse accountability; a named owner concentrates it.
Second, separate build from review. Whoever builds or deploys the AI system should not be the only one evaluating whether it’s safe to launch or continue running. This can be a compliance function, a risk committee, or a designated reviewer outside the development team. The point is independence, not size.
Third, write down your risk tolerance before you need it. Decide now, while you’re calm and not under deadline pressure, what level of AI risk is acceptable for different categories of decisions. High stakes decisions like credit, employment, and healthcare recommendations should have a different bar than a low stakes internal tool. If you wait until a specific system is already deployed to have this conversation, you’ll make the decision under pressure instead of with judgment.
What This Looks Like in a Regulated Environment
If you’re in finance, healthcare, or another regulated industry, this isn’t optional groundwork. It’s what an auditor or regulator will ask for first. “Show me your governance structure” comes before “show me your bias testing results,” because without the first, the second doesn’t mean anything. A bias test that nobody with authority reviewed and nobody was accountable for acting on is just a document. Federal contractors face the same expectation from a different direction. NIST guidance is moving from suggested to expected in federal contracting relationships, which means the agencies you work with will increasingly want to see this structure in place, not promised.
Where This Leaves You
Govern isn’t the exciting part of AI risk management. It doesn’t involve testing models or catching bias. It’s org charts, decision rights, and documented policies. But every other function in the AI RMF depends on it. You can’t map risks you haven’t assigned anyone to look for. You can’t measure controls nobody’s accountable for maintaining. You can’t manage risk that has no owner.
Start here. The next article in this series covers Map: documenting the AI systems you know you have, and the ones hiding in your vendor stack that you don’t.