Imagine an AI system monitoring a power facility. It spots an unusual pattern and recommends an operational change. The recommendation sounds plausible, but the operator cannot tell which data drove it, whether conditions have shifted beyond those used in testing, or what happens if the system is wrong.
In a consumer app, that uncertainty may be frustrating. In critical infrastructure, it can affect physical equipment, essential services, and public safety.
On April 7, 2026, the National Institute of Standards and Technology released a [concept note for an AI Risk Management Framework profile on trustworthy AI in critical infrastructure](https://www.nist.gov/document/concept-note-artificial-intelligence-risk-management-framework-trustworthy-ai-critical). NIST is developing the profile to help operators identify risk management practices for AI and communicate their expectations to developers, vendors, and supply chain partners.
The distinction matters: this is a proposed profile, not a new regulation or a finalized checklist. The broader [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) is voluntary. Still, the concept note offers a useful view of the questions organizations will need to answer before placing more AI into high consequence environments.
Why this matters beyond the power grid
Critical infrastructure includes systems that people depend on every day. NIST’s concept note focuses on AI used across information technology, operational technology, and industrial control systems. Operational technology controls or monitors physical processes, while industrial control systems are a subset commonly used in facilities and plants. An AI mistake in these settings can have consequences far beyond an inaccurate screen display.
For CISOs, the issue is no longer limited to whether an AI service can protect data. They must also ask what the service can influence. Can it recommend a change to a physical process? Can an agent take action during an incident? Will a human see the recommendation before it affects operations?
For AI entrepreneurs, the concept note points toward a product challenge: a capable model is only one part of a trustworthy system. Buyers may also need evidence about operating limits, testing, failure behavior, and who remains in control.
For everyone else, this is a reminder that “AI in infrastructure” does not mean a chatbot running a power station. It can mean software assisting with diagnostics, monitoring a plant, helping respond to cyber incidents, or optimizing resources. The risk depends on what the system observes, recommends, and is permitted to do.
What NIST is signaling
The concept note gives examples of AI systems and the safeguards that could make them more trustworthy. It mentions diagnostic assistants with traceable, auditable rationales; monitoring systems that detect when conditions move outside verified limits; autonomous systems with redundant safety measures; and compliance tools that retain human oversight. It also points to rigorous testing, evaluation, validation, and verification throughout the AI lifecycle. These are examples and areas for development, **not controls that NIST has already mandated**.
Four themes deserve attention now.
Traceable decisions. An operator should be able to examine what supported an AI recommendation. That might include the input data, model and software versions, relevant operating conditions, and the system’s stated limits. A persuasive explanation generated after the fact is not the same as an auditable record of what happened.
Meaningful human oversight. “Human in the loop” should identify a real decision point. Who receives an alert? What information do they see? Can they reject an action? How much time do they have? Oversight is weak if the system acts before a person can intervene.
Predictable safety behavior. NIST identifies deterministic behavior, graceful degradation, and fail safe operation as important needs in critical infrastructure. That does not mean every AI model must produce an identical answer every time. It means the surrounding system needs clear boundaries and dependable responses when something goes wrong. For example, an uncertain diagnostic assistant might stop short of issuing an operational command and route the case to a qualified operator.
Testing under realistic conditions. A system that performs well in a demonstration may fail when sensors degrade, data patterns shift, or an attacker manipulates its inputs. NIST specifically highlights adversarial robustness and rigorous testing across the lifecycle. Organizations should test the deployed system, including its data sources, interfaces, human workflows, and fallback procedures, rather than judging the model in isolation.
What CISOs and operators can do now
You do not need to wait for a finished profile to ask better questions.
Start with an inventory of AI use cases that touch essential operations. Include tools bought by individual teams, AI features added to existing products, and vendor services that influence monitoring or incident response. For each use case, record what the system can observe, recommend, or change. The distance between “drafts a report” and “issues a command” should drive the depth of review.
Next, define the operating boundary. Document where the system has been tested, what data quality it needs, who can approve its actions, and what should happen when those assumptions fail. A useful question is: *What would make us stop trusting this output today?* If no one can answer, the boundary is probably too vague.
Then request evidence that matches the risk. Ask vendors for test results, known limitations, version and change records, incident procedures, and an explanation of how operators can review decisions. An AI bill of materials, which describes relevant AI components and dependencies, may help teams understand what is in a system and what changes over time. NIST includes such documentation in an example of traceable diagnostics.
Finally, exercise the failure path. Test what happens when a sensor provides bad data, connectivity is lost, a model update changes outputs, or an operator disputes a recommendation. A fallback procedure has value only if people know when and how to use it.
What AI builders should take from this
If your company sells into infrastructure, design for scrutiny early. Give customers clear ways to limit permissions, inspect recommendations, review changes, and disable automated actions. State where your product has been tested and where it has not. Make failure modes understandable to the people who will operate it under pressure.
That discipline can help buyers evaluate a product on more than a polished demo. It can also expose design problems while they are still inexpensive to fix.
NIST is inviting operators, vendors, researchers, regulators, and others to help shape the profile through its [community of interest](https://www.nist.gov/programs-projects/concept-note-ai-rmf-profile-trustworthy-ai-critical-infrastructure). Organizations with real operational experience should participate. A useful profile must account for legacy equipment, distributed assets, limited resources, and the practical tradeoffs of keeping essential services running.
The question for leaders is straightforward: If an AI system influences an essential service, can you show why its output should be trusted, who controls its actions, and what happens when its assumptions fail? NIST’s proposed profile is an opportunity to make those answers clearer before AI becomes harder to separate from the systems on which people rely.