AI Safety Should Focus on Control

Our approach to safety should grant us the serenity to accept the things we cannot change, the courage to change the things we can, and the wisdom to know the difference.

— Adapted from the Serenity Prayer, attributed to Reinhold Niebuhr

Start with the system

Note: This article focuses on safety-critical systems exposed to generative and agentic AI, including AI operated by another organization. The same engineering approach applies to conventional software and human-controlled systems.

Public discussion of AI risk is full of proposals for mitigating the harm AI could cause. Many of them depend on conduct that the organizations exposed to harm cannot reliably control. We recognize that there is disagreement about whether AI presents the extraordinary danger described in current warnings. Some regard those warnings as grounds for urgent intervention; others consider the threat overstated.

Whether those warnings prove justified or overstated, safety-critical systems remain vulnerable to human actions, software flaws, malicious misuse and complex component interactions. This protective work is justified by the systems and potential losses at stake, not by predictions about AI.

The practical priority is to examine the design choices that owners and operators of safety-critical systems can make, beginning with whether AI should have operational authority at all. Endrisk advocates retaining non-AI control of critical functions. Where an organization nevertheless introduces AI, it should demonstrate that an authorized person can withdraw its operational authority, that the transition can be made safely, and that essential functions can continue without it. Continued operation may rely on conventional automatic control or manual operation. The necessary controls, information, people and skills must remain available throughout the system's life.

These are design requirements we propose evaluating through whole-system safety analysis, not conclusions that follow merely from choosing the method. The analysis must cover the whole relevant system, including internal AI, external influences and the arrangements needed for operation without AI. An organization's authority determines which changes it can implement; it does not determine where the analysis should stop.

The common weakness in today's proposals

The leading responses to AI risk are to slow development, embed outside evaluators, adopt shared standards and coordinate across companies and countries. Each is worth pursuing and can contribute to safety, but its existence does not establish that a particular system's hazards are controlled. Their contribution depends on what they change in the design, operating arrangements and supporting evidence. In our assessment, these proposals are unlikely to provide dependable primary protection where they rely on participation, understanding and conduct that the organizations exposed to harm cannot ensure.

• Slowing down can create time for analysis, testing and redesign. Development assurance and real-time intervention are different problems: additional development time does not repair a design that requires a person to stop an unsafe action faster than the person can respond. A slower program can deliver the same flawed system later unless the additional time changes the design, the evidence or the conditions of deployment. A slowdown also depends on participation. A critical-system owner cannot base protection on every developer, competitor or external user honoring the same pause. Even where a pause leads to a safer release, the owner remains exposed to systems outside that arrangement.
• Outside evaluators can reveal problems and, when connected to effective authority, require corrective action. But their access, capacity and understanding remain bounded. They may share the developer's information, assumptions and blind spots. Embedding a reviewer in the organization does not remove those dependencies. Even an effective review of one developer leaves other sources of unsafe influence outside that arrangement. Evaluation also needs a defensible link to the actual operating conditions. A result obtained with different tools, safeguards, incentives or supervision cannot be assumed to establish the protection available in service.
• Common standards can spread a common mistake. If a shared rule carries a flawed assumption or misses a hazard, organizations relying on it can reproduce a shared vulnerability. Different company names do not establish independence when the relevant assumptions or dependencies are shared.
• Coordination cannot bind actors outside the arrangement—a competitor, a new entrant or another country that declines to take part. Cooperation must be treated as something that can be incomplete or withdrawn, not assumed.

Control is a whole-system design decision

The starting point is the safety of the whole system, rather than an assumption that AI is included and then supervised. Identify the unacceptable losses, examine how the system could enter hazardous states, and derive requirements for its design and operation. The systems-theoretic safety and security work of William Young and Nancy Leveson is part of the foundation for this perspective. [1]

Endrisk's proposed design position is to keep critical control functions independent of AI. The analysis should examine how that independence would be established and maintained, including indirect influence through information, maintenance, configuration changes and other services. Where AI is introduced, the analysis must also address its operation, withdrawal of its authority, the transition and continued operation without it.

Work through the design options in order of precedence:

1. Eliminate first: Consider eliminating the hazardous condition, unnecessary connection or authority.
2. Reduce by design: Reduce remaining hazards by design before relying on containment and damage reduction. Where stronger design options are not selected, record the reasons.
3. Demonstrate the protection: A hazardous effect can justify a protective requirement before every initiating mechanism is resolved. The protection still needs evidence of its coverage, physical capability and response margin.

Consider a critical process operated by conventional automatic controls. An organization might use AI to explore maintenance options or operating plans without giving it authority to change protective settings or issue operating commands. The analysis would examine whether those limits are actually enforced and whether reliance on AI advice introduces other hazardous interactions. If AI is subsequently given an operational role, the organization would need to establish how that authority can be withdrawn and how the process can continue safely without it. Neither separation nor continued operation should be assumed merely because the design includes an override.

Protection also has to address required actions that are missing, mistimed or stopped too soon. The system must preserve the functions needed for safety, as well as block harmful commands.

A test to apply to the systems we need to protect

For every serious loss, ask: If the AI acts wrongly, the evaluator misses it, the governing standard has a gap and the deploying organization proceeds anyway, what in the affected safety-critical system still prevents the loss?

If the only answer is that someone was supposed to notice, or that a company promised not to proceed, the loss is not yet controlled. The answer should identify the design features and operating arrangements that prevent or contain the hazardous condition. It may involve withholding AI authority altogether, changing the allocation of functions, removing a hazardous interaction, or establishing a protective function whose adequacy has been demonstrated.

The answer cannot be assumed to be another controller checking the AI. Where the design relies on a separate protective function, establish the independence needed for that function:

• The AI cannot alter or bypass it within the assessed operating and threat conditions;
• The commercial organization cannot quietly waive it;
• It does not simply repeat the same model, data or assumptions that might be wrong together; and
• It acts before the hazard outruns the response.

Assess shared dependencies in sensors, data, administration, updates, power and physical support, including conditions that could disable protection and observation together. Separate component names or suppliers do not establish separate means of protection, especially where their requirements share the same omission or incorrect assumption.

If everyone else misses it, what still prevents the loss? Consider what evidence shows that the protection is implemented, available and effective in those conditions, and who acts when that evidence no longer holds.

Keep essential functions available without AI

Withdrawing AI authority is more than stopping a program or disconnecting a service. The remaining control arrangement must have the information, equipment, procedures and personnel needed to carry out its responsibilities. Where conventional automation performs those functions, it must remain usable without depending on the AI for essential inputs or configuration.

The transition and subsequent operation need to be demonstrated under the conditions in which they will be required. Any necessary staffing, skills and operating procedures must be maintained and exercised, rather than reconstructed during an emergency. Operation without AI may have different capacity or operating limits, but it must meet the specified safety and essential-service requirements.

This is not a proposal to assign a person an indefinite monitoring task and expect a split-second rescue. Where prevention of a loss depends on human recognition and intervention, the design still has to establish that the person can obtain the necessary information and act within the available time. Where that cannot be established, a nominal override provides no basis for accepting the design.

The ability to operate without AI does not by itself establish safety while AI is active, and withdrawal of authority does not undo effects already set in motion. If safe transition and essential operation without the AI cannot be established, Endrisk's position is not to introduce it into that critical role.

What this looks like in practice

Controlling consequences means:

• Retaining non-AI control of critical functions; where AI is used for advice, separating capability from authority and assessing the effects of the advice itself;
• Bounding what the AI and the organization can reach, change and cause, including combined, delegated, accumulated and indirect effects;
• Where separate protective functions are used, enforcing decisive limits through components the AI cannot modify and the release chain cannot casually waive;
• Preserving cancellation, containment and recovery, with rollback limited to effects and state changes that can actually be reversed;
• Demonstrating that assigned human intervention is feasible, with usable information, time, authority and independent evidence to act and verify the effect; and
• Applying an assessed restriction, handover or safe transition when a safety-relevant change invalidates the evidence supporting an activity, while preserving necessary safety functions.

Operational and verification principles

• Restrictions can create hazards: A restriction or shutdown can itself create a hazardous state. Assess missing, inappropriate, mistimed or prolonged intervention and harmful combinations with other controls. A pre-start hold and interruption of an essential live service require different safety arguments.
• Integrity of evidence and governing decisions: Protection should extend to the evidence and decisions that govern the systems we want to protect. This includes limiting authority to alter safety records, acceptance criteria or protective settings. Preserve the distinction between what was requested, what was accepted, what was executed and the observed effect. Decisions to approve, modify or restore operation need evidence that remains accessible and trustworthy when the source or its interpretation is questioned.
• Summaries and handovers: A misleading summary can obscure a material finding even when the original records are intact. Preserve governing constraints, source versions, uncertainty, contrary evidence and outstanding obligations through summaries and handovers. A summary must not turn a proposed action into permission or silently change an acceptance criterion. Recommendations should not be presumed authoritative just because they were repeated, retained or echoed by a second reviewer. Agreement based on the same unverified account is not independent verification; consequential claims need review against accessible underlying evidence.
• Control the action, not just the entry: Approving a connection or request does not authorize every subsequent action. The receiving system must enforce limits on where actions go, what purpose they serve, under what conditions they run, and what cumulative impact they might cause.
• Monitor combined activity: Assess the combined activity of connected systems rather than single requests in isolation. Evaluate how repeated tasks and delegated work add up against the system's capacity.
• Cancellation and state reconciliation: Revoking an agent's access does not establish that delegated or queued work has stopped. The receiving system needs its own cancellation or containment path and confirmation of what actually stopped. Reconcile outstanding work and retained state before restart; where a consequence cannot be reversed, protection has to act before that commitment.
• Traceable implementation plans: Put each protective requirement into an implementation plan with an engineering or management owner, acceptance criteria and a completion date. Trace it to the unsafe action or interaction it addresses. Coordinate procedures, training and external dependencies with the technical design.
• Realistic verification and bounded claims: Verify actual use and protective effect under representative conditions, then monitor the assumptions and changes that could invalidate that evidence. Testing provides evidence within a defined scope; it does not establish complete hazard coverage. Safety claims should identify what the tests support, the assumptions needed to apply the results, and what remains unverified.

Limitations

• Limits of external control: No single organization can stop every company or country from building or deploying AI. Organizations can decide which functions they assign to AI and define the design and operating changes within their authority. The whole-system analysis still needs to represent the dependencies they cannot directly control.
• Controlled interfaces: That work includes controlling the interfaces through which AI could gain consequential authority—credentials, networks, transaction systems, data-release paths, physical equipment, operating permissions and protected infrastructure—with protection that remains effective within its demonstrated scope even when an external actor does not cooperate. Where an effect can bypass every controlled interface, that limitation should be stated and clearly defined, and the design reconsidered, rather than resting on an unstated assumption that everyone will cooperate.
• Upstream responsibility: The same principle applies upstream: developers, providers and operators remain responsible for the controls within their authority. Protecting a receiving system does not transfer every responsibility to its last human operator.
• An engineering standard, not speculative certainty: This is not a promise of zero risk, and not a claim that any AI is safe in the abstract. It is a claim about engineering: the safety of a consequential system should rest on the adequacy of the whole operating arrangement, supported by evidence for the actual deployment—not on the hope that its builders, reviewers, standards and competitors all behave as intended.
• Methodological boundaries: A systematic analysis does not prove that every real-world pathway has been identified. Safety claims should state their scope, operating conditions, assumptions and evidence. Unknowns remain open; a completed analysis, policy or installation record is not evidence of operational effectiveness.

Critical functions should not become dependent on AI merely because an organization chooses to deploy it. Whether AI is excluded or introduced, the safety of the whole operating arrangement, including any transition away from AI, has to be established. Protecting critical systems need not wait for agreement about AI. This work addresses hazards involving people, conventional software and other sources even where AI presents no exceptional threat. The resulting protections must be implemented and verified, whether the warnings prove justified or overstated.

Related reading

[1] William Young and Nancy G. Leveson, “An Integrated Approach to Safety and Security Based on Systems Theory,” Communications of the ACM 57(2), 31–35 (2014).

About Endrisk

Endrisk develops collaborative system-safety software for complex, software-intensive systems. Its planned platform connects hazards, safety constraints, requirements, assumptions, evidence and change in one defensible record.

Endrisk's mission is to modernize system safety so organizations can maintain control of consequential systems.

For further details, contact Endrisk: contact@endrisk.io.