Human-in-the-Loop Is Non-Negotiable: Leading AI Adoption in Safety-Critical Systems

Self-driving cars operate at the sharp end of cyber-physical safety, where even minor mistakes can instantly transform two tons of steel into a deadly hazard. The autonomous vehicle industry has experienced catastrophic incidents resulting in fatal and life-changing injuries—Uber in 2018, Cruise in 2023. Yet analysts project a $500-billion mobility market by 2030 for companies that achieve safe and scalable automation.

The essential question facing leaders: Where exactly do we place human oversight in automated systems? And how do we ensure those human-machine interfaces remain reliable, clear, and effective as complexity grows?

According to the recent paper “Leading the Human-AI Revolution” published in the Fall 2025 Enterprise Technology Leadership Journal, the answer is non-negotiable: For safety-critical domains, human-in-the-loop, human-on-the-loop, and oversight are immutable.

The Leadership Challenge Beyond Technology

Building and operating safety-critical cyber-physical systems (CPS) with AI integration presents a formidable leadership challenge that extends beyond technology. Incorporating AI in the design, development, deployment, and operations of these systems involves complex interactions between technology and social systems—people, processes, culture, and organizations.

While AI can greatly enhance system development through automation, adaptability, and data-driven insights, it can also disrupt established engineering practices, existing roles, and cultural norms. The challenge isn’t just technical—it’s deeply human.

As AI becomes increasingly embedded in CPS—from autonomous vehicles to healthcare monitoring systems to factory robots—leaders must deepen their focus on human interactions throughout development and consider the human-AI experience in operations.

Two Models for Managing Risk

The strategy for managing risk in AI integration matches the level of human oversight to the level of operational risk. Two primary models achieve this:

Human-in-the-Loop (HITL): For high-risk or critical tasks, AI cannot operate without direct human input, ensuring a human is accountable for the final action. The human makes the decision; AI provides information and recommendations.

Human-on-the-Loop (HOTL): For lower-risk, supervised autonomous tasks, AI operates independently while a human actively monitors and can intervene or override the system at any time. The AI makes decisions, but humans maintain veto power.

Selecting the correct model is key to ensuring both operational control and adaptability. These strategies preserve accountability, enable real-time decision-making, and ensure system reliability.

Successfully integrating HITL and HOTL requires designing workflows that blend human expertise with AI-driven automation, alongside user interfaces that foster collaboration through transparency and build trust between humans and intelligent systems.

Case Study: Waymo vs. Uber—The Difference Human Oversight Makes

Waymo’s approach to fleet response exemplifies disciplined human-AI boundaries. Their Fleet Response Operations (FRO) team monitors vehicles remotely, with authority to guide vehicles through uncertain scenarios. When a Waymo vehicle encounters an ambiguous situation—construction zones with unclear signage, emergency vehicle presence—it can request human assistance without stopping in traffic.

The FRO specialist sees real-time sensor feeds, provides high-level guidance (“proceed cautiously through the construction zone using the rightmost lane”), and the vehicle executes using its own perception. This is HOTL done right: The human doesn’t micro-manage steering or throttle but provides strategic context the AI lacks.

Result: Waymo has logged millions of autonomous miles with significantly better safety records than competitors.

Contrast this with Uber’s 2018 fatal incident in Tempe, Arizona. The system design had critical flaws:

• Overconfident automation: The AV software detected the pedestrian but classified and reclassified her incorrectly multiple times. Rather than escalating to the human driver, it continued autonomous operation with high confidence in faulty perception.
  
• Insufficient human authority: The safety driver was expected to maintain attentiveness despite vigilance decay being a well-known risk in supervisory roles. The driver was watching a video on their phone.
  
• Systemic boundary breakdown: The human was nominally “in the loop” but received no system prompts or layered safeguards to remain engaged. The AV software offloaded responsibility to a human without supporting it with mechanisms for detection, escalation, or overriding dangerous overreliance on passive human backup.
  

Consequence: Fatality, program shutdown, and a landmark NTSB investigation that highlighted organizational and technical deficiencies.

The difference? Waymo designed for effective human oversight. Uber designed with humans as passive backup—and when that backup failed, the system failed catastrophically.

The Healthcare Example: Wearables and Remote Monitoring

Healthcare provides another critical example. Healthcare providers monitor patients remotely via wearables while leveraging AI to analyze data trends, reducing need for in-person visits and minimizing exposure to secondary infections.

AI facilitates autonomous decision-making and anomaly detection, improving operational resilience and reducing reliance on manual intervention. But the key word is “reducing,” not “eliminating.”

When AI detects anomalies in heart rhythms or blood glucose levels, it alerts human clinicians who make treatment decisions. The AI doesn’t prescribe medication or adjust treatment plans autonomously—that remains firmly in human control, with appropriate medical oversight and accountability.

Factory Floor Robotics: Where Human-Machine Collaboration Works

Collaborative robots (cobots) on factory floors demonstrate how strategic human-AI boundaries enhance both productivity and safety. At companies like Boeing and BMW, cobots execute precision drilling and fastening while human assemblers guide tasks and validate quality.

Effective implementation includes:

• Zone sensors and emergency stop networks monitored by line supervisors
• Technicians handling appropriate end-effectors based on tasks
• Skilled assemblers working with cobots, not replaced by them
• Clear protocols for who has authority in different scenarios

The result: Enhanced precision, streamlined workflows, reduced strain on human technicians, and significantly lower workplace injury rates.

Contrast with flawed implementations: Some factories deployed robots with insufficient safety boundaries, inadequate training, or unclear authority structures. Workers suffered injuries when robots moved unexpectedly, or when unclear protocols led to humans entering robot work zones without proper safeguards.

Successful human-machine collaboration hinges on well-defined operational boundaries, proactive safety measures, and workforce training—not as best practices, but as essential safeguards for human life, corporate integrity, and long-term operational sustainability.

The Security Imperative: Why Human Oversight Is a Security Layer

Security in cyber-physical systems is inseparable from safety. A system with autonomous capabilities can be misled or exploited in ways that have direct kinetic consequences—a vehicle misrouting, a drone misfiring, a robotic system overextending.

According to OWASP’s work on agentic AI security, the attack surface in AI-enabled systems includes:

• Reasoning and planning engines: If an LLM making decisions for a drone fleet is compromised or hallucinated, downstream consequences may be physical rather than informational.
  
• Communications infrastructure: Distributed autonomous systems depend on secure, real-time communications. Compromise here may allow rogue agents to impersonate trusted systems.
  
• Operator interfaces: HITL interfaces may under-communicate uncertainty or overstate confidence. An AI system presenting false output with high confidence can lead to poor human decisions in high-stakes scenarios.
  

Human oversight serves as a critical security layer. Humans can detect when AI behavior seems “off,” when confidence levels don’t match observable reality, when patterns suggest system compromise rather than legitimate operation.

The Leadership Checklist

Leaders responsible for safety-critical systems must answer these questions:

On Human Oversight:

• Are human oversight roles explicitly defined and consistently equipped with tools, access, and the authority needed to be effective?
• Does the system have protocols for escalating uncertain situations to human operators?
• Are all human-AI interactions, including overrides, systematically recorded and auditable?

On Safety and Security:

• Are safety boundaries clearly defined and strictly enforced to prevent human harm?
• Are emergency stop protocols in place and regularly tested?
• What safeguards prevent AI manipulation, misinformation, or adversarial interference?

On Training and Capability:

• Are rigorous training programs maintained to ensure personnel are competent and confident in interacting with automation?
• Have we modeled anticipated human workload and cognitive demand to ensure system resilience under stress, scale, and escalation scenarios?

On Life Cycle Management:

• Are human-AI interactions and roles mapped and trained across the system life cycle, including maintenance?
• Are simulations and digital twins systematically used to validate human-system interactions before deployment?

The Cross-Functional Imperative

To stay agile and competitive in the rapidly evolving AI landscape, technology leaders must collaborate strongly with cross-functional leadership teams to ensure alignment with organizational strategic vision.

The diverse leadership team should include representatives from:

• Engineering and operations
• Legal and compliance
• Human resources and workforce development
• Security and risk management
• Product management
• Ethics and governance

Working together with shared vision, leadership teams can harness AI’s full potential while ensuring alignment with strategic goals and meeting workforce needs. This approach not only accelerates learning and innovation but prevents “shadow AI”—unregulated AI use operating outside coordinated business-technology leadership.

AI Roles and Risk Levels

Understanding where different AI implementations fall on the agency-risk spectrum helps leaders make appropriate oversight decisions:

High Agency, High Risk: AI as autonomous agent making end-to-end decisions within predefined domains. Example: Inventory-replenishment bot automatically ordering stock. Requires robust monitoring, clear boundaries, and strong escalation paths.

High Agency, Low Risk: AI as collaborator/partner in iterative co-creation. Example: Writer leveraging AI to draft story segments, then editing and providing feedback. Human maintains final authority but benefits from AI assistance.

Low Agency, High Risk: AI as decision support providing recommendations. Example: Medical diagnosis system suggesting possible conditions. Human makes final diagnosis and treatment decisions.

What This Means for Your Organization

If you’re leading development or operations of safety-critical systems, recognize that AI integration is not purely a technical decision—it’s a socio-technical transformation requiring careful attention to:

Human factors: How will people interact with AI systems? What cognitive demands will they face? How do we prevent vigilance decay in monitoring roles?

Organizational change: What new roles and skills are needed? How do we train existing workforce? What career paths exist for people in hybrid human-AI roles?

Cultural evolution: How do we build trust in AI systems while maintaining healthy skepticism? How do we celebrate both AI successes and human interventions that prevent AI failures?

Regulatory alignment: How do we demonstrate to regulators that we’ve thought through human oversight? How do we document human-AI interactions for accountability?

The Non-Negotiable Principle

For safety-critical domains—autonomous vehicles, healthcare monitoring, industrial robotics, aviation systems, defense applications—human-in-the-loop and human-on-the-loop oversight are immutable requirements.

This serves as a guiding lens for identifying where trust, control, regulation, and governance must be strong, especially in high-risk, high-agency environments where the cost of failure is significant and institutional responsibility is paramount.

The organizations that thrive won’t be those that eliminate human oversight in pursuit of efficiency. They’ll be those that thoughtfully design human-AI partnerships that leverage the strengths of both—AI’s speed, consistency, and pattern recognition combined with human judgment, ethical reasoning, and contextual understanding.

When two tons of steel is moving at highway speeds, when a surgical robot is operating on a patient, when an industrial system is handling hazardous materials—human oversight isn’t a nice-to-have. It’s the difference between responsible innovation and catastrophic failure.

Design for it. Budget for it. Train for it. Make it non-negotiable.

This blog post is based on “Leading the Human-AI Revolution: Strategic Leadership Guidance for Effective Human-AI Interactions for Development and Operations of Safety-Critical Cyber-Physical Systems” by Dr. Suzette Johnson, Robin Yeman, Steve Wilson, Kim Harrison, and Christine Hudson, published in the Enterprise Technology Leadership Journal Fall 2025.

- About The Authors

Leah Brown

Managing Editor at IT Revolution working on publishing books and guidance papers for the modern business leader. I also oversee the production of the IT Revolution blog, combining the best of responsible, human-centered content with the assistance of AI tools.

Follow Leah on Social Media

•