Physical AI: models that act in the real world
What you’ll learn
Section titled “What you’ll learn”- What “physical AI” means and how it differs from AI that only produces text or images.
- How a physical AI system is layered: perception, a learned policy, planning and control, and a safety layer.
- Why the learned model runs slowly and the safety layer runs fast, and why that split matters.
- How simulation, synthetic data and foundation models such as vision-language-action (VLA) models are used.
- The failure modes and the engineering discipline that physical systems demand.
Summary
Section titled “Summary”Physical AI is AI that perceives the physical world through sensors and acts on it through motors: robots, autonomous vehicles, drones, and machines on factory and warehouse floors. The outputs are not words but forces and motion, and that changes almost everything about how such systems are built. Mistakes can injure people, decisions have hard deadlines measured in milliseconds, and the model must run on the machine because a network round trip is too slow and too unreliable.
The current wave of interest comes from applying foundation-model ideas to robots. Vision-language-action (VLA) models take camera images and a language instruction and output robot actions; NVIDIA’s open Isaac GR00T N1.7, for example, combines a vision-language backbone with a diffusion transformer that generates continuous actions, and is post-trained for a specific robot body. These models are trained on a mix of real robot demonstrations, human video and simulation.
What has not changed is the engineering around the model. A learned policy proposes; a simpler, verifiable control and safety layer decides what actually reaches the motors. The worked example on this page shows that split in miniature.
Where physical AI is used
Section titled “Where physical AI is used”- Warehouses and logistics: autonomous mobile robots moving goods, robotic picking of mixed items, inventory drones.
- Manufacturing: visual quality inspection, robot arms guided by vision, collaborative robots (cobots) working beside people, and predictive maintenance from machine sensor data.
- Autonomous vehicles: driver-assistance systems in production cars, driverless ride-hailing services operating in a number of cities (Waymo is the best-known example), and autonomous trucking programs. Approaches differ, from camera-only systems to stacks that fuse cameras, lidar and radar.
- Humanoid and general-purpose robots: several companies, including Figure AI, Boston Dynamics, Agility Robotics and Tesla, are developing humanoids aimed at factory and logistics work. Capabilities and deployment claims change quickly, so check each company’s current material.
- Healthcare: surgical robots that extend a surgeon’s precision, and rehabilitation and assistive devices. Diagnostic imaging AI analyzes data but does not act physically, so it is usually classed separately.
Context and forces
Section titled “Context and forces”| Concern | Digital AI (chat, code, images) | Physical AI |
|---|---|---|
| Output | Text, images, code | Motion, force, physical decisions |
| Cost of a mistake | Usually reversible: retry, edit, roll back | Possibly irreversible: damage or injury |
| Time budget | Seconds are acceptable | Control loops run hundreds or thousands of times per second |
| Where it runs | Cloud, at scale | On the device, within a power and heat budget |
| Data | Abundant text and images | Scarce, expensive robot data; heavy use of simulation |
| Assurance | Evaluations and human review | Safety standards, certification, hazard analysis |
These forces push toward a layered architecture where each layer has a different rate, a different failure behavior and a different kind of evidence that it is safe.
The design: a layered stack
Section titled “The design: a layered stack”- Sensing and perception. Cameras, depth sensors such as lidar or time-of-flight, radar, force and torque sensors, and joint encoders. Perception turns raw signals into objects, poses and free space. Sensor fusion combines sources so that one sensor’s blind spot (glare for a camera, glass for lidar) is covered by another.
- Learned policy or planner. Decides what to do next: a VLA model mapping images and an instruction to actions, a reinforcement-learned policy, or a classical planner. This is where foundation models sit. It typically runs at tens of hertz; GR00T N1.7’s repository, for example, reports about 28 Hz on a Jetson AGX Thor with TensorRT.
- Motion planning and control. Turns intentions into smooth, feasible trajectories and closes the loop on motor commands at hundreds to thousands of hertz, using well-understood control methods.
- Safety layer. Independent of the learned model: speed and force limits, protective stops, watchdogs on every upstream component, and safety-rated sensors. On certified systems this layer runs on separate, safety-rated hardware and software.
- Fleet and operations. Logging, remote monitoring, over-the-air updates, incident review and data collection for the next training round.
On robots, the middleware that connects these layers is usually ROS 2 (Robot Operating System 2): nodes publish and subscribe to typed topics such as camera images and velocity commands. In November 2026 the supported ROS 2 distributions are Lyrical Luth (released May 22, 2026, end of life May 2031), Kilted Kaiju (end of life December 2026), Jazzy Jalisco (to May 2029) and Humble Hawksbill (to May 2027).
Simulation and sim-to-real
Section titled “Simulation and sim-to-real”Real robot data is slow and expensive to collect, and some situations are too dangerous to stage. Teams therefore train and test in physically based simulation, for example NVIDIA Isaac Sim and Isaac Lab, which can run thousands of simulated robots in parallel, and use simulation to generate synthetic training data. The hard part is the sim-to-real gap: friction, lighting, sensor noise and timing never match reality exactly. Techniques such as domain randomization (varying textures, masses, delays and noise in simulation so the policy learns to tolerate them) narrow the gap, and real-world testing closes it.
Edge inference
Section titled “Edge inference”The learned model runs on the robot. Embedded GPU modules such as NVIDIA’s Jetson family, or automotive-grade compute in vehicles, run quantized models within a power budget of tens of watts. The cloud is used for training, fleet analytics and map updates, not for the control loop. A design that needs the network to stay safe is not safe.
Running the sample
Section titled “Running the sample”The sample is plain Python 3 (tested with Python 3.11; no packages needed). It simulates four seconds of a robot’s control loop with stubbed sensors and a stubbed policy, so it runs deterministically on any machine. Save it as safety_loop.py and run python3 safety_loop.py. On a real robot the same logic would live in a ROS 2 node or, for certified functions, in a safety controller.
Worked example: a safety layer for an OrdersPortal warehouse robot
Section titled “Worked example: a safety layer for an OrdersPortal warehouse robot”An invented shop, OrdersPortal, uses autonomous mobile robots to carry totes from picking aisles to the packing station. A learned policy proposes a speed five times per second. A deterministic safety layer runs one hundred times per second and has the final say: it clamps speed to the site limit, stops when the distance sensor sees something too close, and stops when the policy has gone quiet.
# safety_loop.py - a slow AI policy proposes speeds; a fast, simple safety layer decides what reaches the motors.# Simulates 4 seconds of an OrdersPortal warehouse robot driving a tote to the packing station.CONTROL_HZ, POLICY_HZ = 100, 5 # fast deterministic loop vs slow learned policyMAX_SPEED = 1.2 # m/s, set by the site's risk assessment, not by the modelSTOP_DISTANCE = 0.8 # m, stop if anything is closer than thisPOLICY_TIMEOUT = 0.5 # s, stop if the policy has gone quiet this long
def policy(t: float) -> float: """Stand-in for a learned model (for example a vision-language-action model) on the robot's GPU.""" if 2.0 <= t < 2.9: raise TimeoutError("inference stalled") # simulate a slow or crashed model return 1.6 if t < 1.0 else 1.0 # first command exceeds the speed limit
def obstacle_distance(t: float) -> float: """Stand-in for the safety-rated distance sensor.""" return 0.6 if 1.4 <= t < 1.7 else 3.0 # a person steps into the aisle
def safety_layer(t, proposed, last_policy_time): if obstacle_distance(t) < STOP_DISTANCE: return 0.0, "obstacle" if t - last_policy_time > POLICY_TIMEOUT: return 0.0, "policy timeout" if proposed > MAX_SPEED: return MAX_SPEED, "clamped" return proposed, "ok"
proposed, last_policy_time, last_reason, position = 0.0, 0.0, None, 0.0for step in range(4 * CONTROL_HZ): t = step / CONTROL_HZ if step % (CONTROL_HZ // POLICY_HZ) == 0: # policy runs every 20 control ticks try: proposed, last_policy_time = policy(t), t except TimeoutError: pass # keep the old proposal; the watchdog decides speed, reason = safety_layer(t, proposed, last_policy_time) position += speed / CONTROL_HZ if reason != last_reason: # print only when the decision changes print(f"t={t:4.2f}s proposed={proposed:.1f} sent={speed:.1f} m/s {reason}") last_reason = reasonprint(f"distance travelled: {position:.2f} m")Output:
t=0.00s proposed=1.6 sent=1.2 m/s clampedt=1.00s proposed=1.0 sent=1.0 m/s okt=1.40s proposed=1.0 sent=0.0 m/s obstaclet=1.70s proposed=1.0 sent=1.0 m/s okt=2.31s proposed=1.0 sent=0.0 m/s policy timeoutt=3.00s proposed=1.0 sent=1.0 m/s okdistance travelled: 3.21 mWhat the example shows:
- The model never has the last word. It proposed 1.6 m/s; the motors received 1.2 m/s, because the limit comes from the site’s risk assessment, not from training.
- Stale is dangerous. When inference stalled at 2.0 s, the last command looked valid. The watchdog stopped the robot half a second later, because a command nobody is refreshing is a command nobody is checking.
- The safety layer is small enough to verify. A few comparisons can be reviewed, tested exhaustively and, on real products, implemented on safety-rated hardware. A neural network cannot be verified that way.
- Simplifications. A real system decelerates within physical limits rather than stepping to zero, uses safety-rated sensors and stop functions, and considers many more hazards than these three.
Trade-offs and alternatives
Section titled “Trade-offs and alternatives”| Approach | Strength | Weakness | Fits |
|---|---|---|---|
| Classical pipeline (perception, planner, controller, hand-written rules) | Predictable, analyzable, mature tooling | Brittle in unstructured settings; every new task is engineering work | Structured factories, fixed routes |
| End-to-end learned policy (for example a VLA model) | Generalizes to new objects and instructions; learns skills from demonstrations | Hard to verify; data hungry; failures are hard to explain | Manipulation in varied environments, research, flexible tasks |
| Hybrid: learned policy inside a classical safety envelope | Flexibility of learning with bounded behavior | Two systems to build and keep consistent | Most commercial deployments |
| Teleoperation with AI assistance | A person handles the hard cases; data for training | Labor cost; latency; does not scale | Early deployments, data collection |
Failure modes and operations
Section titled “Failure modes and operations”- Distribution shift. The world differs from the training data: new lighting, a new tote color, a reflective floor. Monitor perception confidence and intervention rates by site, and collect the failures as training data.
- Sim-to-real gap. A policy that is perfect in simulation slips, collides or oscillates on hardware. Randomize simulation widely and gate releases on real-world tests.
- Timing failures. Late, stale or dropped messages cause wrong actions even when every component is correct. Use timestamps, deadlines and watchdogs on every link.
- Sensor degradation. Dirty lenses, miscalibration and interference. Detect it with self-checks and cross-sensor consistency, and degrade to a safe state.
- Adversarial and unusual inputs. Physical patterns that confuse vision models, or instructions that are unsafe. Keep the safety layer independent of perception wherever possible.
- Fleet updates. A new model version reaches hundreds of machines at once. Use staged rollouts, shadow mode (the new model proposes, the old one acts, compare) and fast rollback.
- Human factors. People trust a robot that has behaved well, then stop watching it. Design for predictable motion and clear signals of intent.
When not to use physical AI
Section titled “When not to use physical AI”- The environment is fixed and the task repetitive. A conventional programmed machine or conveyor is cheaper, faster and easier to certify.
- You cannot build the safety case. If you cannot bound the system’s behavior with independent safety functions and evidence, a learned policy should not move near people.
- The data does not exist. Without demonstrations, simulation that matches your setting, or a budget to collect them, a learned policy will not reach reliability.
- A person with a tool is cheaper at your volume. Robots pay off at scale and over years; one-off tasks rarely justify them.
Interview questions
Section titled “Interview questions”1. Why do physical AI systems separate a learned policy from a safety layer? Learned models generalize but cannot be exhaustively verified and can fail unpredictably. A simple, deterministic safety layer with limits, protective stops and watchdogs can be verified and certified, so it bounds whatever the model proposes.
2. What is a vision-language-action model? A model that takes images and a natural-language instruction and outputs robot actions, typically built from a pre-trained vision-language backbone plus an action head; GR00T N1.7 uses a diffusion transformer head that generates continuous actions. It is post-trained for a specific robot body and task.
3. What is the sim-to-real gap and how do teams reduce it? The difference between simulated and real physics, sensing and timing, which makes policies trained in simulation fail on hardware. Teams reduce it with domain randomization, better physical models and sensor noise models, a mix of real and synthetic data, and real-world validation before release.
4. Senior scenario: OrdersPortal wants to replace its fixed-route warehouse robots with ones driven by a new VLA model that can follow spoken instructions from staff. You own the rollout. What is your plan? Keep the existing safety architecture and treat the model as a new planner inside it: the same speed and force limits, protective stops, safety-rated sensors and watchdogs, unchanged and independently validated. Start in simulation with the warehouse’s real layout and randomized clutter, then run the model in shadow mode on the current robots, where it proposes actions that are logged and compared but never executed. Move to a fenced pilot zone during low-traffic hours, with intervention rate, near-misses and task success as the gating metrics, and expand by zone only when they meet targets agreed with the safety team. Constrain the language interface to an allowlist of task types and confirm actions before they start. Update the hazard analysis for the new failure modes (misheard instructions, unexpected paths), train staff on how the new robots signal intent, and plan staged over-the-air updates with rollback. Measure against the fixed-route baseline on throughput and incidents, because flexibility is only worth it if the warehouse is both faster and at least as safe.
Key takeaways
Section titled “Key takeaways”- Physical AI acts through motors, so latency, irreversibility and safety dominate the design.
- Layer the system: perception, a learned policy at tens of hertz, control at hundreds to thousands of hertz, and an independent safety layer.
- Foundation models such as VLA models broaden what robots can do, but they propose; verified safety functions decide.
- Simulation and synthetic data make training feasible; real-world validation makes it safe.
- Roll out like a safety-critical system: shadow mode, staged zones, measured interventions and fast rollback.
Related pages
Section titled “Related pages”- AI agents overview
- The spectrum of agency
- Microsoft small language models (on-device models)
- Running LLMs locally
- Agentic AI risks
- AI governance
Official references
Section titled “Official references”Checked on November 16, 2026.
- NVIDIA Isaac GR00T N1.7 repository: architecture, license (Apache 2.0 code, NVIDIA Open Model License weights) and inference rates
- NVIDIA Isaac GR00T overview and Jetson Thor
- ROS 2 documentation: distributions
- Isaac Lab documentation