Adventures and Amusings of a Mathematician
Go back

The Perception–Planning Gap: What's Actually Hard About Visual AI in 2026

The thesis I want to defend in this piece is structural: perception in pixels is largely solved at the representation level; perception for action is not, and the gap between the two is the central unsolved problem of visual AI in 2026. Frontier vision-language models can pass medical-board questions and explain radiographs at attending-physician level. Frontier robots, after a decade of foundation-model progress, still cannot reliably load an arbitrary dishwasher. Moravec’s paradox is not a quaint historical observation; it is the daily lived experience of every embodied-AI lab.

This post is a technical survey of why. I’ll walk through what is actually hard about visual perception, what is actually hard about planning, and where the four most active research frontiers — world models, vision-language-action policies, 3D foundation models, and embodied evaluation — actually stand. I’ll cite the work I’m leaning on, including LeCun’s position paper on autonomous machine intelligence, the JEPA family (V-JEPA, V-JEPA 2), the Dreamer line (DreamerV3), the world-model-evaluation literature (Physion, IntPhys 2, WorldModelBench), the VLA papers (RT-2, Open X-Embodiment, OpenVLA, π0), Helix from Figure, Gemini Robotics, and the recent feed-forward 3D work (DUSt3R, MASt3R, VGGT).

This is the long version of the argument.

1. What “perception” actually means, and why it has two failure modes

The word “perception” papers over two very different capabilities. The first is passive recognition: given an image or video, produce labels, captions, segmentations, or answers to questions. The second is active grounding for control: produce a representation that supports action — reaching, grasping, navigating, manipulating, planning over.

The 2010s vision community optimized almost exclusively for the first. ImageNet, COCO, Visual Genome, VQA, and their descendants are passive benchmarks. They reward models that map pixels to symbols. They do not reward models that produce metric geometry, action affordances, physical predictions, or counterfactual rollouts. The leaderboards converged. The capability did not.

The honest reading of the field in 2026 is that passive perception is mostly a solved problem at the representation layer. DINOv2-class self-supervised encoders (Oquab et al., 2023), SAM 2 (Segment Anything Model 2) for promptable segmentation, Depth Anything V2 for monocular depth, and frontier vision-language models or VLMs (GPT-4V class, Gemini 2.5, Claude with vision, Qwen2-VL, Molmo, the Llama 3.2 Vision family) collectively give you strong zero-shot 2D understanding. The remaining 2D failure modes are real but increasingly narrow: counting, fine-grained spatial relations, occluded-object reasoning, and adversarial corner cases. The benchmarks that still embarrass these models — BLINK, MMVP, CV-Bench, ARC-AGI-2 — are precisely the ones that probe perception as reasoning over geometry and composition, not perception as recognition.

A necessary caveat on the word “solved.” What I mean is solved on standard benchmarks under controlled conditions, not solved for deployment. A deployed perception system in a warehouse, an operating theater, or a household kitchen still routinely fails on lighting variation, novel materials, motion blur, partial occlusion under unusual viewpoints, and the long tail of objects that simply did not appear in pretraining. Autonomous-vehicle perception is the canonical case study: ImageNet was “solved” in 2015 and AVs still log perception failures in 2026, because real-world distributions are not bounded. The lab-to-deployment gap is wide for exactly the same reason it has always been wide — robustness in the wild is a different problem from accuracy on a held-out test set, and the way we measure progress systematically under-rewards the former.

Active perception is a different story entirely. Almost none of the foundation-model gains transfer cleanly to robots, and the reasons are now well-understood:

The structural problem is that the objectives that produce strong semantic representations are not the objectives that produce strong action-relevant representations. CLIP-style contrastive training optimizes for “image matches caption.” That is a useful signal, but a robot does not need to know that a coffee cup is called a coffee cup. It needs to know how heavy it is, how it deforms, where its handle is in 3D, what its center of mass implies for tipping, and what happens to the liquid inside if you tilt it.

This is what I mean by “the perception–planning gap.” It is not that perception is unsolved. It is that the kind of perception we have solved is the wrong kind for the next set of problems.

A short detour: why vision suffers more from this gap than text or speech

It is worth asking why this gap is so much wider in vision than in language. Three reasons, in order of importance.

Embodiment and irreversibility. Text actions are cheap and reversible. Saying the wrong sentence costs nothing; saying it again is free. Visual actions, in the embodied sense, are not. A robot that grasps wrong drops the cup. The world does not let you re-roll. This asymmetry shapes everything downstream — data collection, evaluation, and the kinds of objectives you can train against.

Closing the human-feedback loop is cheap for language and expensive for vision. Reinforcement learning from human feedback (RLHF) worked for language because language is the substrate of human preference: you can ask a person to rate two sentences in seconds, at scale, on Mechanical Turk. The visual analogue — asking a person to rate two robot trajectories, or two physical reconstructions of a scene — is dramatically more expensive per labeled sample, and the labels are noisier. There is no Mechanical-Turk-for-physics.

Vision is bound to a continuous, uncountable distribution. Text is discrete. Speech is one-dimensional. The set of possible visual scenes a robot might encounter is, for practical purposes, a measure-zero subset of an extremely high-dimensional manifold, and the parts that matter for action are the parts of the manifold least represented in any natural-image corpus. This is not a temporary data problem; it is a structural one.

The implication is that the playbook that worked for language — pretrain on the internet, post-train with RLHF, deploy — does not straightforwardly apply to embodied vision. The next section makes the same point from the planning side.

2. What “planning” actually means, and why long horizons are hard

Planning has the same two-level problem. There is planning in tokens — language-model agents producing sequences of tool calls, math derivations, code edits — and there is planning in the world, where actions have continuous, irreversible, partially observed consequences.

Planning in tokens has improved dramatically. ReAct (Yao et al., 2022) and its descendants, Tree of Thoughts (Yao et al., 2023), and the newer reasoning-model paradigm (the o-series, DeepSeek-R1, Claude with extended thinking) closed a meaningful chunk of the planning gap on math, code, and structured agentic tasks. The reason it works is not principled — it works because text environments tolerate retries. Make a wrong tool call, recover, try again. The world does not always grant that affordance.

Planning in the physical world is bottlenecked by four interlocking problems.

Long-horizon credit assignment. Beyond ~50–100 decisions, search blows up combinatorially and value estimates drift. Hierarchical reinforcement learning (HRL) was supposed to fix this with temporal abstraction (options, sub-goals, skills). It has not, at least not at the scale needed for general embodied agents. The decades of work since Sutton, Precup & Singh (1999) on options has produced strong narrow results and no general method. That said, the field is not stuck. Several active strands are nibbling at the edges: hindsight experience replay (HER, Andrychowicz et al., 2017) and goal-relabeling more broadly turn failed trajectories into useful supervision; off-policy importance-weighted methods reduce the variance of value estimates over long horizons; learned reward models from preference data — the same mechanism behind RLHF — give a path to dense supervision without dense human labels; and the causal-inference / RL interface (Buesing et al., 2018, Lu et al., 2020) is starting to formalize counterfactual credit assignment in ways the pre-2020 literature could not. None of these is a general solution. Together they suggest the bottleneck is more likely to crack than it has at any point in the last twenty-five years.

World-model fidelity. To plan in imagination, you need a model accurate enough that the imagined consequences track reality. This is the topic of section 3.

Goal and reward specification. RLHF and RLAIF work for language because language is the substrate of human preference. For embodied tasks the substrate is much harder. “Wash the dishes” decomposes into thousands of sub-goals that are easy to specify in natural language and brutally hard to specify as a reward function. The reward-hacking literature (Pan et al., 2022, Skalse et al., 2022) has documented dozens of cases where optimizers find degenerate solutions that satisfy the letter of a reward but not the intent.

Compositionality of skills. Stitching learned primitives into novel sequences without retraining is the classic neuro-symbolic gap. We have policies that can pour and policies that can stir; we do not have a clean recipe for combining them on demand to make tea.

The first three problems are research-hard. The fourth is, in some sense, the deepest — because it is the bottleneck on the entire general-intelligence agenda, not just robotics.

3. The world-model debate

Three camps are fighting over the same word.

The generative-pixel camp — Sora, Veo 3, NVIDIA’s Cosmos, Genie 2 and Genie 3 from DeepMind — bets that scale plus action conditioning leads to emergent physics. The training signal is pixel reconstruction or denoising; the architectural commitment is autoregressive transformers or video diffusion; the deployment story is increasingly explicit, with NVIDIA pitching Cosmos as the foundation substrate for “physical AI” simulation.

Genie 3 is the strongest existence proof of this camp’s bet — real-time interactive worlds at decent fidelity, actions affecting future frames in roughly the right ways. The visual quality is staggering. The physical reliability, under controlled probes, is not.

The latent-predictive camp is LeCun’s program, articulated in the autonomous-machine-intelligence position paper and built out as the JEPA family (Joint Embedding Predictive Architecture). The architectural bet is that pixel prediction is wasteful: you should predict the parts of the latent state that matter for control, not the texture of the next frame. V-JEPA 2 (2025) is the headline result — action-conditioned latent rollouts used for zero-shot manipulation planning. Unpacked: instead of generating the next video frame given an action, the model is trained to predict the next latent embedding of the scene given an action, and a planner searches over candidate action sequences by rolling them out in this latent space. The visual blindness is intentional; the question is whether the latent representations are rich enough to plan over realistic distributions.

The compact-world-model-for-RL camp — DreamerV3 (Hafner et al., 2023), IRIS, TWM, DIAMOND — actually closes the loop. Train a small recurrent or transformer world model, plan or learn a policy in imagination, deploy. DreamerV3 was the first general method to solve Minecraft diamond from pixels with no task-specific tuning. The narrowness of the application domain is the price for actually working.

The empirical evidence on the central question — can generative video models plan? — is now starting to come in, and it is not flattering to the pixel camp. WorldModelBench (2025), Physion, IntPhys 2 (2025), and the various Sora and Kling physics-violation studies converge on the same finding: generative video models hallucinate plausible-looking but counterfactually wrong physics. The output looks like a world. It does not respond to interventions like a world. Nudge an action — push the cup left instead of right — and the rollout drifts in ways that real physics does not allow.

The crux is this: counterfactual fidelity under action conditioning is the missing capability, not visual realism. A model that produces beautiful frames but cannot tell you what happens when you push the cup is not a world model, regardless of how its output looks on a website. It is a video decorator with conditioning.

The honest reading: the latent-predictive camp has the cleaner formulation but weaker visual outputs; the generative-pixel camp has the most spectacular demos but the weakest physical grounding; the compact-RL camp has the only end-to-end working pipeline but in narrow domains. Nobody has the full thing.

If world models are the imagination substrate for planning, the next question is what is being built on top of them in the meantime. That is the VLA story.

4. Vision-language-action models — the actual robotics frontier

The fastest-moving area of embodied AI in 2024–2026 is the vision-language-action (VLA) line. The lineage is:

RT-1 (2022) — first transformer policy at robotics scale. RT-2 (2023) — co-trained with web vision-language data, established that web pretraining transfers to robot control. RT-X / Open X-Embodiment (2023) — pooled robot data across 22 institutions and dozens of embodiments, ~1M episodes total. The first credible attempt at an “ImageNet for robots.” OpenVLA (2024) — open 7B VLA, the standard baseline. π0 (Physical Intelligence, 2024) — flow-matching action expert grafted onto a VLM backbone, trained on internet plus a large proprietary teleop corpus. Among the strongest published demos of cross-task generalization. π0.5 (Physical Intelligence, 2025) — better generalization, mobile manipulation, more challenging environments. Helix (Figure, 2025) — explicit System 1 / System 2 split, the cleanest articulation of the hierarchical pattern. Gemini Robotics and Robotics-ER (DeepMind, 2025) — embodied reasoning, long-horizon, large-scale. GR00T N1 (NVIDIA, 2024–2025) — humanoid foundation model with sim-to-real emphasis.

The architectural debate has converged on three choices that are now broadly settled.

Continuous action heads beat discrete action tokens. RT-2 tokenized actions into a vocabulary and predicted them autoregressively. This is fine for coarse manipulation; it loses to Diffusion Policy (Chi et al., 2023) and to flow-matching heads (π0) on fine motor tasks. The reason is that action distributions are multimodal and continuous, and softmax over bins quantizes away the dexterity. The field has moved on.

System 1 / System 2 separation is forced by latency. Helix and Gemini Robotics-ER both factor explicitly: a slow VLM (~5–10 Hz) for semantic reasoning and task decomposition, a fast transformer or diffusion policy (~200 Hz) for low-level control. Monolithic 7B VLAs cannot run at control rates on the kind of compute a robot can carry. The hierarchy is not just architectural elegance — it is a consequence of the speed of light and the price of GPUs.

Cotraining beats frozen backbones. RT-2 established that joint training with web vision-language data preserves semantic generalization in a way that adapter-based fine-tuning does not. Frozen backbones are cheaper but lose the long tail of object and scene knowledge. Every serious VLA since has cotrained.

The actual bottleneck is data, not architecture. Open X-Embodiment is a rounding error next to LAION-class image datasets. There is no embodied analogue to the web — and there cannot be, because robot data has to be collected by robots, in physical environments, by people. This is why Physical Intelligence, Figure, 1X, Tesla, Boston Dynamics, and Google Robotics are all simultaneously building teleoperation farms and deploying early fleets. The “GPT-3 moment” for robotics is gated on a dataset that does not exist yet, and may need to be collected by deployed fleets — a chicken-and-egg problem that the well-funded entrants are betting on solving in production.

The honest read on what these models can do: short-horizon manipulation, single-room, modest novelty, extensive variation within the training distribution. What they cannot do: arbitrary novel kitchens, contact-rich tool use under variation, long-horizon recovery from off-distribution states, and generalization across embodiments. The “fold laundry” demo class is real. “Load any dishwasher in any kitchen” is not, and the gap between the two is mostly data.

It is worth being explicit about what “mostly data” hides, because the long tail in robotics is unforgiving in a way that the long tail in 2D classification is not. A perception model that misclassifies 0.1% of images is a state-of-the-art system. A robot policy that fails on 0.1% of grasps drops one cup per thousand — and if those failures cluster on wet floors, dark cabinets, transparent objects, soft fruit, kids holding the cup at the wrong end, or the cat sitting on the dishwasher door, the policy is unshippable regardless of average-case performance. The autonomous-vehicle decade demonstrated this empirically: each subsequent 9 of reliability cost an order of magnitude more data and engineering than the last, and the failure modes were always concentrated in messy real-world distributions that the lab benchmarks did not span. There is no reason to expect humanoid robotics to be more forgiving on this axis. If anything, the contact-rich, multi-object, multi-agent nature of household tasks makes the long tail thicker.

5. The 3D foundation-model frontier

Three tracks are moving at different speeds.

Feed-forward reconstruction is the breakout area. DUSt3R (Wang et al., 2024) replaced per-scene NeRF/Gaussian-splatting optimization with a single forward pass producing dense geometry from two unposed images. MASt3R (2024) extended it to matching and metric reconstruction. VGGT (Visual Geometry Grounded Transformer, 2025) generalized to N views, jointly producing camera poses, depth, and tracks. Fast3R and Spann3R extend further. This is, in my view, the closest analogue to ImageNet’s effect on 2D — a pretrained backbone that gives you camera and depth and correspondence essentially for free, in a way that wasn’t possible eighteen months ago.

Generative 3D is mature for assets, immature for scenes. TRELLIS, Hunyuan3D 2, Rodin, and the multi-view diffusion line (Zero-1-to-3, MVDream) produce decent meshes and Gaussians from images or text. Useful for content pipelines and for populating simulators. Not useful for understanding the natural world a robot is operating in.

3D-aware semantics is fragmented. The agenda of lifting 2D foundation features (CLIP, DINO, SAM) into 3D — 3D-LLM, ConceptGraphs, OpenScene, SAM-3D variants — is functional but has not produced a clean unifying foundation model. We have many good components and no DINO-of-3D-semantics yet.

The actual open problems are well-defined but unsolved:

The honest read: 3D in 2026 is roughly where 2D was in 2018. The components are strong, the unifying foundation model is not yet here, and the data flywheel is not yet spinning at internet scale. VGGT is the most plausible candidate for the “DINO of 3D” but it is months old and not yet battle-tested across the long tail of natural scenes.

6. Embodied evaluation — the missing infrastructure

A theme runs through all of the above: we do not have benchmarks that reliably distinguish a real capability from a convincing demo. This is the most under-discussed bottleneck in the field.

For perception, the situation is actually decent. BLINK, MMVP, CV-Bench, MME, MMMU, ARC-AGI-2, and the various spatial-reasoning benchmarks have surfaced real failures and driven real progress. The benchmarks lag the models — saturated benchmarks become useless within months of release — but the methodology of probing what models cannot do is mature.

For world models, the methodology is still being invented. WorldModelBench, IntPhys 2, and Physion are early attempts. They show that frontier video models fail at simple physics under counterfactual probes, but they are far from comprehensive. There is no generally accepted procedure for evaluating “is this a world model?” beyond visual inspection and ad-hoc physics tests.

For embodied policies, the situation is worse. Real-world evaluation is expensive, hard to reproduce, and confounded by hardware variation. Simulators (Isaac Sim, MuJoCo MPC, RoboCasa, BEHAVIOR-1K, Habitat) help but inherit sim-to-real gaps. The community runs strong sim benchmarks and weak real benchmarks, and the published numbers from corporate labs are essentially impossible to verify independently. The Open X-Embodiment community is the best counterweight to this, but it is undermanned for the scale of the claims being made.

The result is that the architectural literature is moving faster than the empirical literature can keep up with. We have ten papers per month proposing new VLA recipes, new world-model objectives, new 3D backbones. We have very few papers carefully measuring what any of them actually do under controlled conditions. This is the single highest-leverage place for the academic community to push, because it is the bottleneck the corporate labs cannot easily solve from inside.

7. The honest summary

The through-line across perception, planning, world models, VLAs, and 3D is the same: representations are improving faster than the data and evaluation infrastructure can validate them.

Concretely:

Moravec’s paradox is not a curiosity. It is the daily lived experience of the field. Frontier models can do graduate-level mathematics and write production-quality code, and they cannot reliably load a dishwasher. The reason is not that we lack ideas. It is that we do not yet have an internet-scale dataset of physical interaction, and we do not yet have a world-model objective that produces representations rich enough to plan over realistic distributions.

If I had to bet on what closes the gap, it would be three things, roughly in this order:

  1. Deployed robot fleets generating their own training data, the way self-driving fleets did in the late 2010s. Physical Intelligence, Figure, 1X, Tesla, and the humanoid push are all positioned for this.
  2. Action-conditioned latent world models with serious counterfactual evaluation. Not Sora-as-world-model. Something more like V-JEPA 2 with a richer latent space and a proper evaluation harness.
  3. 3D foundation models for action, in the spirit of VGGT but trained on robot interaction data, with representations chosen for policy learning rather than reconstruction.

None of these are guaranteed. It is worth saying out loud what could go wrong with each. Robot fleet data may be slower, more expensive, and lower-variance than the autonomous-vehicle precedent suggests. Cars are constrained to roads with structured rules; humanoids will operate in homes and warehouses with no such structure, and the per-episode cost of a useful trajectory is far higher than a mile of highway. The flywheel may take fifteen years, not five, and may require regulatory and liability scaffolding that does not yet exist. Latent world models may not scale. The JEPA program is intellectually clean but empirically thin — V-JEPA 2 is a strong existence proof, not a scaling-law demonstration, and there is a real possibility that latent prediction loses the long tail of detail that turns out to matter for fine-grained control. 3D foundation models for action may need representations we have not yet invented. The 2D foundation-model story worked because pixels were already a usable substrate; the embodied analogue may require a representation co-designed with the policy, in which case it is less a scaling problem and more a research problem, and the timeline stretches accordingly.

The field has been here before — the 2010s vision community had the same problem with ImageNet before AlexNet, and the early-2020s language community had the same problem with text data before the scaling laws came in. What we are missing in embodied AI right now is the equivalent of those two unlocks. They will probably arrive. The interesting question is not whether, but who builds them and on what timeline, and whether the resulting capability lands in a regulatory and labor environment that can absorb it without breaking things downstream — which is a separate essay, and one I have already written.

8. Ten claims the field should debate in 2026

A non-exhaustive list of claims I think are roughly right and would benefit from sharper public argument. I do not expect agreement on any of them; I do think the field would be better served by debating them explicitly than by sliding past them.

  1. Generative video models are not world models, and the field should stop calling them that. Counterfactual fidelity under action conditioning is the test. Most of what is called a “world model” today fails it.
  2. The “GPT-3 moment” for robotics is gated on a dataset that may take a decade of fleet deployment to collect. Architectural debates above this layer are second-order.
  3. Hierarchical reinforcement learning has been a twenty-five-year research program with no general method to show for it. This is evidence that the abstraction may be wrong, not that we have not tried hard enough.
  4. Reasoning-model progress on math and code does not transfer to embodied planning. Treating it as evidence for general-intelligence progress is a category error.
  5. 3D foundation models will close the gap to 2D faster than expected, on the order of 18–36 months, because feed-forward methods like DUSt3R and VGGT remove the data-quantity bottleneck that was thought to be terminal.
  6. Sim-to-real transfer will be solved before cross-embodiment transfer. The latter is harder than it looks because the inductive biases of a humanoid and a parallel-jaw gripper are not interchangeable in the way the literature implicitly assumes.
  7. The bottleneck on humanoid robotics is not hardware. It is policy data and evaluation infrastructure. The labs that win will be the ones with the largest deployed fleets, not the ones with the best mechanics.
  8. Counterfactual evaluation should become a first-class methodology for world models, on par with FID for image generation. Until that lands, the visual-quality leaderboard is misleading the field.
  9. The “right” representation for action-relevant 3D is unlikely to be Gaussian splats, neural radiance fields, or meshes. It will be something policy-learned that we do not yet have a clean name for.
  10. RLHF was a one-time wedge that worked because text is the substrate of human preference. The embodied analogue will require fundamentally different infrastructure — likely some combination of teleoperation at scale, learned reward models grounded in physics, and counterfactual evaluation harnesses that do not yet exist.

If any of these turn out to be wrong, I would like to know which, and why. That is the productive disagreement. The unproductive one is the one the field is mostly having: arguing past each other about whether a given demo “works” without specifying what working means under what distribution.

References and further reading

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
The Edge of Artificial Intelligence Research: A Citation-Grounded Survey from ICML, ICLR, NeurIPS (2023–2026)
Next Post
From Luddites to Humanoids: What the Steam-Engine Era Tells Us About AI Robotics Replacing Human Labor