GUI-Primitives: Spatial Reasoning Failures in Vision Models

Computer-use agents, AI systems that interact with graphical interfaces by reading screenshots and issuing mouse/keyboard commands, are gaining capability but failing at a fundamental task: reliably locating UI elements. A new benchmark published on arXiv this week exposes why.

GUI-Primitives, a 994-item evaluation dataset, isolates a critical gap in vision-language models: they struggle to bind spatial language to the correct UI element even when the screenshot is fixed and only the instruction changes. Nineteen tested models hit a maximum of 32% strict point-in-box accuracy on contrastive spatial reasoning tasks. That’s not a rounding error; it’s a fundamental failure mode with security and reliability implications.

What the Benchmark Tests

The researchers, Md Abrar Jahin and Md Rizwan Parvez (accepted to EMNLP 2026), framed the problem clearly: existing benchmarks like ScreenSpot or Ultrafine don’t isolate whether a model truly understands spatial relations or just pattern-matches on visual features. GUI-Primitives fixes this by presenting contrastive pairs: two screenshots, one anchor element, and relational instructions that cause the correct target to shift between two labeled candidates.

The benchmark covers seven spatial relations common to interface design:

• Horizontal: left/right positioning
• Vertical: above/below positioning
• Containment: elements nested or grouped
• Alignment: shared edges or centers
• Proximity: nearby elements
• List ordinal: Nth item in a list
• Occlusion: elements partially or fully hidden behind others

A 196-item subset was validated by five annotators (κ=0.94 for well-formedness, κ=0.79 for target selection). This is solid inter-rater agreement, which rules out dataset ambiguity as an excuse for model failure.

The Findings: Major Blind Spots

Performance varied sharply by relation type. When the model’s predicted coordinate falls within one of the two candidate regions (classified predictions), accuracy reaches 0.82–0.90 for horizontal/vertical and proximity tasks. But for containment and occlusion, models perform at chance (0.50). That’s the critical insight: models don’t reliably locate spatial targets; they struggle to understand the spatial concept itself.

Worse, on strict point-in-box accuracy, 19 models maxed out at 32%. When asked to click “the button to the right of the text field,” the models emit unconstrained coordinates that fall outside both candidates 60–92% of the time. That’s not precision failure; it’s reasoning failure.

The authors tested a diagnostic intervention: marking the two candidate regions visually in the screenshot. Performance jumped 35–57 percentage points. Translation: the bottleneck is not relation comprehension; it’s candidate localization. Models can understand “to the right of” but cannot reliably narrow the search space on their own.

Why This Matters for Automated Agents

Computer-use agents rely on vision-language models to parse screenshots and emit coordinates. If the model cannot accurately locate the target element, the agent either clicks the wrong button, submits a form field to the wrong text box, or fails outright. For a system orchestrating routine IT tasks, including filling out expense reports, configuring cloud infrastructure, and navigating configuration menus, precision at the element level is not optional.

Consider an agent tasked with changing a security policy setting buried in a nested menu. If it misidentifies “the dropdown below the label” or fails to see an element hidden behind a modal, the agent either hangs, takes an incorrect action, or falls back to a human operator. In a compliance or security context, incorrect actions can have teeth.

OWASP LLM08 (Excessive Agency) flags this precise risk: “Granting LLMs unchecked autonomy to take action can lead to unintended consequences.” When the underlying perception mechanism is unreliable, autonomy without oversight becomes a liability. The question is not whether models can perform GUI tasks; they can. It is whether they do so with predictable precision.

The Larger Ecosystem of GUI Benchmarks

GUI-Primitives is not alone. The past year has seen a surge in fine-grained GUI evaluation:

• DiffSpot (May 2026) tests detection of subtle visual changes in web interfaces
• FineState-Bench (April 2026) evaluates state-conditioned grounding across desktop, web, and mobile
• AutoGUI-v2 (April 2026) focuses on GUI comprehension and interaction outcome prediction
• GUI-360° (November 2025) is a large-scale desktop automation suite

Each surfaces different failure modes. Together, they sketch a picture of what reliable agent deployment requires: not just a capable model, but one that degrades gracefully, provides uncertainty signals, and operates within defined guardrails.

What Defenders Should Take Away

Organizations deploying computer-use agents should treat spatial reasoning failures as a first-order risk, not as a known limitation that QA will shake out:

1. Test on deployment UIs. General benchmarks do not predict performance on proprietary or internal interfaces. Teams should run GUI-Primitives-style contrastive tests on the tools their agents will actually use.

2. Mandate uncertainty signals. The model should report confidence, not just coordinates. Low confidence on a high-stakes action (changing permissions, modifying infrastructure) should require human review.

3. Layer detection guards. Image processing should validate that the clicked element matches semantic expectations. If the model clicked a “Cancel” button when aiming for “Confirm,” the action should be blocked before it completes.

4. Monitor for silent failures. Agents that emit coordinates outside the UI bounds or that click in empty space are not failures; they are diagnostic signals. Teams should log them, alert on them, and retrain on them.

5. Scope autonomy tightly. Teams should constrain agents to read-only tasks or low-risk actions until spatial reasoning is demonstrably reliable on their own interfaces.