Navimaster:The First Unified Navigation Model Across Digital and Physical Worlds, with Minecraft Support

Table of Contents

Abstract

Navigation now happens in two worlds at once: digital interfaces such as mobile, desktop, and web GUIs, and embodied environments that require movement, localization, and interaction in physical or simulated space. Although both are fundamentally navigation problems, they have long been studied through separate datasets, separate action spaces, and separate training pipelines.

That separation creates several practical bottlenecks:

separate models increase system cost
cross-domain generalization remains weak
sparse RL rewards slow training down
models often reason correctly but execute poorly

NaviMaster tackles this fragmentation by introducing a unified navigation agent that learns GUI navigation and embodied navigation under the same framework. Through a shared trajectory formulation, a unified RL pipeline, and distance-aware dense rewards, it substantially improves cross-task generalization, optimization efficiency, and grounding precision.

Task Demos

Spatial Grounding

The model predicts a target point directly from visual context and language constraints.

The agent interacts with interfaces through multi-step actions such as click, type, and wait.

Mixed GUI and Embodied Interaction

NaviMaster can also combine navigation and action execution in more complex settings such as Minecraft.

Existing systems usually treat GUI navigation and embodied navigation as different task families. GUI agents operate through clicks and scrolling, while embodied agents rely on movement, viewpoint shifts, and spatial control. Even if both can be abstracted as Markov decision processes, their data formats and training pipelines remain hard to merge.

NaviMaster starts from a different premise: once actions, goals, and trajectories are aligned, both domains can be learned together as one navigation problem. This turns GUI and embodied data into complementary supervision instead of isolated silos.

Three Core Innovations

1. A Unified Visual-Goal Trajectory Formulation

The first challenge is that GUI trajectories and embodied trajectories do not share a common language. NaviMaster resolves this by converting both into a unified visual-goal trajectory format.

The action space is systematically aligned:

task-specific actions are preserved and inserted into a shared action vocabulary
GUI [SCROLL] and embodied [TURN] actions are both discretized into directional changes
GUI [CLICK(x, y)] and embodied movement actions are unified through explicit target-point actions such as [MOVETO(x, y)]

GUI trajectories are built from existing datasets such as GUI-Odyssey, while embodied trajectories are derived from shortest-path keypoints extracted with A* search and then converted into visual-goal action sequences. NaviMaster also adds reasoning intents for each step, generated with GPT-4o, so history contains not only actions but also why those actions were taken.

Unified action space and trajectory construction

2. A Unified Reinforcement Learning Framework

Once trajectories are aligned, NaviMaster trains directly with GRPO on mixed GUI and embodied data instead of using separate warm-start pipelines. Both domains are treated as the same decision process: given an observation, an instruction, and execution history, the model selects the next action from a language-defined action space.

This allows one policy to learn from 2D GUI data and 3D embodied data together, strengthening cross-domain generalization rather than overfitting to a single environment family.

3. Distance-Aware Dense Rewards

To address sparse supervision, NaviMaster decomposes success into three components:

whether the output is executable
whether the action type is correct
whether the predicted target is sufficiently close to the ground truth

Instead of binary success-or-failure signals, the model receives graded feedback based on how close it is to the target. This makes learning more stable, reduces useless exploration, and improves convergence speed.

Experimental Highlights

NaviMaster shows strong out-of-domain generalization on GUI tasks. Evaluation is performed entirely on OOD test sets isolated from the training distribution. Against strong baselines, NaviMaster consistently improves success rate across mobile, web, and desktop benchmarks.

More importantly, mixed GUI-plus-embodied training outperforms training on only one domain, showing that the two data sources provide complementary navigation signals.