February 12, 2026
Evaluating Autonomy Under Uncertainty
An early evaluation framework for autonomous planning systems operating with incomplete and delayed observations.
Autonomous systems are often evaluated in environments where observations arrive on time, state estimates are well behaved, and the objective remains fixed. Those conditions are useful for development, but they do not represent many remote or space-based operations.
We have begun evaluating planning agents in simulated scenarios that introduce delayed telemetry, intermittent sensors, uncertain resource estimates, and goals that change during execution. The present objective is not to identify a single highest-scoring model. It is to understand which evaluation methods reveal unsafe confidence before deployment.
Measuring recovery, not only completion
Task completion remains an important metric, but it says little about how a system behaves after its assumptions become wrong. Our test harness therefore records constraint violations, time to recognize a state mismatch, quality of fallback behavior, and the amount of intervention required to recover.
Initial experiments suggest that aggregate success rates conceal meaningful differences between agents. Two systems can complete the same proportion of scenarios while exhibiting very different failure patterns. One may stop conservatively when observations disagree; another may continue acting on an outdated state estimate.
Next phase
The next evaluation set will introduce simulated component degradation and stricter compute budgets. We are also adding replay tools that allow researchers to inspect the information available to the agent at each decision point.
These results are preliminary and based entirely on simulation. We will publish a more complete description of the benchmark after the scenario definitions and scoring methodology have stabilized.