Back to the Future: Predicting Long-Horizon Dynamics with Retrospective Correction for Disturbance-Aware Retrieval

Abstract

Retrieving objects from densely packed storage requires anticipating how contact disturbs surrounding objects when limited clearance prevents lifting the target over clutter. However, contact can propagate beyond the target’s immediate neighbors, while accumulated prediction errors make learned dynamics unreliable over long planning horizons. We introduce DARE (Disturbance-Aware REtrieval), a geometry-aware relational dynamics framework that uses retrospective correction to improve long-horizon prediction for disturbance-aware retrieval. DARE encodes compact object descriptors and predicts action-conditioned disturbance propagation through object-level message passing. When a sparse observation becomes available, a learned corrector uses the prediction discrepancy to amend an earlier predicted state and reruns the intervening dynamics, propagating the correction through the predicted contact chain. These predictions guide a receding-horizon planner that balances retrieval progress against displacement and rotation of surrounding objects, permitting necessary contact while discouraging harmful disturbance. Experiments in planar and RoboCasa environments evaluate long-horizon prediction, generalization, and robustness for disturbance-aware retrieval in dense clutter.

Highlights

Each clip shows the four methods retrieving the same target from one cabinet scene side by side, one column per method, rendered with the arm in the control loop. The result under each method is the paper’s for that scene.

Overview

Disturbance-aware retrieval

Collision avoidance: the target is boxed in by its neighbors, so no contact-free path exists. — **Fig. 1: Disturbance-aware retrieval.** (a) Collision avoidance leaves the target blocked. (b) Direct retrieval topples a neighboring object. (c) Our method predicts contact-induced disturbance to plan retrieval through controlled contact, preserving stability of surrounding objects.

Method

Pipeline Overview

Pipeline: object geometry and the scene state enter an interaction graph, action propagation and prediction produce rollouts of candidate actions, and the planner picks the retrieval path. — **Fig. 2: Pipeline Overview.** Given object geometry and the current scene state, DARE propagates action-conditioned interactions over an object graph to predict scene evolution under candidate target motions. The resulting rollouts capture direct and indirect disturbances to surrounding objects, enabling the planner to select a retrieval trajectory that balances goal progress against collateral disturbance.

A
Problem Statement

A grasped target is retrieved from a densely cluttered compartment where limited overhead clearance rules out lifting, so retrieval requires contact that can displace, topple, or eject bystanders. The robot commands the target’s planar translation, tilt, and yaw, while bystanders move only through contact, whose dynamics are approximated by a learned relational model. The planner seeks motions that make progress toward the opening while limiting bystander displacement, tilt, and yaw.
B
Relational Dynamics Prediction

Each object is one graph node carrying its mass, inertia, and a learned descriptor of its convex collision pieces, so the graph size does not depend on mesh resolution. A propagation network passes messages along proximity edges rebuilt after every predicted step, relaying the target’s push through intermediate objects; compartment faces are static nodes, so object–wall contact uses the same computation. The command enters only on edges sent by the target, so indirect motion must be explained through relational paths.
C
Retrospective Correction from Sparse Observations

Autoregressive rollouts compound small pose errors, and in clutter a small error can shift a contact and change the whole disturbance cascade. Overwriting the prediction with a sparse observation copies sensor noise and leaves velocities incomplete; instead, a corrector reads the blind rollout and the observed pose discrepancy and amends the earliest predicted state. The dynamics are then rerun from the amended state, so the correction propagates through the contact history and a dynamically consistent state is carried into the next cycle.
D
Future-aware Joint Predictor–Corrector Learning

Predictor and corrector are trained jointly in one backward pass through the blind rollout, the amendment, and the rerollout. The corrected rollout is also scored beyond the observation, so an amendment is rewarded only if the future it produces stays accurate. The predictor thus learns rollouts whose errors are correctable from sparse observations, not only accurate on their own.
E
Disturbance-Aware Retrieval Planning

Rather than rejecting all contact, the planner scores the predicted consequences of each candidate motion and excludes candidates that exceed bystander displacement or tilt tolerances. Its cost trades the target’s progress toward the opening against bystanders’ displacement, tilt, and yaw; spline-shaped candidates are refined with MPPI-style weighting, and the lowest-cost admissible plan is executed. Each executed segment ends at the next observation, so every planning cycle is also a correction cycle.

Experiments

Experiments Setup

Five pucks on a flat table with the pusher above them. — Planar puck tableDynamics prediction (E1) and joint training (E2).

Household objects on a tabletop with the pusher beside them. — RoboCasa tabletop scenesDynamics prediction on seen and unseen objects (E1), joint training and correction under noise (E2, Fig. 4).

A kitchen cabinet compartment packed with bottles, the Panda arm reaching in. — RoboCasa kitchen cabinetsDynamics prediction (E1), correction under noise (E2), and every retrieval experiment (E3, Table I).

Dynamics Prediction

Fig. 6: Dynamics prediction. Terminal translation (left) and rotation (right) errors normalized by the no-motion error at each horizon. Rotation is omitted for symmetric pucks. The seen-object worlds are added here beside the paper's three. Choose the world, the error and the methods above; hover for the values, click the legend to hide a method.

Terminal translation and rotation errors at each horizon in centimetres and degrees, on a log axis: the numbers behind Fig. 6 before normalization. Choose the world, the error and the methods above; hover for the values, click the legend to hide a method.

Error at every self-fed step, normalized by the no-motion error at the same step; translation steps whose no-motion error is under 0.01 cm are left out. Choose the world, the error and the methods above; hover for the values, click the legend to hide a method.

Error at every self-fed step in centimetres and degrees, on a log axis. Choose the world, the error and the methods above; hover for the values, click the legend to hide a method.

Retrospective Correction and Joint Training

Two rows of predicted frames: the blind rollout drifts by step 40, while the corrector uses the observation mismatch at step 15 to amend step 11 and reroll, staying on the true states. — **Fig. 3: Retrospective correction for long-horizon prediction.** The corrector combines prior predictions with the observation mismatch at t₁₅ (yellow) to estimate a corrected state t̃₁₁, the first step after the last observation t₁₀. Rerolling from it propagates the correction through the learned dynamics and initializes the next cycle, mitigating error drift.

Three rows of four tabletop frames: predicted outlines drift off the objects without correction, inherit noise with direct observation, and stay aligned with the noise-trained corrector. — **Fig. 4: Tabletop dynamics prediction under noise.** Outlines in purple, light blue and dark blue indicate predicted object poses in a held-out scene as the target (yellow) pushes through clutter. Uncorrected predictions drift (top), direct observation updates inherit noise (middle), and noise-trained correction maintains alignment with the objects (bottom).

Fig. 7: Effect of joint training. Terminal translation (top) and rotation (bottom) errors with shared pretrained initialization and matched additional optimization steps. The dashed line is the pretrained network before any additional step. Choose the world and the error above; hover for the values, click the legend to hide a method.

Fig. 8: Retrospective correction under noise. Translation (top) and rotation (bottom) RMSE over 40-step rollouts versus observation noise σ, fixed within each rollout. Shading indicates noise levels beyond the noise training range. Choose the world and the error above; hover for the values, click the legend to hide a method.

Disturbance-aware Retrieval Planning

1.00success rate of DARE at 20 objects, hard placement, clean observations

0.10success rate of direct retrieval in the same scenes

0.00 → 1.00success rate under noisy observations at 10 objects, easy placement, without and with retrospective correction

Direct retrieval (left) and Ours (right) in the scene of Fig. 5, seen from above and sped up, rendered with the arm in the control loop.

Top-down views of one cabinet scene in three rows: direct retrieval pushes neighbors over, collision avoidance barely moves the target, and our method brings the target out along a curved path with its neighbors standing. — **Fig. 5: Object Retrieval Visualization.** Top-down views compare three planners, with target motion marked in orange. Each row shows four evenly spaced execution steps with translucent next-step poses and the executed target path in the final frame.

Table IRetrieval from cluttered compartments

Supplementary Video

The supplementary video submitted with the paper: the problem, the method, and the experiments, narrated with captions (2 min 57 s).

BibTeX

@article{anonymous2026backtothefuture,
  title  = {Back to the Future: Predicting Long-Horizon Dynamics with Retrospective Correction for Disturbance-Aware Retrieval},
  author = {Anonymous},
  year   = {2026}
}

Abstract

Highlights

Disturbance-aware retrieval

Pipeline Overview

Problem Statement

Relational Dynamics Prediction

Retrospective Correction from Sparse Observations