Temporal Costmap Aggregation for Object-Relative Navigation

Problem

ObjectReact predicts waypoints from the current WayObject Costmap. When a single frame is corrupted by segmentation or matching errors, the feed-forward controller has no memory to recover from the bad input.

K=6Recent costmap embeddings in each temporal window.

4ObjectReact tasks: Imitate, Alt-Goal, Shortcut, Reverse.

5Aggregators: EMA, GRU, cosine gate, reliability gate, soft fusion.

3Corruption geometries: full-frame, object-level, localized patch.

System Design

The project leaves ObjectReact's map, perception pipeline, expert generator, encoder, and waypoint head intact. Only the embedding passed from the costmap encoder to the waypoint head is replaced by a temporal aggregate.

Temporal Aggregation

Each WayObject Costmap is encoded as a 1024-dimensional embedding. Instead of sending only the current embedding to the controller, the system aggregates a six-frame window and predicts the next waypoint rollout from that temporally smoothed representation.

Encode
Convert each costmap frame into an ObjectReact embedding.

Aggregate
Fuse recent embeddings with EMA, GRU, gated GRU, or soft fusion.

Act
Use the original waypoint head to predict a 10-step local trajectory.

Why Gates?

A plain recurrent model can average useful history, but may also integrate corrupted observations. Gated variants test whether a reliability signal can discount anomalous frames before they disturb the controller's hidden state.

A central finding is negative but useful: a scalar reliability gate can improve behavior without becoming an interpretable failure detector, and explicit detector supervision can damage navigation.

Results

The evaluation separates recovery from fully invalid frames, mechanism interpretability, and the paper-aligned object-level corruption that ObjectReact was trained to tolerate.

60.2Avg SPL for reliability-gated GRU under full-frame zero-out after corruption training.

0.312ROC-AUC of 1-alpha as a corruption detector: not interpretable reliability.

12/12Cosine-gated GRU task-rate wins over the per-step baseline under segment outliers.

67.70Avg SPL for reliability-aware soft fusion under localized patch dropout.

Main Object-Level Result

Method	Imitate SR	Alt SR	Shortcut SR	Reverse SR
Baseline	21.21	13.04	0.00	10.00
Cosine-Gated GRU	42.42	21.74	19.23	26.67
Reliability-Gated GRU	36.36	13.04	15.38	20.00

Interpretation

Under paper-aligned per-segment outlier dropout, the cosine-gated GRU is the strongest recurrent aggregator. Under localized patch failures, EMA and reliability-aware soft fusion are stronger.

The robust conclusion is not that one learned reliability detector solves perception failure. The evidence supports temporal aggregation as a useful ObjectReact extension, with the aggregation mechanism matched to corruption geometry.

Qualitative Rollouts

Matched videos compare Plain GRU and Reliability-Gated GRU under the same full-frame corruption schedule. Left is Plain GRU; right is Reliability-Gated GRU. Red borders mark injected noise.