Synthetic-to-Real Translation for Class-Agnostic Motion Prediction

1. Introduction: The High Cost of Real-World Data

Motion prediction is a non-negotiable pillar of autonomous driving safety. By forecasting the future displacement of environmental point clouds, perception systems provide the critical temporal context required for obstacle avoidance and path planning. However, the development of these systems is currently restricted by a severe “annotation bottleneck.” Obtaining cell-wise motion labels from sparse, non-uniform real-world LiDAR data is prohibitively expensive and technically difficult to execute at scale.

Synthetic-to-Real Motion Prediction (SRMP) offers a theoretical escape from this bottleneck by utilizing high-fidelity simulators to generate infinite 4D LiDAR sequences with perfect ground-truth labels. Yet, the transition from simulation to reality remains fraught with risk. The authors of the SR-Motion framework demonstrate that models trained exclusively on synthetic data face catastrophic performance degradation when deployed in real-world settings, exhibiting a 150.5% increase in prediction error due to discrepancies in sensor noise, occlusion patterns, and motion distributions.

2. The Failure Modes of Naive Knowledge Transfer

The transition to real-world data exposes systematic vulnerabilities in standard cell-wise regression models. Even when employing traditional teacher-student frameworks—where a synthetic-trained teacher generates pseudo-labels for unlabeled real data—performance continues to erode. The authors report a 202.7% error increase in these naive transfer scenarios, identifying two primary failure modes that destabilize perception:

• Motion Jitter: The emergence of erratic, non-smooth motion vectors at the individual cell level, where neighboring points fail to maintain a coherent trajectory.
• Object-Level Inconsistency: A breakdown in structural integrity where cells belonging to the same physical instance (e.g., a car) produce divergent motion patterns, effectively “tearing” the object apart in the model’s internal representation.

These failures highlight the context-agnostic nature of standard regression, which lacks the structural priors necessary to survive significant domain shifts.

3. SR-Motion: A Framework for Robust Knowledge Translation

To mitigate these systematic failures, the paper introduces SR-Motion, a framework that utilizes “objectness” as a structural prior to stabilize motion prediction.

The Spatio-Temporal Pyramid Network (STPN) Backbone

The model processes 4D LiDAR data by aligning past frames to the current local coordinate system via ego-motion compensation. These points are voxelized into a Bird’s-Eye View (BEV) pseudo-image, where the height dimension ($D$) is encoded into the feature channels. The underlying architecture is the Spatio-Temporal Pyramid Network (STPN), which utilizes 3D convolutions to extract temporal dynamics and 2D convolutions for spatial features, fusing multi-scale context before passing it to the regression heads.

The Teacher-Student Pipeline

SR-Motion implements a four-step iterative learning process:

1. Teacher Initialization: Training on labeled synthetic data (Motion4D).
2. Pseudo Label Generation: The teacher generates motion and objectness labels for real data.
3. Student Learning: The student model learns from a combination of synthetic data and augmented real-world data (applying consistent rotations and flips).
4. Teacher Update: The teacher model is updated via Exponential Moving Average (EMA) using a smoothing factor of $\alpha = 0.999$. This ensures the teacher progressively adapts to the real-world distribution while filtering out transient noise from the student.

Objectness-Aware Motion Prediction (OAMNet)

The OAMNet architecture adds an objectness-aware branch parallel to the motion branch. It is trained to predict “centroid offsets”—the relative distance from a cell to the center of its parent object. This multi-task learning approach forces the network to learn instance-level geometry, enabling the motion branch to maintain object-level consistency even under domain stress.

4. Cleaning the Signal: Objectness-Aided Motion Enhancement (OAME)

The OAME module serves as the framework’s explicit filter, refining noisy pseudo-labels through a “dual-path consistency mechanism.” By demanding a geometric consensus between the objectness branch (centroids) and the motion branch (vectors), the system suppresses false positives.

• Centroid-aware clustering: Cells are grouped into rigid clusters based on their shared predicted centroids.
• Cluster Outlier Filtering (COF): This two-stage process projects cells into a future frame using predicted motion and then applies centroid clustering. If a cluster contains only a single cell ($T_N = 1$), it is discarded as an unreliable jitter.
• Spatial Consistency Smoothing (SCS): The module ensures motion uniformity within a cluster by averaging vectors. To prevent the smoothing of unreliable data, it uses the Coefficient of Variation (CV). Clusters exceeding a threshold of $T_{CV} = 10$ are rejected as high-divergence noise.

5. Motion4D: A New Standard for Synthetic LiDAR Datasets

Addressing the lack of 4D LiDAR benchmarks for SRMP, the authors developed Motion4D, a physically-grounded synthetic dataset.

The Synthesis Pipeline

Using the BLAINDER LiDAR tracing engine, the authors placed dynamic CAD assets from CARLA and Render People into static real-world backgrounds from the Waymo Open Dataset. Realistic traffic trajectories were sourced from the Waymo Motion Dataset, and multi-iteration ray tracing was used to simulate realistic light paths and occlusions.

Data Characteristics

Motion4D provides 1,370 sequences and 124K frames, featuring a significantly higher label density than existing benchmarks:

Dataset	Synthetic	Traffic Scenes	4D Sequences	Motion Labels (1/sample)
Waymo	No	Yes	Yes	1243.3
nuScenes	No	Yes	Yes	432.2
FT3D	Yes	No	No	-
GTA-SF	Yes	Yes	No	-
Motion4D	Yes	Yes	Yes	1779.6

6. Results: Closing the Performance Gap

Quantitative evaluations on Waymo and nuScenes datasets confirm that SR-Motion effectively bridges the domain gap.

• Gap Closure: For the static group on Waymo, SR-Motion closed the performance gap between synthetic-only and Oracle (fully supervised) models by 76.3%.
• Comparative Superiority: The framework significantly outperformed self-supervised baselines like PillarMotion and SelfMotion, narrowing the static performance gap by up to 86.4%.
• Mixed Training Advantage: A high-impact finding for practitioners is that training on a mix of real (Waymo) and synthetic (Motion4D) data outperformed Waymo-only training by 9.4%.
• Real-to-Real Generalization: Motion4D training proved highly robust; the Motion4D $\rightarrow$ Waymo adaptation (1.545 mean error for fast agents) significantly outperformed nuScenes $\rightarrow$ Waymo adaptation (2.327 mean error).
• Deployment Feasibility: The architecture is optimized for real-time safety applications, running at 20 ms per frame on an RTX 4090.

7. Conclusion: The Future of Robust Autonomous Systems

The SR-Motion framework demonstrates that the sim-to-real gap in 4D perception is solvable through the integration of objectness-aware priors and geometric consensus mechanisms. By utilizing Motion4D’s high-fidelity synthetic data, models can acquire the foundational knowledge necessary to adapt to real-world environments without exhaustive manual labeling.

The authors acknowledge a current limitation in the “rigid-object assumption.” Because the model operates on BEV maps, it loses some fine-grained geometry required for modeling non-rigid or deformable agents, such as pedestrians. Despite this, the research marks a significant advance in AI safety, providing a method to mitigate the systematic motion jitter and structural inconsistencies that have historically plagued simulation-trained perception systems during real-world deployment.

Read the full paper on arXiv · PDF