GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments
Training embodied agents in the real world requires skilled operators and expensive hardware.
GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments
The High Stakes of Sim-to-Real
Training “Embodied AI”—autonomous agents capable of navigating and manipulating the physical world—is paralyzed by the “demonstration tax.” In the real world, training a robot is prohibitively expensive, requiring thousands of hours of manual demonstration by skilled operators on fragile, high-end hardware. To scale robot learning, the industry must move to the “gym”: high-fidelity simulations.
Simulation enables massive data generation and “domain randomization,” but it has long been haunted by the sim-to-real gap. If a simulation doesn’t perfectly mirror the visual and physical properties of reality, a policy trained in the gym will crumble when deployed on a real factory floor or in a home. GASE (Gaussian Splatting–Based Automated System) obliterates this gap, providing an automated pipeline to transform real-world scans into high-fidelity, simulation-ready environments with unprecedented accuracy.
The Bottlenecks: Why Current Pipelines Struggle
Despite the potential of 3D reconstruction, two critical bottlenecks have historically prevented automated simulation construction from being “production-ready”:
- Inefficiency in Data Acquisition: Most existing workflows rely on monocular (single-camera) video scanning. Digitizing an expansive indoor environment this way is a labor-intensive slog that fails to scale.
- Ineffectiveness in Object-Scene Decomposition: Traditional methods fail to cleanly separate foreground objects from the background.
- The “Structural Hole” Problem: Methods like Gaussian Grouping or LangSplat attempt to segment objects directly in 3D space. Because individual Gaussian ellipsoids often straddle object boundaries, direct separation leads to fuzzy edges and “structural holes” in both the object and the background.
- Poor Inpainting: These structural artifacts make it impossible to perform high-fidelity inpainting, resulting in unrealistic simulation assets.
Introducing GASE: The Automated Solution
GASE introduces a radical, highly automated workflow that leverages panoramic camera arrays to scan environments in a single pass. By capturing multiple video streams simultaneously, GASE digitizes complex scenes in a fraction of the time required by monocular methods.
The GASE Workflow
As visualized in the system pipeline, GASE follows a rigorous five-step process:
- Input: The system ingests multi-view frame streams, precise camera poses, and point clouds from the initial panoramic scan.
- Mask Extraction: Utilizing a 2D-domain strategy, the system identifies target objects with surgical precision.
- Decomposition: Foreground objects are decoupled from the static background.
- Reconstruction: GASE uses a hybrid approach: 3D Gaussian Splatting (3DGS) provides superior visual rendering for the background, while TRELLIS generates high-quality physical collision meshes for foreground objects—a non-negotiable requirement for physical interaction.
- Simulation Integration: These decoupled assets are exported directly into physics simulators like NVIDIA Isaac Sim 5.1 for immediate policy training.
The Secret Sauce: Why 2D Extraction Wins
The core innovation of GASE is its “2D-first” strategy. While competitors struggle with 3D ellipsoids that blur boundaries, GASE performs mask extraction and background inpainting in the 2D image domain before 3D reconstruction begins.
Spatial Awareness and Localization
To ensure an object’s identity is maintained across intermittent camera views, GASE employs a sophisticated “Localization Strategy” (driven by Equation 1 and 2 logic). By projecting points using camera extrinsics and applying a depth filtering strategy, GASE handles inter-object occlusion with ease. If an object is hidden behind another, the system recognizes it, preventing incorrect mask assignments and giving the pipeline true “spatial awareness.”
GASE leverages the cutting edge of vision models: SAM3 for initial semantic querying and mask extraction, and SAM2 specifically for robust video propagation and tracking across frames. Combined with LaMa for high-fidelity inpainting, GASE produces a clean, “empty” background and perfectly isolated foreground assets, completely bypassing the structural artifacts of 3D-centric methods.
Benchmark Performance: Crushing the Competition
GASE’s 2D-domain strategy delivers a masterclass in segmentation accuracy. In evaluations on the LERF dataset, GASE crushed existing 3D Gaussian methods, showing a >10% improvement in both mIoU and mBIoU (Boundary IoU)—the metric that proves GASE has finally solved the “fuzzy boundary” problem.
| Method | Figurines (mIoU/mBIoU) | Ramen (mIoU/mBIoU) | Teatime (mIoU/mBIoU) |
|---|---|---|---|
| LangSplat | 54.70% / 50.20% | 86.32% / 75.79% | 64.06% / 55.79% |
| Gaussian Grouping | 77.76% / 73.86% | 61.50% / 58.71% | 89.43% / 86.30% |
| ObjectGS | 78.41% / 77.73% | 84.89% / 82.35% | 68.94% / 66.43% |
| GASE (Ours) | 93.32% / 89.59% | 95.63% / 94.34% | 95.91% / 92.61% |
Beyond segmentation, GASE delivers state-of-the-art visual consistency. In “sundries” scene benchmarks, GASE achieved a PSNR of 27.09 and an SSIM of 0.94, with FID/m-FID scores of 166.14 and 313.80. These numbers prove the inpainted simulation environments are visually indistinguishable from reality.
Real-Robot Success: Putting Sim-to-Real to the Test
The ultimate proof of GASE is the Real-to-Sim-to-Real (R2S2R) paradigm. We deployed policies trained entirely in GASE-generated simulations onto physical hardware.
Manipulation Mastery
Testing on the Piper robotic arm demonstrated that GASE-trained policies handle real-world materials with elite precision:
- Grasping Cola Cans: 73% success rate.
- Bottling Objects: 60% success rate.
- Pushing Cups: 67% success rate. Critically, the performance gap between simulation-trained and real-world-trained policies remained under 10%.
Navigation Success
We deployed GASE to a Kitt 15 robot in a “bar” environment. The impact of simulation training was transformative:
- Find Microwave: 100% success (Sim-to-Real).
- Find Red Fire Extinguisher: 100% success (Sim-to-Real).
- Straight to wall, then turn left: Jumped from 50% (Sim-to-Sim) to 100% (Sim-to-Real).
Key Takeaways for the Robotics Industry
- Scalability: Panoramic camera arrays and automated pipelines allow teams to slash demonstration costs by thousands of hours.
- Fidelity: Processing masks in the 2D domain eliminates the “structural holes” that previously made automated sim-assets unusable for manipulation.
- Utility: Simulation-trained policies are now deployment-ready, maintaining a performance gap of less than 10% compared to expensive real-world data.
What’s Next? While GASE is a massive leap forward, purely vision-based point cloud generation remains computationally heavy. Future research into feed-forward 3DGS reconstruction could potentially bring these high-fidelity environments to real-time speeds.
Closing
GASE provides an efficient and highly effective solution for the future of robot learning, providing the high-fidelity environments necessary to bridge the final gap between artificial intelligence and physical action.
“These results confirm that GASE provides an efficient and highly effective solution for bridging the sim-to-real gap.”
- Project Page: https://jar7visz.github.io/gase-page/
- Code: Stay tuned for the official GitHub release!
Read the full paper on arXiv · PDF