GraspIT: A Dataset Bridging the Sim-to-Real gap and back for Validated Grasping SE(3) Pose Generation

1. Introduction: The Hidden Failure Modes of Robotic Grasping

In robotic manipulation, the “sim-to-real gap” remains a fundamental obstacle to reliable deployment. Conventional grasping pipelines often rely on analytic force-closure—a geometric metric that predicts stability based on contact points. However, Koch et al. demonstrate that theoretical stability is a poor proxy for physical success. Their research reveals a critical discrepancy: 17% of grasps that pass theoretical force-closure metrics fail when subjected to physical slip-tests. This divergence indicates that current models often suffer from a “physical reasoning deficit,” where a robot may execute a kinematically valid grasp that cannot withstand real-world dynamic perturbations. The GraspIT dataset was engineered to identify these systematic failure modes and provide a grounded foundation for robust manipulation.

2. The GraspIT Design Philosophy: Four Pillars of Physical Grounding

To address the shortcomings of purely analytic datasets, the authors established four design pillars to ensure data is both physically actionable and transferable:

• Quality over Quantity: Rather than prioritizing billions of unvalidated candidates, the authors utilize a staged physical validation process. This produces continuous quality scores ($s$) that account for dynamic failures—such as gravity and oscillation—missing from static metrics.
• Robot-in-the-Loop Reachability: Every grasp is pre-filtered for collision-free trajectory reachability using the cuRobo and Lula planning frameworks. This ensures that every “good” label in the dataset is kinematically actionable by a Franka Panda arm.
• Bidirectional Real↔Sim Link: The framework enables the registration of real-world scene geometry into NVIDIA Isaac Sim. Validated labels are back-projected onto real RGB-D images, maintaining a consistent annotation coordinate frame across synthetic and physical sensors.
• Open Infrastructure: To facilitate community scaling, the authors have released all tools as Docker-containerized modules, allowing for the integration of new objects and robot models.

3. The “Robot School”: A Four-Stage Physical Validation Process

The “Robot School” methodology utilizes parallel Franka Panda instances in Isaac Sim to execute sequential perturbations. This process quantifies grasp reliability beyond binary success.

The Four Stages of the GraspIT Slip-Test

Stage Name	Physical Action	Failure Mode Identified
Trajectory + Close	Movement to contact; 70N gripper closure	Geometric collisions or insufficient contact
Lift	20 cm vertical displacement under gravity	Friction failure ($\mu \approx 1.0$)
Horizontal Oscillation	Linear perturbation in the horizontal plane	Rotational instability
Pendulum Swing	Rotation of joint 5 (wrist)	Sensitivity to off-center mass/moment arms

Technical Parameters and Scoring The authors define failure as positional slip $> 2$ cm ($N_{xyz}$) or orientational slip $> 10^\circ$ ($N_{rpy}$). The continuous quality score $s$ is derived from a base score $b$ (determined by the last successfully completed stage) and a penalty for excessive slip:

$s = b - \frac{N_{xyz>2cm}}{12} - \frac{N_{rpy>10}}{12}$

This formula ensures that “messy” grasps are penalized; for instance, a grasp passing the lift stage ($b=0.50$) but exhibiting significant slip is pushed toward a “bad” label ($s < 0.50$). The simulation utilizes a consistent closure force of 70N and a default material friction coefficient of $\mu \approx 1.0$.

4. Quantifying the Dataset: Statistics and Composition

The GraspIT initial release provides a high-density corpus designed to prevent object-identity overfitting:

• Scene Volume: 1,035 simulated scenes and 100 real-world scenes.
• Frame Count: ~316,000 annotated RGB-D frame sets.
• Object Diversity: ~7,400 unique object identities drawn from the ABC dataset.
• Grasp Candidates: ~2.3 million total candidates evaluated; 82.94% are labeled “good” ($s \geq 0.50$).
• Attrition Metrics: 11.46% of candidates fail at the trajectory-planning stage, highlighting the necessity of reachability filtering.

5. Bridging the Gap: The Real↔Sim Bidirectional Loop

The authors achieve metric-accurate transfer between simulation and reality through a specialized registration pipeline. Physical objects are scanned using the TRELLIS generative image-to-3D model, with absolute scale recovered via multi-view Iterative Closest Point (ICP) fitting.

For scene capture, a UR-5 arm equipped with an Intel RealSense D405 camera executes a hemispherical scan. The authors utilize eye-in-hand calibration to derive the rigid transform $T_{EE}^{cam}$ from forward kinematics. This allows simulation-validated labels to be back-projected onto real sensor data with high precision. By registering real scenes into Isaac Sim, the pipeline generates photorealistic synthetic views that share a perfectly aligned coordinate frame with real-world geometry.

6. Critical Takeaways for AI Safety and Red-Teaming

For researchers focused on AI safety and systematic failure, GraspIT provides a repository of “graded hard negatives.” A primary contribution is the “penalized bucket”—a subset of 94,475 grasps (4.15% of the total) that nominally pass physical stages but fail due to excessive slip. These represent the exact “near-miss” failure modes that evade standard evaluation but cause catastrophic failure in unconstrained environments.

The authors argue that current Vision-Language-Action (VLA) models suffer from a “physical reasoning deficit” because their encoders (e.g., CLIP) prioritize semantic similarity over metric geometry. GraspIT addresses this by providing “generated object properties” missing from semantic models, including:

• Metric 3D surface point clouds derived from CAD.
• Analytic physical properties such as center-of-mass proxies and convex-hull volume fractions.
• Validated reachability data that grounds visual affordance in kinematic reality.

7. Conclusion: A Living Foundation for Manipulation

GraspIT is established as an open-source foundation for the robotics community, moving beyond the limitations of static benchmarks. By releasing Docker-containerized tools, the authors invite researchers to extend the “Robot School” to new environments and robot models.

The project’s long-term significance lies in its transition from analytic metrics to dynamic, physically-validated quality scores. By quantifying the divergence between force-closure and physical reality, GraspIT provides the grounded data necessary to develop the next generation of robust, safety-critical robotic manipulation systems.

Read the full paper on arXiv · PDF