NVIDIA Omniverse NuRec#
The NVIDIA Omniverse NuRec models and services support the seamless ingestion of real-world camera and lidar data to create a simulated 3D environment. These simulated scenarios can be used to train and test Physical AI Agents, including robotics and autonomous driving systems
How NuRec Works#
Reconstruction and rendering with NuRec follows a series of steps:
Reconstructing scenarios as 3D scenes: The reconstruction models and services for NuRec are available as a core NuRec container from NVIDIA. When you run NuRec, it converts real-world data into a 3D scene and saves the output as a USDZ file on the local host system. The USDZ file includes a Universal Scene Description (USD) file that defines the scene, a checkpoint file that includes the AI-trained reconstruction data, and a JSON file that includes the sequence track and rig trajectories. An OpenDrive (XODR) map might also be included in the USDZ package for use in simulation tasks like traffic modeling and scenario editing, but it is not required to run NuRec.
Rendering the 3D scenes: NuRec offers a gRPC API and is built into Omniverse Kit for seamless integration with simulation platforms. The NuRec gRPC API serves as a conduit of data and rendering between the simulation runtime and the NuRec container, where the scenes are reconstructed and rendered. Users of Omniverse Kit apps like Isaac Sim can load a compatible scene in the app and control rendering through the
OmniNuRecVolumeAPIproperties.Refining the rendered images: The Fixer model refines the rendered images to improve the quality of the rendered images during reconstruction and rendering.
Learn more about How NuRec Works.
Tip
To streamline the pipeline from real-world data to a NuRec-compatible rendered scene, NVIDIA offers a growing collection of pre-reconstructed scenes in the NVIDIA NuRec-trained Physical AI dataset on HuggingFace.
How to Use NuRec#
To use NuRec in an AV simulation platform, you can get the pre-reconstructed scenes from the NVIDIA NuRec-trained Physical AI dataset on HuggingFace and then use them on a simulation platform that has built a NuRec integration. To use NuRec in a robotics simulation platform, you can reconstruct a scene from mono camera data or stereo camera data and then use the generated assets in Isaac Sim.
Get started with NuRec for robotics by following the Quickstart guide for robotics developers.
Get started with NuRec for AV by following the Quickstart guide for AV developers.
Get started with NuRec for simulation platform developers by following the Quickstart guide for simulation platform developers.
Get Started#
Understand How NuRec Works.