NVIDIA has released NVIDIA Cosmos Predict-2, a new world foundation model aimed at improving future world state prediction for synthetic data generation in autonomous vehicle development. The company said the model is part of the NVIDIA Cosmos platform and is intended to help developers build next-generation AV architectures that use larger end-to-end models to execute driving actions directly from sensor data.
Cosmos Predict-2 builds on Cosmos Predict-1, which was designed to predict and generate future world states using text, image and video prompts. NVIDIA said the new model better understands context from text and visual inputs, which leads to fewer hallucinations and richer details in generated videos.
The company said Cosmos Predict-2 speeds up synthetic data generation on NVIDIA GB200 NVL72 systems and NVIDIA DGX Cloud using the latest optimization techniques. NVIDIA also said the model improves text adherence and common sense for a stop sign at the intersection.
NVIDIA said developers can post-train Cosmos models on AV data to generate videos that match existing physical environments and vehicle trajectories, and to create multi-view videos from a single-view video such as dashcam footage. The company said this can turn widely available dashcam data into multi-camera data and can also replace real camera data from broken or occluded sensors.
The NVIDIA Research team post-trained Cosmos models on 20,000 hours of real-world driving data. NVIDIA said the AV-specific models improved performance in challenging conditions such as fog and rain.
Several companies are already using Cosmos models. Plus, which is building its solution with the NVIDIA DRIVE AGX platform, is post-training Cosmos Predict on trucking data to generate synthetic driving scenarios. Oxa is using Cosmos Predict to support the generation of multi-camera videos with high fidelity and temporal consistency.
Alongside Cosmos Predict-2, NVIDIA announced Cosmos Transfer as an NVIDIA NIM microservice preview for deployment on data center GPUs. NVIDIA said the service augments datasets and generates photorealistic videos using structured input or ground-truth simulations from the NVIDIA Omniverse platform. It also announced the NuRec Fixer model to inpaint and resolve gaps in reconstructed AV data.
CARLA, the open-source AV simulator, will integrate Cosmos Transfer and NVIDIA NuRec into its latest release. NVIDIA said this will allow CARLA’s user base of over 150,000 AV developers to render synthetic simulation scenes and viewpoints with high fidelity and generate variations of lighting, weather and terrain using simple prompts.
The latest release of the NVIDIA Physical AI Dataset includes 40,000 clips generated using Cosmos, along with sample reconstructed scenes for neural rendering. NVIDIA said developers can use the data to author new trajectories, reposition sensors and simulate drives.
NVIDIA also said Bosch, Easyrain and Nuro are the latest automotive leaders to join the NVIDIA Halos AI Systems Inspection Lab to verify the safe integration of their products with NVIDIA technologies and advance AV safety. Earlier members announced this year include Continental, Ficosa, OMNIVISION, onsemi and Sony Semiconductor Solutions.
Source: blogs.nvidia.com.
Companies can share verified announcements through Newz9’s international press release submission page.
