Nvidia’s new Cosmos-Transfer1 AI model aims to revolutionise robot training through simulation

Nvidia has introduced a groundbreaking artificial intelligence (AI) model designed to advance the training of AI-powered robotic systems through simulation. Dubbed Cosmos-Transfer1, this newly released large language model (LLM) aims to provide granular control over simulation environments, making it a significant tool for developers working on robotics training.

The tech giant launched the model as an open-source resource under a permissive licence, making it accessible to interested developers and researchers on popular repositories like GitHub and Hugging Face. The model is the latest addition to Nvidia’s Cosmos Transfer World Foundation Models (WFMs), a series of AI models aimed at enhancing simulation-based robotics training.

Simulation-based training is becoming increasingly popular in the robotics sector, particularly when it comes to developing hardware capable of using AI as its core processing unit. Unlike conventional robots in factories that are designed to perform specific tasks, this approach allows machines to be trained for a broader range of real-world scenarios, significantly enhancing their versatility.

Nvidia’s Cosmos-Transfer1 uses structured video input, such as segmentation maps, depth maps, lidar scans, and more, to generate high-quality, photorealistic video outputs. These outputs can then be utilised as training grounds for AI-powered robots, allowing them to learn from diverse simulated environments.

According to a paper published by the company in the arXiv journal, Cosmos-Transfer1 offers superior customisation compared to previous models. “It enables varying the weight of different conditional inputs based on spatial location, allowing developers to create highly controllable simulation environments,” Nvidia stated.

Cosmos-Transfer1 is a diffusion-based model equipped with seven billion parameters and is optimised for video denoising in the latent space. Its control branch allows the model to accept text and video inputs to generate photorealistic output videos. Four types of control input videos are supported: canny edge, blurred RGB, segmentation mask, and depth map.

The AI model has been thoroughly tested on Nvidia’s Blackwell and Hopper series chipsets, with inference conducted on the Linux operating system. Its design allows for real-time world generation, providing a more efficient and diverse training experience for AI systems.

Nvidia has made the Cosmos-Transfer1 AI model available under the Nvidia Open Model License Agreement, which permits both academic and commercial use. Developers and researchers can download the model from Nvidia’s GitHub listing and Hugging Face listing.