Nvidia has unveiled its Cosmos World Foundation Model platform for the development of physical AI systems such as autonomous vehicles and robots.
Cosmos comprises state-of-the-art generative world foundation models, advanced tokenizers, guardrails and an accelerated video processing pipeline built to “advance the development of physical AI systems”, says Nvidia.
Physical AI models are costly to develop, and require vast amounts of real-world data and testing. Cosmos world foundation models, or WFMs, offer developers an easy way to generate massive amounts of photoreal, physics-based synthetic data to train and evaluate their existing models. Developers can also build custom models by fine-tuning Cosmos WFMs.
Cosmos models will be available under an open model license to accelerate the work of the robotics and AV community. Developers can preview the first models on the Nvidia API catalog, or download the family of models and fine-tuning framework from the Nvidia NGC catalog or Hugging Face.
Leading robotics and automotive companies, including 1X, Agile Robots, Agility, Figure AI, Foretellix, Fourier, Galbot, Hillbot, IntBot, Neura Robotics, Skild AI, Virtual Incision, Waabi and XPENG, along with ridesharing giant Uber, are among the first to adopt Cosmos.
Jensen Huang, founder and CEO of Nvidia, says: “The ChatGPT moment for robotics is coming. Like large language models, world foundation models are fundamental to advancing robot and AV (autonomous vehicle) development, yet not all developers have the expertise and resources to train their own.
“We created Cosmos to democratize physical AI and put general robotics in reach of every developer.”
In his opening keynote at CES, Nvidia founder and CEO Jensen Huang showcased ways physical AI developers can use Cosmos models, including for:
Video search and understanding, enabling developers to easily find specific training scenarios, like snowy road conditions or warehouse congestion, from video data.
Physics-based photoreal synthetic data generation, using Cosmos models to generate photoreal videos from controlled 3D scenarios developed in the Nvidia Omniverse platform.
Physical AI model development and evaluation, whether building a custom model on the foundation models, improving the models using Cosmos for reinforcement learning or testing how they perform given a specific simulated scenario.
Foresight and “multiverse” simulation, using Cosmos and Omniverse to generate every possible future outcome an AI model could take to help it select the best and most accurate path.
Cosmos WFMs are now available under Nvidia open model license on Hugging Face and the Nvidia NGC catalog. Cosmos models will soon be available as fully optimized Nvidia NIM microservices.
Developers can access Nvidia NeMo Curator for accelerated video processing and customize their own world models with Nvidia NeMo. Nvidia DGX Cloud offers a fast and easy way to deploy these models, with enterprise support available through the Nvidia AI Enterprise software platform.
Nvidia also announced new Nvidia Llama Nemotron large language models and Nvidia Cosmos Nemotron vision language models that developers can use for enterprise AI use cases in healthcare, financial services, manufacturing and more.
