How Nvidia is planning to use AI to generate in-game elements

Nvidia CEO Jensen Huang recently offered a sneak peek at what the US chip giant is planning to do with the upcoming versions of its Deep Learning Super Sampling (DLSS). This real-time deep learning image enhancement and upscaling technologies developed by Nvidia are exclusive to its RTX line of graphics cards and available in multiple video games.
During an interview session with More Than Moore at Computex 2024, Huang answered a DLSS-related topic, saying that in the coming days, gamers will be able to see generated textures and objects that will be created through artificial intelligence (AI).Huang also stated that AI-generated NPCs in games will also be created through DLSS.

On being asked about AI in gaming, Huang said: “AI for gaming - we already use it for neural graphics, and we can generate pixels based off of a few input pixels. We also generate frames between frames - not interpolation, but generation. In the future, we’ll even generate textures and objects, and the objects can be of lower quality and we can make them look better. We’ll also generate characters in the games - think of a group of six people, two may be real, and the others may be long-term use AIs. The games will be made with AI, they’ll have AI inside, and you’ll even have the PC become AI using G-Assist. You can use the PC as an AI assistant to help your game. GeForce is the biggest gaming brand in the world, we only see it growing, and a lot of them have AI in some capacity. We can’t wait to let more people have it.”

Nvidia aims to improve in-game graphics with the next version of DLSS (Deep Learning Super Sampling) technology. This update aims to leverage the power of Tensor Cores on RTX GPUs to generate game assets like textures and objects directly using AI.
This shift will free up traditional shader (CUDA) cores, resulting in smoother gameplay with potentially higher frame rates that will help players experience games with enhanced visuals without sacrificing performance.

Huang also noted that DLSS will not just enhance frame rates but will also generate high-quality textures and objects similar to how the technology currently upscales existing frames. This could lead to a significant leap in visual fidelity without compromising performance on RTX GPUs.
According to a report by Tom’s Hardware, Nvidia is already developing a texture compression technology that utilises AI neural networks. This tech promises to significantly improve the quality of in-game textures while keeping memory usage under control.
Traditional compression methods typically achieve an 8x compression ratio. However, Nvidia's approach will reportedly use AI to achieve a higher 16x compression ratio. This essentially translates to double the texture detail without demanding double the video memory (VRAM) from the graphics card. This will offer smoother gameplay with better visuals, even on systems with limited VRAM.
Apart from generating in-game graphics, DLSS will also be able to generate non-playing characters (NPC), Huang added. He also gave an example of six people existing in a video game where two of them were real characters, while the other four were generated by AI.