The Video Frontier: When AI Stopped Watching and Started Understanding

Author(s): Ampatishan Sivalingam Originally published on Towards AI. Part IV of the Multimodal Intelligence Series · The model learned to see. Then it learned to remember what it saw. This stack did not exist in 2023. The U-Net diffusion models that produced Stable Diffusion’s images could not, in any architecturally coherent sense, be extended to handle the temporal dimension of video. The entire infrastructure had to be rebuilt from first principles. This article is the story of that rebuilding. This stack did not exist in 2023. The U-Net diffusion models that produced Stable Diffusion’s images could not, in any architecturally coherent sense, be extended to handle the temporal dimension of video. The entire infrastructure had to be rebuilt from first principles. This article is the story of that rebuilding.The article discusses the evolution of video generation models, highlighting significant advancements in technology and architecture. It emphasizes the challenges faced in maintaining consistency across frames, the importance of spatio-temporal patching, and the integration of audio with video for a cohesive experience. Notably, it covers the emergence of leading models like OpenAI’s Sora 2 and Google’s Veo 3.1, which both significantly enhance video generation and audio-visual integration, leading to realistic simulations and greater alignment with real-world physics. Furthermore, it examines the ethical implications of such technologies, acknowledging the potential for misuse while presenting industry responses such as the C2PA for maintaining content authenticity. Read the full blog for free on Medium. Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor. Published via Towards AI