It's Tuesday, July 21st: Welcome to another edition of The Byte.

In this piece, Dr. Yanpei Cao, Chief Scientist at Tripo AI, argues that AI video generation models are building on a flawed foundation: they can produce convincing pixels, but they have no understanding of the persistent world behind them. When an object leaves the frame, the model forgets it, and when the camera pans back, it has to guess what was there. The result is closer to an optical illusion than actual world simulation.

Cao lays out Tripo's alternative: a decoupled architecture called Project Eden that separates the "physics" from the "pixels." Instead of forcing one network to both reason about the world and render it, Eden maintains a persistent state layer, a shared source of truth for geometry, objects, and physics, and hands off visuals to a dedicated renderer that translates rather than invents.

The larger ambition is a "ChatGPT moment" for interactive media: a future where anyone can describe a world in natural language and receive a playable environment in return, no expertise required. For Cao, this is not just a product vision but a step toward AGI, where AI moves beyond speaking and seeing to genuinely understanding the physical world.Editor’s Note

World Models as the Basis for the Next Generation of User-Generated Content

From the time that Tripo was founded in 2023, our vision for the company has been to build the bedrock infrastructure for the next generation of interactive user-generated content (UGC) platforms. We operate on a first-principles engineering culture: rather than indexing against competitor roadmaps, the team takes a pragmatic approach by focusing entirely on core problem-solving. The team continuously monitors cutting-edge developments in the global research community, looking for new tools and methodologies that align with our vision.  

Back then, high-quality 3D asset generation was the mandatory foundational step. You cannot compute the dynamics of a world if you have not first defined the objects inside it. We put significant resources into building native 3D generative architectures that would produce structurally sound, usable assets instead of mere visual approximations that required massive manual cleanup. Today, that technology has matured and is actively transforming studio workflows. 

At the same time, we observed a growing interest across the AI research community in state-prediction models—the ability of models to understand and predict how environments evolve over time. For us, this trend validated our long-held roadmap. To be truly persistent and interactive, a world model must simultaneously address "State" (the objective physical and structural condition of the world at any given moment, independent of the camera perspective) and "Transition" (the continuous evolution of the world driven by actions and physics).

The Illusion of Pure Video Models

Currently, many research initiatives are limited by the conceptual framework of video generation models. The real world is 3D, and the 2D pixels in a video are merely a compressed, low-dimensional projection.

In pure action-conditioned video generation, the state of the world is implicitly hidden within a history of pixels. When an object leaves the field of view, its independent state is not saved: the model simply forgets it. When the camera pans back, it relies on guessing to hallucinate the object anew. It has time, but no true spatial memory. Conversely, static 3D scene generation has a strong spatial structure but no time evolution: it has space, but no dynamics.

For a truly interactive experience, the system must understand how objects physically transform and interact. Video models merely act as "optical simulators" hallucinating pixels, whereas a true world model must act as a "physical simulator" deducing physical realities.

The Decoupled Architecture

The challenge with end-to-end monolithic video systems is that they face significant capacity bottlenecks. Forcing a single neural network to simultaneously compute complex state logic and generate high-fidelity pixel arrays is computationally crippling.

It wasn't until we hit upon the idea of decoupling the underlying state deduction from the final visual rendering that the basis for Project Eden, our world model, was born. Project Eden's architecture natively separates the "physics" from the "pixels" across three distinct layers:

  • Evolving Structured State: Instead of relying on pixels to remember the world, we maintain a persistent, globally shared world state. This compact, implicit/structured representation governs the underlying geometry, object semantics, and the physical consequences of any action inputs. It allows the world to exist independently of any single camera view.

  • State-to-Observation Interface: When a specific viewpoint is queried, this interface converts the underlying state into geometric and semantic constraints for that exact camera angle. Because every view is derived from a single, unified source of truth, it guarantees absolute spatial alignment and epipolar consistency.

  • Generative Rendering: Finally, the generative renderer receives these state constraints and translates them into high-fidelity visuals (dynamic textures, lighting, smoke, etc.). The renderer is no longer blindly "guessing" the structure; it is solely responsible for "translating" rather than "remembering."

The Data Engine: Dual-State Alignment

To train this decoupled architecture, we had to solve a massive data bottleneck. If you only train on traditional physics engines, your model becomes rigid and lacks the infinite diversity of the real world. If you only train on 2D video, your model learns hallucinations instead of physical boundaries.

Our solution is a "dual-state alignment" training base. First, we utilize Tripo's accumulated 3D foundation models to reverse-engineer massive amounts of unlabeled internet video, extracting depth, camera poses, and geometric trajectories (L1 Data). Second, we use traditional game engines to generate synthetic data with absolutely precise 3D state annotations and action logic (L2 Data). By combining the boundless generative diversity of video with the strict structural consistency of game engines, Project Eden learns the true "latent general actions" of the physical world. As a result, it has already demonstrated unprecedented capabilities in environmental persistence, scene reusability, and multi-agent concurrency.

The ChatGPT Moment for 3D

Returning to Tripo's overall vision from the beginning, we don't see Project Eden as just a game development tool, but as the underlying engine for the next generation of interactive content creation. For regular content creators, Project Eden bypasses the technical barriers and specialized expertise traditionally required for 3D world creation. 

Our long-term objective is to trigger the "ChatGPT Moment" for interactive media. For example, a user could input "a medieval tavern escape room containing a locked chest, with the key hidden inside a wine keg," and the system will instantly compile a fully realized, playable sandbox. Because all logic and physics run natively on the world model, the engineering cost of defining digital world rules drops to zero. Regular consumers can become active world-builders, laying the foundation for a universal next-generation UGC interactive platform. 

True world models are an indispensable milestone on the path toward Artificial General Intelligence (AGI). Language models taught AI to "speak," video models taught AI to "see" and "draw," while 3D foundational models paired with world models are teaching AI to "understand, generate, and deduce the physical world." The next step for AI will be to directly generate a truly interactive world, rather than just producing better content at a faster pace.

The AI Collective is built by volunteers across 180+ chapters in 40 countries.

Thank you to the thousands of volunteers around the world who make this work possible. We truly could not do this without you.

🧑‍💻 About the Author & the Editorial Team

Dr. Yanpei Cao is the Chief Scientist and Co-founder of VAST (Tripo AI), where he leads research on 3D foundation models and generative AI for interactive content creation. He holds a PhD in Computer Graphics from Tsinghua University and previously served as Principal Researcher at Tencent and CTO of Owlii, a startup focused on 4D reconstruction and 3D avatars. His team has had all six submissions accepted to CVPR 2024, and Tripo has been recognized by a16z as a key player in the 3D generation space.

About Josh Evans

Josh is a Managing Editor at The AI Collective Newsletter and leads content for The Byte. Outside of AIC, Josh works in Content Protection at Spotify.

Add Your Thoughts

Avatar

or to participate

Keep Reading