Modeling the Visual World Across Space and Time: From Observation to Interaction

A single image entangles the scene, the viewpoint, and the moment at which it was captured, and separates none of them; extended through time, images become video, a projection of the visual world across both space and time. Recovering structure that individual measurements do not carry is the precondition for almost everything a machine might do with visual data, from answering where an object is to changing what a video shows while keeping the change coherent. This dissertation studies that recovery and the interaction it makes possible.

The first part treats the visual world as something to be recovered from imperfect observation. When views are few and their poses approximate, correspondences that are satisfied pairwise can remain jointly inconsistent; carrying them globally as feature tracks, so that one three-dimensional location explains every view in which a point appears, recovers scene geometry and camera poses where pairwise supervision fails. The same requirement, applied to long first-person video, makes an object-centred spatial query answerable from episodic visual memory, provided the camera poses are recovered from the footage itself rather than by registering it against an external scan. The part closes by asking whether large video generative models already encode geometry that can be read out directly as four-dimensional structure.

The second part turns from observation to interaction. Once the visual world is modeled through video, interacting with that model means changing the video itself while keeping it coherent, which is the problem of video editing. An instruction, an optional spatial or temporal region, and an optional reference image are shown to be sufficient conditioning for such editing. The final contribution removes the human from the timing decision: given a rule stating that an edit should apply once a described event occurs, the system determines from the pixels whether and when it does, preserves everything before it, edits only the continuation, and can run incrementally on a stream without access to future frames.

Together, these contributions establish spatial consistency, maintain temporal coherence, and ground transformations in user intent and observed events, modeling the visual world across space and time, from observation to interaction.