Body, Object, Motion

A robotic hand with twenty or more joints poses a control problem that does not yield to hand-written rules. We learn these policies instead, using human demonstrations as the supervision signal and looking for representations of contact and grasp that carry over to objects the robot has never held.

The binding constraint is data. Real manipulation demonstrations are slow and expensive to collect, so much of this work is about extracting more from each one — through augmentation that stays physically plausible, and through policy representations that generalise from a handful of examples rather than thousands.

Before a system can act on an object it has to know where the object is and what shape it takes. We reconstruct 3D structure from images and extend it through time into 4D, so that geometry and motion are estimated jointly instead of one after the other — each constrains the other, and solving them separately throws that constraint away.

We are particularly interested in what can be recovered without per-scene optimisation or dense supervision: priors strong enough that a single forward pass returns structure you can actually use.

Motion is the thread that connects perception to action. We work on point tracking through long videos, where the difficulty is holding onto identity through occlusion and deformation, and on trajectory forecasting, where the difficulty is that the future is genuinely multi-modal — a single most-likely path is usually the wrong answer to give.

Both problems are expensive to solve densely. We look for formulations that spend computation where the motion is, rather than uniformly across every frame and every pixel.