Skip to content
04 · Just What the Doctor Ordered/Evidence · 28
Home ↗
Slide 28 the pharmacist and the generator

The pharmacist and the generator

The pharmacist grounds each ideal request in a realizable scene; the generator renders it with exact labels. The pharmacist uses a Perceiver IO encoder and a DETR-style set decoder. Motion vectors enter only as keys and values, so the read is permutation-invariant and linear in set size; each object query emits placement and motion as one committed six-vector. Trained on near-uniform renders so it carries minimal bias toward any motion regime. Round-trip A→B→A median error is exactly zero at 1 px quantization.

Pharmacist architecture
The label guarantee. Deformation bends the space, not the object. Every deformed point maps back to one point on the undeformed surface, so the labels stay exact under bending, and a one-line cycle check rules out folds.
Interactive · try it hereOpen standalone ↗
Same scene. Any order.
The model predicts a set of objects. Should it lose points just for listing them in a different order? Try it.
STEP 1 OF 3

Move the orange piece into target A.

Two objects already fit. Drag the orange disc onto the dashed circle, or use the button below.

◌ Dashed outline = target● Solid disc = prediction
Show the matching and the cost calculation

This flat puzzle uses |Δx| + |Δy| + 2(|Δmotion-x| + |Δmotion-y|), in scene units. The training model uses full 3D placement and motion vectors; this is a simplified 2D illustration.

The highlighted cells form the cheapest one-to-one assignment, found by checking all six pairings. Reordering rows cannot change that minimum.

Why this matters: a scene has no natural “first object.” The pharmacist's object queries predict placement and motion together. Assignment pairs the predictions with the true objects before computing their loss, so output order does not define object identity. The separate camera prediction is outside this object matching.
Interactive · try it hereOpen standalone ↗
Pharmacist Round Trips
A request goes in as a bag of motion vectors; the pharmacist emits a scene; the generator renders it with exact labels. Teal = requested, orange = delivered, measured from the rendered pair's flow.
"Copy" requests take a real benchmark pair's own annotated motion as the bag; the round trip shows how faithfully a fabricated scene can reproduce it (the labels are exact either way). "Gap-fill" requests are what the doctor actually writes: a fabricated bag aimed at deficient cells of the coverage table (a placement, a magnitude band, a direction sector, and a camera kind). The pharmacist never sees an image; it reads the vectors as keys and values, so the read is permutation-invariant, and emits six committed numbers per object. Delivered magnitude tracks the request; direction spread is realised by the camera kind.