One task, many names
Visual correspondence connects locations across images. The tasks share a matching problem, but differ in what a match means, which labels are available, and how success is measured.
Given a point here,
where is its counterpart there?
A correspondence is a mapping between image locations. What counts as a counterpart changes with the task: the same surface, the same point over time, an analogous object part, or the same place seen by another sensor.
The displacement is q − p = (u, v).
Across object instances, this is a spatial offset—not physical motion through time.
Choose an example to see what is matched, what the labels contain, and where the difficulty lies.
Where did this visible surface point move?
Optical flow describes apparent motion in the image plane. A point on a car can move because the car moves, the camera moves, or both. The output is a two-dimensional displacement, not the object’s three-dimensional velocity.
- What counts as a match?
- The same physical surface point in two frames.
- Labels and evaluation
- A displacement (u, v) at each labeled pixel. Endpoint error measures the distance between predicted and annotated destinations; benchmarks may also report outlier rates. Dense prediction does not imply dense ground truth: KITTI labels only a subset of pixels.
- What makes it hard?
- Occlusion, motion blur, repeated textures, large displacement and regions with little visual detail. A point that disappears behind something has no visible counterpart in the next frame.
- In this dissertation
- Flow benchmarks help test whether the motion supplied during training transfers to new scenes.
Where does this point go over time?
Tracking extends correspondence from a pair of frames to a trajectory. Given a query point and its starting frame, the model follows the same point through the sequence and estimates when it is visible.
- What counts as a match?
- The same physical point over multiple frames, including after temporary occlusion.
- Labels and evaluation
- A sequence of image coordinates, with visibility or occlusion labels. Evaluation can measure localization accuracy and whether visibility was predicted correctly. It must assess the trajectory, not just one good pairwise match.
- What makes it hard?
- Errors can accumulate over time. A point may leave the image or disappear behind an object, then return; maintaining its identity is harder than finding a similar-looking patch.
- In this dissertation
- PointOdyssey supplies trajectories that can be sampled into image pairs and displacement vectors. The picture shows two frames from a longer track.
Where is the corresponding part on another object?
The source and target can show different objects of the same category: the eye of one cat and the corresponding eye of another. There is no single physical point moving between the images. The match preserves the meaning of the part across changes in pose, shape and appearance.
- What counts as a match?
- An analogous part or location, rather than the same physical surface point.
- Labels and evaluation
- Benchmarks such as PF-PASCAL annotate sparse semantic keypoints. Percentage of Correct Keypoints (PCK) counts predictions within a specified, normalized distance of the annotation; the threshold and normalization must be stated.
- What makes it hard?
- Different proportions, articulation, viewpoint and left–right ambiguity. A feature can look similar but belong to the wrong eye or leg.
- In this dissertation
- Semantic benchmarks test whether training on generated motion helps matching across instances, even though the generator does not create those object categories.
How should one image align with the other?
Geometric matching emphasizes the spatial mapping between images: how locations in one view map into another. For two views of a scene, this can support registration or reconstruction. Some mappings use a global transform; others need a dense field to capture parallax or deformation.
- What counts as a match?
- Locations consistent with the alignment being modeled. Depending on the problem, they may be the same scene points or corresponding points across object shapes.
- Labels and evaluation
- Matched points, a transformation, or a dense correspondence field. A single homography describes a planar projective mapping; it cannot represent arbitrary parallax in a three-dimensional scene.
- What makes it hard?
- Large viewpoint changes, nonrigid shape differences, repeated structures and areas visible in only one image.
- In this dissertation
- The slide’s TSS example illustrates dense alignment of foreground objects. This overlaps with semantic correspondence: “geometric” and “semantic” are not mutually exclusive dataset categories.
Where is this location in a different sensor’s image?
The same scene can look very different in RGB, infrared and synthetic-aperture radar (SAR). Registration seeks a spatial alignment across those observations. Matching by raw brightness is unreliable because the sensors respond to different physical properties.
- What counts as a match?
- A corresponding scene location, accounting for the sensors’ viewing geometry and visibility.
- Labels and evaluation
- Paired landmarks or a spatial mapping between the modalities. Image translation predicts how one sensor’s observation might look in another modality; that does not, by itself, establish correct geometric registration.
- What makes it hard?
- Different imaging geometry, resolution, shadows and sensor-specific effects. SAR layover can combine scene structures that appear separately in an optical image.
- In this dissertation
- Chapter 2 studies the aerial sensor gap, aligned data and latent translation. It establishes why obtaining suitable paired supervision is itself a research constraint.
What if we know the scene that made both images?
A renderer can project a known surface point into two views and compute where it goes. Scene geometry, camera parameters and object motion provide correspondence labels together with the images, instead of requiring someone to annotate every pixel.
- What counts as a match?
- A surface point carried through the known scene transformation and projected into the other view.
- Labels and evaluation
- Image pairs, per-pixel displacements and validity or visibility masks. The labels describe the motion the renderer actually produced. In Motion Sketch, they need not match every arrow the visitor requested.
- What makes it hard?
- Correct labels do not guarantee useful training data. A generator can produce plausible images while rarely supplying the motions a target task needs.
- In this dissertation
- Synthetic supervision is a source of training data, not a sixth matching task. Chapters 3 and 4 measure the motion it supplies and manufacture motion that a training set is missing.
Dense versus sparse
Dense prediction estimates a match across the image. Sparse annotation supplies selected points. Unlabeled pixels are not automatically incorrect or invisible; a label mask and a visibility mask answer different questions.
Matching versus recognition
Recognition can say “this is a cat.” Correspondence must locate the matching part in another image. Knowing the category does not resolve which eye, which pixel, or which surface.
Why the training pairs matter
The same displacement representation can describe very different tasks. Your training pairs determine which distances, directions, positions and appearance changes the model encounters. That is the link to measuring—and manufacturing—the motion.