Stereo depth & triangulation

Before we begin

Lesson 2 matched points between views. With two calibrated cameras (or one moving camera), those matches become depth — how far each point is from the camera.

This lesson covers rectified stereo, disparity, and triangulation.

Learning objectives

• Define disparity and relate it to depth.
• Explain rectification and why search becomes 1D.
• Triangulate a 3D point from two views.
• Name failure modes: textureless regions, occlusions, reflective surfaces.

Disparity and depth

For a horizontal baseline setup (cameras side by side), a scene point projects to $x_L$ in the left image and $x_R$ in the right. Disparity $d = x_L - x_R$ (sign convention varies).

With focal length $f$ (pixels) and baseline $B$ (meters):

Smaller disparity → farther away. Points at infinity have $d = 0$.

Rectified stereo

General stereo has matches along epipolar lines. Rectification warps both images so epipolar lines are horizontal rows — for each row, search disparity along columns only.

OpenCV: cv2.stereoRectify + cv2.remap using calibration from Lesson 1.

Matching cost

For each pixel (or patch), compare left patch to right patch at candidate disparities. Pick disparity minimizing SAD, SSD, or normalized cross-correlation.

Semi-global matching (SGM) adds smoothness — popular in robotics and automotive.

Triangulation

Given two camera projection matrices $P_1$, $P_2$ and matching pixels, solve for 3D point $X$ minimizing reprojection error (DLT or cv2.triangulatePoints).

Noise in pixel location → depth uncertainty grows with distance and narrow baseline.

Failure modes

Checkpoint

If baseline doubles and focal length stays fixed, what happens to disparity for the same scene point at fixed depth?

Answer sketch: Disparity doubles — same depth gives larger pixel shift; depth formula unchanged if you measure d correctly.

What's next

Module 2 quiz — then panorama stitching project.