How the depth model works

a primer to Painting → Bas-Relief, on monocular depth estimation and Depth Anything V2
No machine-learning background is assumed.

0. What the model does

One image goes in. Out comes a map of the same size in which every pixel holds a number meaning “how near is this?” — larger is nearer. That is the whole interface.

Two properties of that output matter more than anything else on this page. First, it is an estimate of relative order, not of metres: the model says the tree is in front of the village, not that it is 40 m in front. Second, the numbers come with an arbitrary scale and offset — run the model twice on two crops and the ranges will not match. Both properties are consequences of how the model is trained (§4), and both shape the relief pipeline (§6).

Strictly the map is disparity — proportional to 1/distance, the convention inherited from stereo vision — which is why near things get large values. For ordering purposes disparity and depth carry the same information.

1. How one image can contain depth

Depth from a single image is geometrically impossible: infinitely many 3-D scenes project to the same picture. What makes it possible in practice is that the world is not arbitrary. Real scenes leak depth through cues that painters have exploited for six centuries:

• Occlusion — the thing that blocks is nearer than the thing blocked. The strongest cue, and purely ordinal.
• Familiar size — a person 15 pixels tall is far away, because people come in roughly one size.
• Texture gradient — surface detail compresses with distance; individual ripples become smooth water.
• Perspective — parallel edges converge; things sit higher in the frame as they recede toward the horizon.
• Atmosphere — contrast and saturation fall with distance; far mountains are pale and blue.

The model has no rule for any of these. It has seen tens of millions of images and absorbed the statistics that connect appearance to distance — a photographic prior. This is the reason the build guide’s §1 argues about the painting choice: a painting works exactly to the degree that it is painted with photographic cues. Bruegel builds his recession from occlusion, scale and atmosphere, so the prior grabs on. Abstract art gives the prior nothing, and the output is confident nonsense — the model always answers, whether or not the question makes sense.

2. The architecture, briefly

Depth Anything V2 is a vision transformer (ViT) with a dense-prediction head. The image is cut into 14 × 14-pixel patches; each patch becomes a token — a vector summarising that square; a stack of attention layers then lets every patch look at every other patch. That global view is the point: whether one grey square is snow-in-shadow or distant-hillside depends on the horizon, the haze, and the objects around it, and attention is the mechanism that carries that context. A decoder (DPT, for dense prediction transformer) reassembles features from several depths of the stack into one full-resolution map.

image  ->  14x14 patches  ->  tokens  ->  attention stack   ->  DPT decoder  ->  depth map
           (a few thousand)              (every patch sees            (multi-scale
                                          every other patch)           reassembly)

The patch size is why the build guide’s depth.py rounds working resolutions to multiples of 14, and the fixed token budget is why the model has a natural working resolution — feed it the whole painting and fine branches fall below the patch grid, which is what the guide’s tiled refinement (§5.4) exists to recover.

3. How it is trained

The obstacle is labels. Real photographs with true per-pixel depth are scarce and flawed: LiDAR is sparse, stereo rigs mismatch on glass and sky, and no sensor traces a bare branch pixel-perfectly. Synthetic images from renderers have perfect labels but do not look quite like the world. Depth Anything V2 splits the problem into three steps:

• Teacher. A very large model (based on DINOv2-Giant, 1.3B parameters) is trained on 595 K synthetic images only — rendered scenes whose depth is exact to the pixel. This is where the razor-sharp edges come from: the teacher has never seen a blurry label.
• Pseudo-labels. The teacher then labels 62 million real, unlabeled photographs. The labels are imperfect, but they are dense, sharp, and span the whole visual world.
• Students. The released models (25 M to 1.3 B parameters) are trained on those pseudo-labeled real photos alone — inheriting the teacher’s edge quality and the real world’s diversity at a fraction of the size.

The lineage matters for expectations. This family (from MiDaS through Depth Anything) was built to generalise to arbitrary images rather than to win one benchmark — which is exactly the property that lets it work on a 460-year-old painting it was never trained for.

4. The loss, and why the output is relative

Training data mixes sources whose depth units disagree — one dataset in metres, another in disparity, another normalised. The standard solution, inherited from MiDaS, is an affine-invariant loss: before comparing prediction to ground truth, fit the best scale-and-offset that aligns them, and penalise only what remains. The model is therefore never rewarded for absolute numbers, only for getting the shape of the depth field right — and so absolute scale is simply not something it learns. A gradient-matching term is added so predicted depth edges land sharply where true edges are.

This is a fair trade, made deliberately: give up metres, gain the ability to train on everything. It is also why the build guide normalises the model’s output to 0–1 immediately (norm01, §6) and why the physical height of the printed relief is set by RELIEF_MM in the pipeline’s config, never by the model. (Metric variants of these models exist, fine-tuned to output metres for robotics; a relief that rescales everything anyway has no use for them.)

5. How depth models are scored

Two metrics dominate the papers, both computed after the affine alignment above:

• AbsRel (absolute relative error): the average of |predicted − true| / true over all pixels. An AbsRel of 0.05 means the typical pixel is within 5 % of its true depth. Lower is better.
• δ₁ (“delta-one”): the fraction of pixels whose predicted depth is within a factor of 1.25 of the truth. Current large models score in the high 0.9s on standard indoor benchmarks — nearly every pixel lands within 25 %. (δ₂ and δ₃ loosen the factor to 1.25² and 1.25³.)

Useful context, with a caveat: neither metric measures what a relief cares about most. A model can score well while smearing the boundary of every branch by three pixels — benchmark ground truth is too coarse to punish it. Edge sharpness is exactly where V2 improved over its predecessors (the synthetic-teacher design above), and it is why this pipeline uses V2. The benchmark that finally matters here is physical: whether the hunters separate from the snow on a printed panel.

6. What this means for the relief pipeline

• Relative output → the pipeline normalises immediately and owns the physical scale itself (RELIEF_MM, guide §4).
• Arbitrary polarity conventions across model families → the polarity check on 01_depth_global.png (guide §9): verify bright = near before anything else consumes the map.
• Patch grid of 14 → working resolutions are rounded to multiples of 14 (guide §5).
• Fixed token budget → fine detail falls below the grid at panel scale; tiled refinement re-runs the model on overlapping crops and keeps the high frequencies (guide §5.4).
• A prior, not a measurement → the estimate is plausible rather than true, and paintings amplify that: depth is reconstructed from the same cues the painter composed. The conditioning stages of §6 and §12–13 of the guide treat the map as raw material to be reshaped, not as ground truth.

7. Which model to use

The released family spans Small (25 M parameters), Base (98 M) and Large (335 M); the build guide uses Depth-Anything-V2-Large-hf through the Hugging Face transformers API. Large is the right choice for this use: the run is offline, seconds of inference are irrelevant, and edge quality is the commodity the relief consumes. Small exists for real-time and embedded use, where this pipeline never goes.

References

• Yang et al. (2024), Depth Anything V2 — the model this pipeline runs: synthetic-only teacher, 62 M pseudo-labeled reals, student distillation.
• Yang et al. (2024), Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data — V1, where the pseudo-labeling recipe was established.
• Ranftl et al. (2020), Towards Robust Monocular Depth Estimation (MiDaS) — the affine-invariant loss and cross-dataset training that made “depth for any image” possible.
• Ranftl et al. (2021), Vision Transformers for Dense Prediction (DPT) — the decoder architecture.
• Oquab et al. (2023), DINOv2 — the self-supervised backbone the teacher grows from.