Monitoring, drift & retraining
Before we begin
Models degrade when reality diverges from training data. Production CV needs telemetry, alerts, and a retraining playbook.
Learning objectives
- Distinguish input drift vs concept drift.
- Monitor confidence, latency, and input statistics.
- Use shadow deployments before cutover.
- Decide fine-tune vs full retrain.
Input distribution shift
Examples: new camera sensor, seasonal lighting, different geographic region.
Monitor histograms of brightness, blur estimates, aspect ratios — alert when KL divergence or simple z-scores exceed thresholds.
Concept drift
Same pixels, different meaning — new product SKU, changed defect types. Labels from training no longer match reality.
Requires new labeled data and model update — monitoring accuracy proxies (human review sample) is essential.
Shadow deployment
Run candidate model on live traffic in parallel; log disagreements with production model — no user impact until validated.
Human-in-the-loop
Route low-confidence predictions to reviewers; corrected labels feed next training round (active learning).
Retraining policy
| Signal | Action |
|---|---|
| Small drift, backbone still valid | Fine-tune on new batch |
| Large domain change | New data collection + full pipeline review |
| Latency regression | Profile ONNX/TRT; consider smaller model |