Split Conformal Prediction
The foundational algorithm. Split data into training and calibration sets, compute nonconformity scores on calibration data, then construct prediction intervals with guaranteed marginal coverage. Watch how the prediction band adapts to the true data-generating process.
Theory
Given a calibration set {(X₁,Y₁),...,(Xₙ,Yₙ)} and a fitted model μ̂, compute nonconformity scores:
Rᵢ = |Yᵢ − μ̂(Xᵢ)|
For a new test point X_{n+1}, the prediction interval at level 1−α is:
Ĉ(X_{n+1}) = μ̂(X_{n+1}) ± Q₁₋α(R₁,...,Rₙ, ∞)
where Q₁₋α is the ⌈(1−α)(n+1)⌉/n quantile of the augmented scores. This guarantees P(Y_{n+1} ∈ Ĉ(X_{n+1})) ≥ 1−α under exchangeability.
Data & Prediction Interval
Nonconformity Score Distribution
Adaptive Conformal Inference (ACI)
Standard conformal prediction assumes exchangeability — but real-world data drifts. ACI (Gibbs & Candès, 2021) adapts the miscoverage level αₜ online using a gradient update, maintaining long-run coverage even under arbitrary distribution shift. Watch the prediction sets expand and contract as the data distribution changes.
Theory
ACI updates the effective miscoverage level at each step:
αₜ₊₁ = αₜ + γ(α − errₜ)
where errₜ = 𝟙{Yₜ ∉ Ĉₜ(Xₜ)} indicates miscoverage at time t, γ is the learning rate, and α is the target miscoverage. The key theorem: for any sequence of distributions, ACI achieves:
lim_{T→∞} (1/T) Σₜ errₜ = α (almost surely)
Time Series with Adaptive Prediction Intervals
Coverage & Adaptive α Over Time
Conformal Classification Sets
For classification, conformal prediction produces prediction SETS — subsets of labels guaranteed to contain the true class with probability ≥ 1−α. Observe how the set sizes vary with model confidence: high-confidence regions get singleton sets, while ambiguous regions get larger sets. This is the essence of epistemic humility.
Theory
Given softmax scores π̂(x) = (π̂₁(x),...,π̂ₖ(x)), define nonconformity scores:
s(x,y) = 1 − π̂ᵧ(x) (simple) or Σ_{j: π̂ⱼ≥π̂ᵧ} π̂ⱼ (APS)
The prediction set includes all labels y where s(x,y) ≤ q̂, with q̂ the (1−α)-quantile of calibration scores. The Adaptive Prediction Set (APS) method by Romano et al. produces smaller, more efficient sets.
Decision Regions & Prediction Set Sizes
Set Size Distribution
Coverage Guarantee Visualizer
The central promise of conformal prediction: finite-sample, distribution-free coverage guarantees. Run Monte Carlo experiments to empirically verify that coverage holds across thousands of independent trials. Explore the gap between marginal and conditional coverage, and see how calibration set size affects coverage reliability.
Theory
For exchangeable data, conformal prediction guarantees:
1 − α ≤ P(Y_{n+1} ∈ Ĉ(X_{n+1})) ≤ 1 − α + 1/(n+1)
This is a finite-sample guarantee — no asymptotics needed. However, this is marginal coverage (averaged over X). Conditional coverage P(Y ∈ Ĉ(X) | X=x) ≥ 1−α is generally impossible without assumptions (Vovk, 2012; Barber et al., 2021).
Empirical Coverage Distribution (Histogram)
Coverage vs Calibration Size
Conformalized Quantile Regression (CQR)
CQR (Romano, Patterson, Candès, 2019) combines quantile regression with conformal calibration. Unlike standard conformal prediction which produces constant-width intervals, CQR produces locally adaptive intervals — wider where uncertainty is high, narrower where the model is confident. The best of both worlds: adaptivity AND guaranteed coverage.
Theory
Fit quantile regression models q̂_lo(x) at level α/2 and q̂_hi(x) at level 1−α/2. Compute conformity scores on calibration data:
Eᵢ = max(q̂_lo(Xᵢ) − Yᵢ, Yᵢ − q̂_hi(Xᵢ))
The CQR prediction interval is:
Ĉ(x) = [q̂_lo(x) − Q₁₋α(E), q̂_hi(x) + Q₁₋α(E)]
This inherits the adaptivity of quantile regression while restoring the coverage guarantee via conformal calibration.
CQR Adaptive Intervals
Interval Width Comparison
Mondrian Conformal Prediction
Standard conformal prediction guarantees marginal coverage, but coverage can vary wildly across subgroups — a critical fairness concern. Mondrian conformal prediction partitions the data into groups and calibrates separately within each group, guaranteeing group-conditional coverage. Essential for equitable AI deployment.
Theory
Partition the feature space into groups G₁,...,Gₖ. For each group Gⱼ, compute the conformal quantile q̂ⱼ using only calibration points in that group:
q̂ⱼ = Quantile₁₋α({Rᵢ : Xᵢ ∈ Gⱼ})
For a new point x ∈ Gⱼ, the prediction interval uses q̂ⱼ instead of the global quantile. This guarantees:
P(Y ∈ Ĉ(X) | X ∈ Gⱼ) ≥ 1 − α for each group Gⱼ
Trade-off: group-conditional coverage requires more calibration data per group, leading to wider intervals in small groups.
Marginal vs Mondrian Coverage by Group
Interval Widths by Group