arviz_stats.reloo

arviz_stats.reloo(wrapper, loo_orig=None, k_threshold=-inf, pointwise=None)

Recalculate exact Leave-One-Out cross validation refitting where the approximation fails.

arviz_stats.loo estimates the values of Leave-One-Out (LOO) cross validation using Pareto Smoothed Importance Sampling (PSIS) to approximate its value. PSIS works well when the posterior and the posterior_i (excluding observation i from the data used to fit) are similar. In some cases, there are highly influential observations for which PSIS cannot approximate the LOO-CV, and a warning of a large Pareto shape is sent by ArviZ. These cases typically have a handful of bad or very bad Pareto shapes, and a majority of good or ok shapes.

Therefore, this may not indicate that the model is not robust enough nor that these observations are inherently bad, only that PSIS cannot approximate LOO-CV correctly. Thus, we can use PSIS for all observations where the Pareto shape is below a threshold and refit the model to perform exact cross validation for the handful of observations where PSIS cannot be used. This approach allows us to properly approximate LOO-CV with only a handful of refits, which in most cases is still much less computationally expensive than exact LOO-CV, which needs one refit per observation.

Parameters:

wrapperSamplingWrapper

An instance of a SamplingWrapper subclass that implements the necessary methods for model refitting. This wrapper allows reloo to work with any modeling framework.

loo_origELPDData, optional

Existing LOO results with pointwise data. If None, will compute PSIS-LOO-CV first using the data from wrapper.

k_thresholdfloat, optional

Pareto shape threshold. Observations with k values above this threshold will trigger a refit. Defaults to \(\min(1 - 1/\log_{10}(S), 0.7)\), where S is the number of samples.

pointwisebool, optional

If True, return pointwise LOO data. Defaults to rcParams["stats.ic_pointwise"].

Returns:

ELPDData

Updated LOO results where high Pareto k observations have been replaced with exact LOO-CV values from refitting.

Warning

Refitting can be computationally expensive. Check the number of high Pareto k values before using reloo to ensure the computation time is acceptable.

See also

loo

Pareto smoothed importance sampling LOO-CV

loo_moment_match

Moment matching for problematic observations

Notes

It is strongly recommended to first compute arviz_stats.loo on the inference results to confirm that the number of values above the threshold is small enough. Otherwise, prohibitive computation time may be needed to perform all required refits.

As an extreme case, artificially assigning all pareto_k values to something larger than the threshold would make reloo perform the whole exact LOO-CV. This is not generally recommended nor intended, however, if needed, this function can be used to achieve the result.

The term exact cross-validation is used here to refer to the process of refitting the model for each observation, which is also commonly called brute-force cross-validation. While this eliminates the PSIS approximation, other approximations remain in the process, such as those inherent in the fitting procedure itself and the use of cross-validation as a proxy for expected log predictive density (ELPD).

References

[1]

Vehtari et al. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5) (2017) https://doi.org/10.1007/s11222-016-9696-4 arXiv preprint https://arxiv.org/abs/1507.04544.

[2]

Vehtari et al. Pareto Smoothed Importance Sampling. Journal of Machine Learning Research, 25(72) (2024) https://jmlr.org/papers/v25/19-556.html arXiv preprint https://arxiv.org/abs/1507.02646