From 303aac8f00c3d0fa57f07ddefab9bd5f7506e420 Mon Sep 17 00:00:00 2001 From: Valentin Laurent Date: Fri, 10 Jan 2025 13:05:03 +0100 Subject: [PATCH] DOC: clarified that calibration and alpha are now conformalization and confidence level (#595) --- doc/theoretical_description_binary_classification.rst | 7 +++++++ doc/theoretical_description_calibration.rst | 7 +++++++ doc/theoretical_description_classification.rst | 7 +++++++ doc/theoretical_description_conformity_scores.rst | 7 +++++++ doc/theoretical_description_metrics.rst | 7 +++++++ doc/theoretical_description_mondrian.rst | 7 +++++++ doc/theoretical_description_multilabel_classification.rst | 7 +++++++ doc/theoretical_description_regression.rst | 7 +++++++ doc/v1_migration_guide.rst | 8 ++++---- 9 files changed, 60 insertions(+), 4 deletions(-) diff --git a/doc/theoretical_description_binary_classification.rst b/doc/theoretical_description_binary_classification.rst index 9c8f6f336..d06914ca8 100644 --- a/doc/theoretical_description_binary_classification.rst +++ b/doc/theoretical_description_binary_classification.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + There are mainly three different ways to handle uncertainty quantification in binary classification: calibration (see :doc:`theoretical_description_calibration`), confidence interval (CI) for the probability :math:`P(Y \vert \hat{\mu}(X))` and prediction sets (see :doc:`theoretical_description_classification`). diff --git a/doc/theoretical_description_calibration.rst b/doc/theoretical_description_calibration.rst index c62540337..fe5588d14 100644 --- a/doc/theoretical_description_calibration.rst +++ b/doc/theoretical_description_calibration.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + One method for multi-class calibration has been implemented in MAPIE so far : Top-Label Calibration [1]. diff --git a/doc/theoretical_description_classification.rst b/doc/theoretical_description_classification.rst index 445fcfe42..c96bddd7e 100644 --- a/doc/theoretical_description_classification.rst +++ b/doc/theoretical_description_classification.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + Three methods for multi-class uncertainty quantification have been implemented in MAPIE so far : LAC (that stands for Least Ambiguous set-valued Classifier) [1], Adaptive Prediction Sets [2, 3] and Top-K [3]. The difference between these methods is the way the conformity scores are computed. diff --git a/doc/theoretical_description_conformity_scores.rst b/doc/theoretical_description_conformity_scores.rst index ed281962a..e0b808fca 100644 --- a/doc/theoretical_description_conformity_scores.rst +++ b/doc/theoretical_description_conformity_scores.rst @@ -6,6 +6,13 @@ Theoretical Description for Conformity Scores ############################################# +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + The :class:`mapie.conformity_scores.ConformityScore` class implements various methods to compute conformity scores for regression. We give here a brief theoretical description of the scores included in the module. diff --git a/doc/theoretical_description_metrics.rst b/doc/theoretical_description_metrics.rst index f94606ded..98eef4ebc 100644 --- a/doc/theoretical_description_metrics.rst +++ b/doc/theoretical_description_metrics.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + This document provides detailed descriptions of various metrics used to evaluate the performance of predictive models, particularly focusing on their ability to estimate uncertainties and calibrate predictions accurately. 1. General Metrics diff --git a/doc/theoretical_description_mondrian.rst b/doc/theoretical_description_mondrian.rst index 7b93b3164..52774e728 100644 --- a/doc/theoretical_description_mondrian.rst +++ b/doc/theoretical_description_mondrian.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + Mondrian conformal prediction (MCP) [1] is a method that allows to build prediction sets with a group-conditional coverage guarantee. The coverage guarantee is given by: diff --git a/doc/theoretical_description_multilabel_classification.rst b/doc/theoretical_description_multilabel_classification.rst index e3ff05da3..a3055faff 100644 --- a/doc/theoretical_description_multilabel_classification.rst +++ b/doc/theoretical_description_multilabel_classification.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + Three methods for multi-label uncertainty quantification have been implemented in MAPIE so far : Risk-Controlling Prediction Sets (RCPS) [1], Conformal Risk Control (CRC) [2] and Learn Then Test (LTT) [3]. The difference between these methods is the way the conformity scores are computed. diff --git a/doc/theoretical_description_regression.rst b/doc/theoretical_description_regression.rst index a56827201..55d6800db 100644 --- a/doc/theoretical_description_regression.rst +++ b/doc/theoretical_description_regression.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + The methods in `mapie_v1.regression` use various resampling methods based on the jackknife strategy recently introduced by Foygel-Barber et al. (2020) [1]. diff --git a/doc/v1_migration_guide.rst b/doc/v1_migration_guide.rst index a92b279e5..ca93b227a 100644 --- a/doc/v1_migration_guide.rst +++ b/doc/v1_migration_guide.rst @@ -40,11 +40,11 @@ In v0.9, data splitting is handled by MAPIE. In v1, the data splitting is left to the user, with the exception of cross-conformal methods (``CrossConformalRegressor``). The user can split the data into training, conformalization, and test sets using scikit-learn's ``train_test_split`` or other methods. -Step 2 & 3: Model training and conformalization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In v0.9, the ``fit`` method handled both model training and conformalization. +Step 2 & 3: Model training and conformalization (ie: calibration) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +In v0.9, the ``fit`` method handled both model training and calibration. -In v1.0: MAPIE separates between the training and conformalization: +In v1.0: MAPIE separates between training and calibration. We decided to name the *calibration* step *conformalization*, to avoid confusion with probability calibration. - ``.fit()`` method: - In v1, ``fit`` only trains the model on training data, without handling conformalization.