diff --git a/doc/theoretical_description_binary_classification.rst b/doc/theoretical_description_binary_classification.rst index 9c8f6f33..d06914ca 100644 --- a/doc/theoretical_description_binary_classification.rst +++ b/doc/theoretical_description_binary_classification.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + There are mainly three different ways to handle uncertainty quantification in binary classification: calibration (see :doc:`theoretical_description_calibration`), confidence interval (CI) for the probability :math:`P(Y \vert \hat{\mu}(X))` and prediction sets (see :doc:`theoretical_description_classification`). diff --git a/doc/theoretical_description_calibration.rst b/doc/theoretical_description_calibration.rst index c6254033..fe5588d1 100644 --- a/doc/theoretical_description_calibration.rst +++ b/doc/theoretical_description_calibration.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + One method for multi-class calibration has been implemented in MAPIE so far : Top-Label Calibration [1]. diff --git a/doc/theoretical_description_classification.rst b/doc/theoretical_description_classification.rst index 445fcfe4..c96bddd7 100644 --- a/doc/theoretical_description_classification.rst +++ b/doc/theoretical_description_classification.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + Three methods for multi-class uncertainty quantification have been implemented in MAPIE so far : LAC (that stands for Least Ambiguous set-valued Classifier) [1], Adaptive Prediction Sets [2, 3] and Top-K [3]. The difference between these methods is the way the conformity scores are computed. diff --git a/doc/theoretical_description_conformity_scores.rst b/doc/theoretical_description_conformity_scores.rst index ed281962..e0b808fc 100644 --- a/doc/theoretical_description_conformity_scores.rst +++ b/doc/theoretical_description_conformity_scores.rst @@ -6,6 +6,13 @@ Theoretical Description for Conformity Scores ############################################# +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + The :class:`mapie.conformity_scores.ConformityScore` class implements various methods to compute conformity scores for regression. We give here a brief theoretical description of the scores included in the module. diff --git a/doc/theoretical_description_metrics.rst b/doc/theoretical_description_metrics.rst index f94606de..98eef4eb 100644 --- a/doc/theoretical_description_metrics.rst +++ b/doc/theoretical_description_metrics.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + This document provides detailed descriptions of various metrics used to evaluate the performance of predictive models, particularly focusing on their ability to estimate uncertainties and calibrate predictions accurately. 1. General Metrics diff --git a/doc/theoretical_description_mondrian.rst b/doc/theoretical_description_mondrian.rst index 7b93b316..52774e72 100644 --- a/doc/theoretical_description_mondrian.rst +++ b/doc/theoretical_description_mondrian.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + Mondrian conformal prediction (MCP) [1] is a method that allows to build prediction sets with a group-conditional coverage guarantee. The coverage guarantee is given by: diff --git a/doc/theoretical_description_multilabel_classification.rst b/doc/theoretical_description_multilabel_classification.rst index e3ff05da..a3055faf 100644 --- a/doc/theoretical_description_multilabel_classification.rst +++ b/doc/theoretical_description_multilabel_classification.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + Three methods for multi-label uncertainty quantification have been implemented in MAPIE so far : Risk-Controlling Prediction Sets (RCPS) [1], Conformal Risk Control (CRC) [2] and Learn Then Test (LTT) [3]. The difference between these methods is the way the conformity scores are computed. diff --git a/doc/theoretical_description_regression.rst b/doc/theoretical_description_regression.rst index a5682720..55d6800d 100644 --- a/doc/theoretical_description_regression.rst +++ b/doc/theoretical_description_regression.rst @@ -6,6 +6,13 @@ Theoretical Description ####################### +Note: in theoretical parts of the documentation, we use the following terms employed in the scientific literature: + +- :math:`alpha` is equivalent to :math:`1-confidence_level`. It can be seen as a *risk level* +- *calibrate* and *calibration*, are equivalent to *conformalize* and *conformalization*. + +— + The methods in `mapie_v1.regression` use various resampling methods based on the jackknife strategy recently introduced by Foygel-Barber et al. (2020) [1]. diff --git a/doc/v1_migration_guide.rst b/doc/v1_migration_guide.rst index a92b279e..ca93b227 100644 --- a/doc/v1_migration_guide.rst +++ b/doc/v1_migration_guide.rst @@ -40,11 +40,11 @@ In v0.9, data splitting is handled by MAPIE. In v1, the data splitting is left to the user, with the exception of cross-conformal methods (``CrossConformalRegressor``). The user can split the data into training, conformalization, and test sets using scikit-learn's ``train_test_split`` or other methods. -Step 2 & 3: Model training and conformalization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -In v0.9, the ``fit`` method handled both model training and conformalization. +Step 2 & 3: Model training and conformalization (ie: calibration) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +In v0.9, the ``fit`` method handled both model training and calibration. -In v1.0: MAPIE separates between the training and conformalization: +In v1.0: MAPIE separates between training and calibration. We decided to name the *calibration* step *conformalization*, to avoid confusion with probability calibration. - ``.fit()`` method: - In v1, ``fit`` only trains the model on training data, without handling conformalization.