Add Adaptive Conformal Inference method for Time Series #341

thibaultcordier · 2023-08-11T09:47:02Z

Description

Adaptive Conformal Inference for time series with general dependency.

Gibbs, I., & Candes, E. (2021). Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems, 34, 1660-1672.
Zaffran, M., Féron, O., Goude, Y., Josse, J., & Dieuleveut, A. (2022, June). Adaptive conformal predictions for time series. In International Conference on Machine Learning (pp. 25834-25866). PMLR.

Fixes #334
Fixes #390

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

Test that ACI method works.
Test that infinite prediction intervals can be produced in regressor classes.

Checklist

I have read the contributing guidelines
I have updated the HISTORY.rst and AUTHORS.rst files
Linting passes successfully : make lint
Typing passes successfully : make type-check
Unit tests pass successfully : make tests
Coverage is 100% : make coverage
Documentation builds successfully : make doc

Optimisation méthode ACI, update chaque pas du gamma

thibaultcordier

@QMaphan, I offer you a quick first review for your contribution for scores applied to regression. All suggestions will improve the style and visibility of the code.

I'd also suggest adapting your score so that it's called with the following signature: (y_true, y_interval) instead of (y_true, y_low, y_up) .

mapie/metrics.py

HISTORY.rst

mapie/metrics.py

mapie/tests/test_metrics.py

mapie/regression/time_series_regression.py

vincentblot28 · 2023-12-21T13:30:53Z

mapie/regression/time_series_regression.py

+            alpha = alpha_np
+        return alpha
+
+    def adapt_conformal_inference(


Suggested change

def adapt_conformal_inference(

def _adapt_conformal_inference(

Why not, but then we'd have to replace partial_fit with _partial_fit.

no because it would be a breaking changes (cf my comment about partial_fit)

In this case, I prefer to leave it as is.

vincentblot28 · 2023-12-21T13:43:09Z

mapie/regression/time_series_regression.py

@@ -96,45 +112,27 @@ def _relative_conformity_scores(
        )
        return scores

-    def fit(
+    def partial_fit(


If I well understood your PR, now if method == "enbpi" then you can either call .update or .partial_fit. For now maybe we can keep both, but if we want to keep only the .update we could add a DepreciationWarning to partial_fit

For the moment, I have no intention of withdrawing it.

vincentblot28 · 2023-12-21T14:03:46Z

mapie/regression/time_series_regression.py

+        self,
+        X: ArrayLike,
+        y: ArrayLike,
+        ensemble: bool = False,


shoudnl't we keep the same value of ensemble for all the class ?

I agree, but we also have the same problem with sym in this case. I'm doing this to reproduce the results.

mapie/regression/time_series_regression.py

mapie/tests/test_time_series_regression.py

Co-authored-by: Vincent Blot <[email protected]>

…classes

vincentblot28 · 2023-12-22T17:02:04Z

mapie/regression/quantile_regression.py

@@ -633,6 +633,8 @@ def predict(
        X: ArrayLike,
        ensemble: bool = False,
        alpha: Optional[Union[float, Iterable[float]]] = None,
+        optimize_beta: bool = False,


why this change ?

Because of function signature: https://stackoverflow.com/questions/51003146/python-3-6-signature-of-method-incompatible-with-super-type-class

LacombeLouis

Some last small changes but overall this PR will go a long way for MAPIE!

examples/regression/3-scientific-articles/plot_zaffran2022_comparison.py

mapie/conformity_scores/conformity_scores.py

Ajout ACI

237f8a6

thibaultcordier linked an issue Aug 11, 2023 that may be closed by this pull request

Adaptive Conformal Predictions for Time Series #334

Closed

thibaultcordier added the enhancement label Aug 11, 2023

thibaultcordier assigned rcolliaux Aug 11, 2023

remi colliaux and others added 24 commits August 11, 2023 11:49

FIX : Correction passages lint, type check.

21e8e80

Optimisation méthode ACI, update chaque pas du gamma

FIX: fix type-check and add warning

96d47d0

UPD: format code and update metainfo

8647ec1

FIX: scikit-learn error

f6f171e

FIX: take account conformity score function

1897b91

FIX: redefine previous tests for beta optimization

e5e0400

ajout metrics cwc

20b49e8

FIX: fix type-check

0c18c9f

FIX: fix type check

ec0b07f

FIX= fix test_metrics

e13771d

FIX: fix test_metrics

6b50c65

FIX: fix type_check

34ff553

fix: correction erreur typing

35e620f

FIX: fix static type and metrics

3db5ad2

FIX: fix linting

24ae12c

FIX: fix typing

21448bf

FIX: pytest

4bc21f8

FIX: pytest

2f839e1

FIX: fix pytest

f87c4e7

FIX: pytest

69dbb27

FIX: Pytest

8894bf5

FIX: pytest

9cbe0f4

FIX: correction coverage utils and test_metrics

06c41d5

FIX: fix coverage test

c97cf57

thibaultcordier commented Aug 29, 2023

View reviewed changes

vincentblot28 reviewed Dec 21, 2023

View reviewed changes

FIX: hyperlinks

1e1911f

vincentblot28 reviewed Dec 21, 2023

View reviewed changes

Thibault Cordier added 3 commits December 21, 2023 15:06

UPD: else condition in update

4bf1da2

UPD: change gamma value error message

e8c2ba4

UPD: change name of alpha method

176bbaf

vincentblot28 reviewed Dec 21, 2023

View reviewed changes

mapie/regression/time_series_regression.py Show resolved Hide resolved

vincentblot28 reviewed Dec 21, 2023

View reviewed changes

mapie/tests/test_time_series_regression.py Show resolved Hide resolved

Thibault Cordier and others added 9 commits December 21, 2023 16:10

UPD: docstring

e83389d

UPD: modify the user warning with a better function call

fc0522f

UPD: add DeprecationWarning for partial_fit

5a1c3ab

Co-authored-by: Vincent Blot <[email protected]>

FIX: add warnings and better indent

b4c3029

FIX: remove useless parameter

a0587a0

UPD: change docstring and section names

765b23f

UPD: allow infinite prediction intervals to be produced in regressor …

a2fbd3a

…classes

UPD: ACI doctring

505a500

UPD: add plot compared results

6b56953

vincentblot28 reviewed Dec 22, 2023

View reviewed changes

thibaultcordier requested review from vincentblot28 and LacombeLouis January 1, 2024 17:03

LacombeLouis approved these changes Jan 3, 2024

View reviewed changes

examples/regression/3-scientific-articles/plot_zaffran2022_comparison.py Outdated Show resolved Hide resolved

mapie/conformity_scores/conformity_scores.py Show resolved Hide resolved

thibaultcordier added 2 commits January 3, 2024 09:32

FIX: shorter hyperlink

e6e7529

FIX: remove hyperlink

116eb2d

LacombeLouis approved these changes Jan 3, 2024

View reviewed changes

thibaultcordier merged commit 941ca35 into master Jan 3, 2024
8 checks passed

github-actions bot mentioned this pull request Jun 24, 2024

Monthly issue metrics report for opened issues and prs thibaultcordier/MAPIE#11

Open

Valentin-Laurent deleted the 334-adaptive-conformal-predictions-for-time-series branch November 12, 2024 13:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Adaptive Conformal Inference method for Time Series #341

Add Adaptive Conformal Inference method for Time Series #341

thibaultcordier commented Aug 11, 2023 •

edited

Loading

thibaultcordier left a comment

vincentblot28 Dec 21, 2023

thibaultcordier Dec 21, 2023

vincentblot28 Dec 21, 2023

thibaultcordier Dec 21, 2023

vincentblot28 Dec 21, 2023

thibaultcordier Dec 21, 2023

vincentblot28 Dec 21, 2023

thibaultcordier Dec 21, 2023

vincentblot28 Dec 22, 2023

thibaultcordier Dec 22, 2023

LacombeLouis left a comment

	def adapt_conformal_inference(
	def _adapt_conformal_inference(

Add Adaptive Conformal Inference method for Time Series #341

Add Adaptive Conformal Inference method for Time Series #341

Conversation

thibaultcordier commented Aug 11, 2023 • edited Loading

Description

Type of change

How Has This Been Tested?

Checklist

thibaultcordier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LacombeLouis left a comment

Choose a reason for hiding this comment

thibaultcordier commented Aug 11, 2023 •

edited

Loading