[ENH] Bayesian Linear Regression using Normal Conjugate Prior

[meraldoantonio]

What does this implement/fix? Explain your changes.

Implementing a new Bayesian linear regression estimator module using normal conjugate prior
with known noise variance.

Does your contribution introduce a new dependency? If yes, which one?

No

What should a reviewer concentrate their feedback on?

Correctness of implementation

Did you add any tests for the change?

No

Any other comments?

No

PR checklist

For all contributions

• [] I've added myself to the list of contributors with any new badges I've earned :-)
  How to: add yourself to the all-contributors file in the skpro root directory (not the CONTRIBUTORS.md). Common badges: code - fixing a bug, or adding code logic. doc - writing or improving documentation or docstrings. bug - reporting or diagnosing a bug (get this plus code if you also fixed the bug in the PR).maintenance - CI, test framework, release.
  See here for full badge reference
• The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.

For new estimators

• I've added the estimator to the API reference - in docs/source/api_reference/taskname.rst, follow the pattern.
• I've added one or more illustrative usage examples to the docstring, in a pydocstyle compliant Examples section.
  (https://www.sktime.net/en/latest/developer_guide/dependencies.html#adding-a-soft-dependency).

[meraldoantonio]

Hi @fkiraly,

Thanks for the review! I’ve restructured the folders to improve organization. Now, we have a bayesian subfolder containing both of our new estimators. Similarly, the example notebooks have been moved into a bayesian subfolder inside the examples directory.

For this conjugate estimator, I haven’t yet enabled the setting of mu priors for different coefficients, as implementing this will require a more complex update formula. This will take a bit more time to complete.

Looking ahead, I suggest we create at least two versions of the conjugate estimator:

1. Simple Version (the current one): Assumes the prior mu is all zeros, and the covariance matrix is isotropic.
2. Flexible Version: Allows users to set both the priors mu and the covariance matrix covar.
3. A longer term goal will be to create a Bayesian Linear Regressor for which both precision and mu are unknown; this will require us to use Normal-Wishart conjugate model. However, this probability distribution is not yet available in scipy (it's coming in the next version).

My next goal is to work on these later versions. Once it’s ready, we could rename the current estimator to something like SimpleBayesianConjugateLinearRegressor to distinguish it more clearly.

[fkiraly]

the BayesianLinearRegressor is in another folder, so will not show up here.

[meraldoantonio]

Modified the file

[fkiraly]

do you want to add math to make this precise? Also, the handling of the intercept should be discussed.

[meraldoantonio]

I added some high-level explanation on normal conjugacy. I tried adding the math but it looks too cluttered.
Also, added information on the handling of intercept

[fkiraly]

Great, only some smaller comments:

• the BayesianLinearRegressor is not in the folder linked to in the API ref
• non-blocking suggestions about improving the docstring

Non-blocking, to think about:

The notebook could be about Bayesian linear regression, not just about the conjugate approach. It would be informative to separately showcase the Bayes rule, and different ways to estimate/approximate the actual posterior. Conjucate and MCMC are two of a longer list (other strategies are variational and approximate).

[fkiraly]

Excellent work! My comments are mainly on documentation, PR structure, nothing content-wise.

• the notebooks should be reconciled with PR [DOC] Adding a Notebook companion to BayesianMCMCLinearRegressor, Restructuring Estimator Folder #480, otherwise I see some inconsistencies that could cause merge headaches. I would suggest to split off the notebooks in a separate PR, with dependency graph either 500, 480 -> new PR, or 500 -> 480 -> new PR (left side of arrow gets merged earlier).
• in the docstring, I think it is essential to say what exactly the model assumptions are, a little bit of math would be great there, and how these map onto python parameters.
• the changes in [DOC] Adding a Notebook companion to BayesianMCMCLinearRegressor, Restructuring Estimator Folder #480 are inconsistent with the directory structure here - I would recommend to make all submodues of bayesian private, as long as we are not certain about the exact end state desired.