Possible bug in ablation

[Michael190502]

Hi,

I think I found a possible bug in the ablation file.

Firstly I think after the following line

DeepCAVE/deepcave/evaluators/ablation.py

Line 128 in 58d6801

inc_cost, _ = self._model.predict(np.array([incumbent_encode]))

inc_cost = inc_cost[0] is missing (which is there for def_cost). This though only affects the comparison between these two values a few lines later I think.

Secondly the ablation path analysis did not work correctly for me when I had only categorical hyperparameters. I traced the problem to the ablation file where "encode_config" is used instead of "get_array". This resulted, in my case at least, in an encoding normalised to [0,1] instead of the number of the choice of the categorical hyperparameter. This resulted in the predicted performance of the default configuration being equal to the predicted performance of the incumbent.

It worked correctly when I replaced "encode_config" with "get_array".

Please let me know if this is something specific to my setup or if I did something wrong. I will append an image comparing the encoding of "get_array" to "encode_config" for clarity.

[mwever]

Hi @Michael190502 ,

thanks for reporting this issue, we will have a look at it!

[KrissiHub]

Hi @Michael190502 ,

I am currently looking into the issue, could you provide a minimal code example/snippet so I can recreate the problem?

[Michael190502]

Hi @KrissiHub,

as I dont have a prepared run at hand I utilized the example for ablation from examples/api/ablation_paths.py to create an example showing the difference. To demonstrate the difference I made the following simple changes:

• changed the loaded run in line 16 of the examples/api/ablation_paths.py file to "logs/DeepCAVE/digits_sklearn/run_1" to include a categorical variable
• changed the default value of the activation categorical variable in "logs/DeepCAVE/digits_sklearn/run_1/configspace.json" from logistic to tanh to avoid having the encoding be zero.
• Added print(default_encode[0], self.default_config.get_array()[0]) after line 123 in deepcave/evaluators/ablation.py to display the difference in encoding

I hope this suffices as an example even if it is not a code snippet.

Afterwards you could replace all encode_config with get_array to see the difference in the computed ablation paths. I included the two improvement plots before and after changing the encoding. (The first one is after)