Simplifying using pre-trained COSEM models

[jdeschamps]

Hi there,

Finally found the time to look around and start playing with dacapo! I have a few questions and remarks regarding the COSEM models, some of which might just be born out of missing something. 😄

Fine-tuning COSEM requires knowing architecture and task

The documentation lists available COSEM models, but not their architecture parameters. Looking into the model repo (example for setup04), the parameters are different from those of a CNNectomeUNetConfig.

It would make sense to me to also list the parameters somewhere on the dacapo documentation, or better yet, provide ready-to-use convenience configuration.

Example:

from dacapo.experiments.architectures import COSEM_1820500
architecture_config = COSEM_1820500()

More generally, the example from the documentation leaves out all the configuration that are necessary to create a RunConfig. But not knowing anything but the setup04 and 1820500 makes it unlikely that people will know how to instantiate the correct configurations!

Predicting with pre-trained COSEM model requires RunConfig

This point probably has to do with the fundamental aim of dacapo, but if one wants to simply load a model and predict on some images, the overhead is quite large! This notebook seems to do that (predict using fine-tuned), and a lot of the lines in the notebooks will be irrelevant to users.

Simplifying using pre-trained models

In my opinion, all this could be made more intuitive by (or/and):

• Providing ready-made RunConfig for the pre-trained models. These would allow simply predicting using dacapo.predict.predict. Obviously, this would not be usable for training and would need to be documented. Alternatively, the sub-configurations could be retrieved that way, giving users the opportunity to not use the train and datasplit configs from the pre-trained model.
• Providing in the documentation the architecture parameters and tasks used to train the various pre-trained models would also allow a better understanding of the models, and help users set up the configurations correctly when fine-tuning.
• Providing a predict function that does not require a full RunConfig. I haven't looked too deep in the code yet, so I am not sure how feasible it is to only pass the relevant configurations (starter, architecture, etc..). This could be alleviated by using the ready-made config.

Apologies if this all stem from missing some important points!

Thanks!!

[mzouink]

Hi @jdeschamps !
Thanks a bunch for diving into dacapo and sharing your thoughts! 😄

COSEM Architecture Parameters

I agree, having the architecture details in the docs would be super helpful. Good news— @pattonw is currently working on importing/exporting bioimage models' architectures and weights. Once that’s done, we can re-export all the CellMap models with the necessary info, which should fix the issue you mentioned.

Prediction with Zero-Shot or New Datasets

Predicting with zero-shot or on new datasets is definitely a bit tricky right now. I love both of your ideas:

• Run for Pretrained Models: This would make it easier for users to use pretrained models without a lot of setup.
• Predict Without a Run: Allowing predictions without needing a full RunConfig would make things a lot smoother and less cumbersome.

I think we can make both of these options available to improve the user experience.