Non-mp predict support [WIP] #1

surajpaib · 2024-11-21T13:17:26Z

No description provided.

surajpaib · 2024-11-21T13:18:31Z

I've looked at the code and added non-MP support in the data-processing and prediction pipeline. I'm yet to test this - i haven't really setup nnunet workflow yet. Could you test it and let me know if it looks good?

LennyN95 · 2024-11-21T15:26:07Z

Nice; I set up a test env and am looking into it.

LennyN95 · 2024-11-21T15:34:38Z

FYI; we get the following assertion error.

MHub Error Log

[EXTERNAL|21.11.24 16:16:07]: Running preprocessing in non-multiprocessing mode
[EXTERNAL|21.11.24 16:16:07]: 
[EXTERNAL|21.11.24 16:16:07]: Predicting VOLUME_001:
[EXTERNAL|21.11.24 16:16:07]: perform_everything_on_device: True
[EXTERNAL|21.11.24 16:16:07]: Traceback (most recent call last):
[EXTERNAL|21.11.24 16:16:07]: File "/app/.venv/bin/nnUNetv2_predict", line 8, in <module>
[EXTERNAL|21.11.24 16:16:07]: sys.exit(predict_entry_point())
[EXTERNAL|21.11.24 16:16:07]: ^^^^^^^^^^^^^^^^^^^^^
[EXTERNAL|21.11.24 16:16:07]: File "/nnunet-src/nnunetv2/inference/predict_from_raw_data.py", line 906, in predict_entry_point
[EXTERNAL|21.11.24 16:16:07]: predictor.predict_from_files(args.i, args.o, save_probabilities=args.save_probabilities,
[EXTERNAL|21.11.24 16:16:07]: File "/nnunet-src/nnunetv2/inference/predict_from_raw_data.py", line 258, in predict_from_files
[EXTERNAL|21.11.24 16:16:07]: return self.predict_from_data_iterator(data_iterator, save_probabilities, num_processes_segmentation_export)
[EXTERNAL|21.11.24 16:16:07]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[EXTERNAL|21.11.24 16:16:07]: File "/nnunet-src/nnunetv2/inference/predict_from_raw_data.py", line 378, in predict_from_data_iterator
[EXTERNAL|21.11.24 16:16:07]: prediction = self.predict_logits_from_preprocessed_data(data).cpu()
[EXTERNAL|21.11.24 16:16:07]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[EXTERNAL|21.11.24 16:16:07]: File "/nnunet-src/nnunetv2/inference/predict_from_raw_data.py", line 532, in predict_logits_from_preprocessed_data
[EXTERNAL|21.11.24 16:16:07]: prediction = self.predict_sliding_window_return_logits(data).to('cpu')
[EXTERNAL|21.11.24 16:16:07]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[EXTERNAL|21.11.24 16:16:07]: File "/nnunet-src/nnunetv2/inference/predict_from_raw_data.py", line 655, in predict_sliding_window_return_logits
[EXTERNAL|21.11.24 16:16:07]: assert isinstance(input_image, torch.Tensor)
[EXTERNAL|21.11.24 16:16:07]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[EXTERNAL|21.11.24 16:16:07]: AssertionError

LennyN95 · 2024-11-21T15:36:29Z

Okay test is running; have no results yet but the gpu spins up 🔥
fingers crossed :D

LennyN95 · 2024-11-21T17:28:40Z

Okay, so the good news is, it ran. Logs below.
Looks like the prediction is not saved though (output is empty), should be simple to fix.

nnUNet_results=/app/nnunet_out uv run nnUNetv2_predict \
   -i /app/tmp/2b238249-d307-4757-b9f2-9541f4444f18 \
   -o /app/nnunet_out -d Dataset007_Nodules -c 3d_fullres

Output

nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.

#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################

/nnunet-src/nnunetv2/inference/predict_from_raw_data.py:84: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(join(model_training_output_dir, f'fold_{f}', checkpoint_name),
/nnunet-src/nnunetv2/utilities/plans_handling/plans_handler.py:37: UserWarning: Detected old nnU-Net plans format. Attempting to reconstruct network architecture parameters. If this fails, rerun nnUNetv2_plan_experiment for your dataset. If you use a custom architecture, please downgrade nnU-Net to the version you implemented this or update your implementation + plans.
  warnings.warn("Detected old nnU-Net plans format. Attempting to reconstruct network architecture "
There are 1 cases in the source folder
I am process 0 out of 1 (max process ID is 0, we start counting with 0!)
There are 1 cases that I would like to predict
Running preprocessing in non-multiprocessing mode
Running preprocessing in non-multiprocessing mode
nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.
nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.
nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.

Predicting VOLUME_001:
perform_everything_on_device: True
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:20<00:00,  4.04it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
sending off prediction to background worker for resampling and export
done with VOLUME_001

This might be our problem:

sending off prediction to background worker for resampling and export

nnUNet/nnunetv2/inference/predict_from_raw_data.py

Line 384 in 6cb4e7a

print('sending off prediction to background worker for resampling and export')

surajpaib · 2024-11-21T17:47:52Z

Ok was just able to check the code. You'll have to set both nps and npp to 0. It seems like thats where that
print('sending off prediction to background worker for resampling and export') comes from

LennyN95 · 2024-11-21T17:53:26Z

Now, the proper test with -npp 0 -nps 0:

nnUNet_results=/app/nnunet_out uv run nnUNetv2_predict \
   -i /app/tmp/2b238249-d307-4757-b9f2-9541f4444f18 \
   -o /app/nnunet_out -d Dataset007_Nodules -c 3d_fullres \
   -npp 0 -nps 0

Output

nnUNet_raw is not defined and nnU-Net can only be used on data for which preprocessed files are already present on your system. nnU-Net cannot be used for experiment planning and preprocessing like this. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up properly.
nnUNet_preprocessed is not defined and nnU-Net can not be used for preprocessing or training. If this is not intended, please read documentation/setting_up_paths.md for information on how to set this up.

#######################################################################
Please cite the following paper when using nnU-Net:
Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
#######################################################################

/nnunet-src/nnunetv2/inference/predict_from_raw_data.py:84: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(join(model_training_output_dir, f'fold_{f}', checkpoint_name),
/nnunet-src/nnunetv2/utilities/plans_handling/plans_handler.py:37: UserWarning: Detected old nnU-Net plans format. Attempting to reconstruct network architecture parameters. If this fails, rerun nnUNetv2_plan_experiment for your dataset. If you use a custom architecture, please downgrade nnU-Net to the version you implemented this or update your implementation + plans.
  warnings.warn("Detected old nnU-Net plans format. Attempting to reconstruct network architecture "
There are 1 cases in the source folder
I am process 0 out of 1 (max process ID is 0, we start counting with 0!)
There are 1 cases that I would like to predict
Running in non-multiprocessing mode
Running preprocessing in non-multiprocessing mode
Running preprocessing in non-multiprocessing mode

Predicting VOLUME_001:
perform_everything_on_device: True
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:20<00:00,  4.03it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 567/567 [02:19<00:00,  4.07it/s]
done with VOLUME_001

root@07144416098b:/app# ls nnunet_out/
Dataset007_Nodules  VOLUME_001.nii.gz  dataset.json  plans.json  predict_from_raw_data_args.json

New environment variables: - nnUNet_npp - nnUNet_nps Default values remain unchanged, cli parameter -npp and -nps overwrite environment variables if set.

Non-mp predict support [WIP]

69fd9a7

add torchification

a595f64

update preprocess_fromfiles_noqueue

6cb4e7a

dev-only: remove enforced num_processes = 1

b570e8a

LennyN95 and others added 2 commits November 21, 2024 19:28

add env to overwrite npp and nps

3628055

New environment variables: - nnUNet_npp - nnUNet_nps Default values remain unchanged, cli parameter -npp and -nps overwrite environment variables if set.

Fix preprocessor initialization order

6bcec21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Non-mp predict support [WIP] #1

Non-mp predict support [WIP] #1

surajpaib commented Nov 21, 2024

surajpaib commented Nov 21, 2024

LennyN95 commented Nov 21, 2024

LennyN95 commented Nov 21, 2024 •

edited

Loading

LennyN95 commented Nov 21, 2024

LennyN95 commented Nov 21, 2024 •

edited

Loading

surajpaib commented Nov 21, 2024

LennyN95 commented Nov 21, 2024 •

edited

Loading

Non-mp predict support [WIP] #1

Are you sure you want to change the base?

Non-mp predict support [WIP] #1

Conversation

surajpaib commented Nov 21, 2024

surajpaib commented Nov 21, 2024

LennyN95 commented Nov 21, 2024

LennyN95 commented Nov 21, 2024 • edited Loading

LennyN95 commented Nov 21, 2024

LennyN95 commented Nov 21, 2024 • edited Loading

surajpaib commented Nov 21, 2024

LennyN95 commented Nov 21, 2024 • edited Loading

LennyN95 commented Nov 21, 2024 •

edited

Loading

LennyN95 commented Nov 21, 2024 •

edited

Loading

LennyN95 commented Nov 21, 2024 •

edited

Loading