Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vertebra_pipeline errors #1770

Open
linfanluntan opened this issue Oct 30, 2024 · 7 comments
Open

vertebra_pipeline errors #1770

linfanluntan opened this issue Oct 30, 2024 · 7 comments

Comments

@linfanluntan
Copy link

Describe the bug
I'm trying to do vertebra segmentation using the pre-trained models, localization_spine and localization_vertebra are working, however, vertebra_pipeline gives errors.

Server logs
(monailabel02) c:>monailabel start_server --app MLapp008/radiology --studies ML_dataset/khoury --conf models all --conf bundles wholeBody_ct_segmentation,pancreas_ct_dints_segmentation --log_level debug
Using PYTHONPATH=C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;
""
C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\ignite\handlers\checkpoint.py:17: DeprecationWarning: TorchScript support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the torch.compile optimizer instead.
from torch.distributed.optim import ZeroRedundancyOptimizer
[2024-10-30 06:43:08,350] [28940] [MainThread] [INFO] (main:285) - USING:: version = False
[2024-10-30 06:43:08,350] [28940] [MainThread] [INFO] (main:285) - USING:: app = C:\MLapp008\radiology
[2024-10-30 06:43:08,352] [28940] [MainThread] [INFO] (main:285) - USING:: studies = C:\ML_dataset\khoury
[2024-10-30 06:43:08,352] [28940] [MainThread] [INFO] (main:285) - USING:: verbose = INFO
[2024-10-30 06:43:08,353] [28940] [MainThread] [INFO] (main:285) - USING:: conf = [['models', 'all'], ['bundles', 'wholeBody_ct_segmentation,pancreas_ct_dints_segmentation']]
[2024-10-30 06:43:08,354] [28940] [MainThread] [INFO] (main:285) - USING:: host = 0.0.0.0
[2024-10-30 06:43:08,354] [28940] [MainThread] [INFO] (main:285) - USING:: port = 8000
[2024-10-30 06:43:08,355] [28940] [MainThread] [INFO] (main:285) - USING:: uvicorn_app = monailabel.app:app
[2024-10-30 06:43:08,356] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile = None
[2024-10-30 06:43:08,356] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_certfile = None
[2024-10-30 06:43:08,357] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile_password = None
[2024-10-30 06:43:08,357] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_ca_certs = None
[2024-10-30 06:43:08,358] [28940] [MainThread] [INFO] (main:285) - USING:: workers = None
[2024-10-30 06:43:08,359] [28940] [MainThread] [INFO] (main:285) - USING:: limit_concurrency = None
[2024-10-30 06:43:08,359] [28940] [MainThread] [INFO] (main:285) - USING:: access_log = False
[2024-10-30 06:43:08,360] [28940] [MainThread] [INFO] (main:285) - USING:: root_path =
[2024-10-30 06:43:08,361] [28940] [MainThread] [INFO] (main:285) - USING:: log_level = debug
[2024-10-30 06:43:08,362] [28940] [MainThread] [INFO] (main:285) - USING:: log_config = None
[2024-10-30 06:43:08,362] [28940] [MainThread] [INFO] (main:285) - USING:: dryrun = False
[2024-10-30 06:43:08,363] [28940] [MainThread] [INFO] (main:285) - USING:: action = start_server
[2024-10-30 06:43:08,364] [28940] [MainThread] [INFO] (main:296) -
Allow Origins: ['']
[2024-10-30 06:43:09,418] [28940] [MainThread] [INFO] (uvicorn.error:82) - Started server process [28940]
[2024-10-30 06:43:09,418] [28940] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup.
App Init...
[2024-10-30 06:43:09,419] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: C:\MLapp008\radiology; studies: C:\ML_dataset\khoury; conf: {'models': 'all', 'bundles': 'wholeBody_ct_segmentation,pancreas_ct_dints_segmentation'}
[2024-10-30 06:43:09,566] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2024-10-30 06:43:09,576] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2024-10-30 06:43:09,577] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2024-10-30 06:43:09,577] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2024-10-30 06:43:09,578] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2024-10-30 06:43:09,579] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2024-10-30 06:43:09,580] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2024-10-30 06:43:09,581] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2024-10-30 06:43:09,582] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2024-10-30 06:43:09,582] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.sw_fastedit.SWFastEditConfig'>
[2024-10-30 06:43:09,583] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit
[2024-10-30 06:43:10,175] [28940] [MainThread] [INFO] (lib.configs.deepedit:141) - EPISTEMIC Enabled: False; Samples: 5
[2024-10-30 06:43:10,175] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: deepgrow_2d => lib.configs.deepgrow_2d.Deepgrow2D
BasicUNet features: (32, 64, 128, 256, 512, 32).
[2024-10-30 06:43:10,219] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: deepgrow_3d => lib.configs.deepgrow_3d.Deepgrow3D
BasicUNet features: (32, 64, 128, 256, 512, 32).
[2024-10-30 06:43:10,329] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: localization_spine => lib.configs.localization_spine.LocalizationSpine
[2024-10-30 06:43:10,425] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: localization_vertebra => lib.configs.localization_vertebra.LocalizationVertebra
[2024-10-30 06:43:10,520] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2024-10-30 06:43:10,613] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation_spleen => lib.configs.segmentation_spleen.SegmentationSpleen
[2024-10-30 06:43:10,644] [28940] [MainThread] [INFO] (lib.configs.segmentation_spleen:79) - EPISTEMIC Enabled: False; Samples: 5
[2024-10-30 06:43:10,644] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation_vertebra => lib.configs.segmentation_vertebra.SegmentationVertebra
[2024-10-30 06:43:10,740] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: sw_fastedit => lib.configs.sw_fastedit.SWFastEditConfig
{'tumor': 0, 'background': 1}
Downloading from C:\MLapp008\radiology\model\pretrained_sw_fastedit.pt
[2024-10-30 06:43:11,035] [28940] [MainThread] [INFO] (main:98) - +++ Using Models: ['deepedit', 'deepgrow_2d', 'deepgrow_3d', 'localization_spine', 'localization_vertebra', 'segmentation', 'segmentation_spleen', 'segmentation_vertebra', 'sw_fastedit']
[2024-10-30 06:43:11,419] [28940] [MainThread] [INFO] (monailabel.utils.others.generic:304) - +++ Adding Bundle from Zoo: wholeBody_ct_segmentation => C:\MLapp008\radiology\model\wholeBody_ct_segmentation
[2024-10-30 06:43:11,420] [28940] [MainThread] [INFO] (monailabel.utils.others.generic:304) - +++ Adding Bundle from Zoo: pancreas_ct_dints_segmentation => C:\MLapp008\radiology\model\pancreas_ct_dints_segmentation
[2024-10-30 06:43:11,421] [28940] [MainThread] [INFO] (monailabel.utils.others.generic:364) - +++ Using Bundle Models: ['wholeBody_ct_segmentation', 'pancreas_ct_dints_segmentation']
[2024-10-30 06:43:11,422] [28940] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: C:\ML_dataset\khoury
[2024-10-30 06:43:11,423] [28940] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '*.xml']
[2024-10-30 06:43:11,437] [28940] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0
[2024-10-30 06:43:11,437] [28940] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD)
[2024-10-30 06:43:11,447] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x000001FC2DD5F070>
[2024-10-30 06:43:11,447] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x000001FC2E7AFD30>
[2024-10-30 06:43:11,448] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepgrow_2d => <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E837520>
[2024-10-30 06:43:11,448] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepgrow_3d => <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E8375E0>
[2024-10-30 06:43:11,448] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: localization_spine => <lib.infers.localization_spine.LocalizationSpine object at 0x000001FC2E837880>
[2024-10-30 06:43:11,449] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: localization_vertebra => <lib.infers.localization_vertebra.LocalizationVertebra object at 0x000001FC2E837430>
[2024-10-30 06:43:11,449] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x000001FC2E837040>
[2024-10-30 06:43:11,449] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation_spleen => <lib.infers.segmentation_spleen.SegmentationSpleen object at 0x000001FC2E8372B0>
[2024-10-30 06:43:11,450] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation_vertebra => <lib.infers.segmentation_vertebra.SegmentationVertebra object at 0x000001FC2E837250>
[2024-10-30 06:43:11,460] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: sw_fastedit => <lib.infers.sw_fastedit.SWFastEdit object at 0x000001FC2E837160>
[2024-10-30 06:43:11,577] [28940] [MainThread] [INFO] (main:137) - +++ Adding Bundle Inferer:: wholeBody_ct_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC3914F490>
You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
2024-10-30 06:43:11.609109 - Length of input patch is recommended to be a multiple of 32.
[2024-10-30 06:43:13,635] [28940] [MainThread] [INFO] (main:137) - +++ Adding Bundle Inferer:: pancreas_ct_dints_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC2E837220>
[2024-10-30 06:43:13,636] [28940] [MainThread] [INFO] (main:194) - {'deepedit': <lib.infers.deepedit.DeepEdit object at 0x000001FC2DD5F070>, 'deepedit_seg': <lib.infers.deepedit.DeepEdit object at 0x000001FC2E7AFD30>, 'deepgrow_2d': <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E837520>, 'deepgrow_3d': <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E8375E0>, 'localization_spine': <lib.infers.localization_spine.LocalizationSpine object at 0x000001FC2E837880>, 'localization_vertebra': <lib.infers.localization_vertebra.LocalizationVertebra object at 0x000001FC2E837430>, 'segmentation': <lib.infers.segmentation.Segmentation object at 0x000001FC2E837040>, 'segmentation_spleen': <lib.infers.segmentation_spleen.SegmentationSpleen object at 0x000001FC2E8372B0>, 'segmentation_vertebra': <lib.infers.segmentation_vertebra.SegmentationVertebra object at 0x000001FC2E837250>, 'sw_fastedit': <lib.infers.sw_fastedit.SWFastEdit object at 0x000001FC2E837160>, 'wholeBody_ct_segmentation': <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC3914F490>, 'pancreas_ct_dints_segmentation': <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC2E837220>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x000001FC2EC3C190>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x000001FC2EB94610>, 'deepgrow_pipeline': <lib.infers.deepgrow_pipeline.InferDeepgrowPipeline object at 0x000001FC2EB945E0>, 'vertebra_pipeline': <lib.infers.vertebra_pipeline.InferVertebraPipeline object at 0x000001FC2EB945B0>}
[2024-10-30 06:43:13,637] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x000001FC2EB946D0>
[2024-10-30 06:43:13,638] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: deepgrow_2d => <lib.trainers.deepgrow.Deepgrow object at 0x000001FC2EB94700>
[2024-10-30 06:43:13,639] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: deepgrow_3d => <lib.trainers.deepgrow.Deepgrow object at 0x000001FC2EB94730>
[2024-10-30 06:43:13,639] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: localization_spine => <lib.trainers.localization_spine.LocalizationSpine object at 0x000001FC2EB94760>
[2024-10-30 06:43:13,640] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: localization_vertebra => <lib.trainers.localization_vertebra.LocalizationVertebra object at 0x000001FC2EB94790>
[2024-10-30 06:43:13,640] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x000001FC2EB947C0>
[2024-10-30 06:43:13,641] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation_spleen => <lib.trainers.segmentation_spleen.SegmentationSpleen object at 0x000001FC2EB947F0>
[2024-10-30 06:43:13,642] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation_vertebra => <lib.trainers.segmentation_vertebra.SegmentationVertebra object at 0x000001FC2EB94820>
[2024-10-30 06:43:13,644] [28940] [MainThread] [INFO] (main:221) - +++ Adding Bundle Trainer:: wholeBody_ct_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x000001FC2EB94670>
[2024-10-30 06:43:13,668] [28940] [MainThread] [INFO] (main:221) - +++ Adding Bundle Trainer:: pancreas_ct_dints_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x000001FC2EB94640>
[2024-10-30 06:43:13,669] [28940] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\FullerLabAI.cache\monailabel\sessions
[2024-10-30 06:43:13,669] [28940] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2024-10-30 06:43:13,670] [28940] [MainThread] [INFO] (monailabel.interfaces.app:469) - App Init - completed
[2024-10-30 06:43:13,670] [timeloop] [INFO] Starting Timeloop..
[2024-10-30 06:43:13,670] [28940] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2024-10-30 06:43:13,672] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000001FC2B8D1F70>
[2024-10-30 06:43:13,672] [28940] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000001FC2B8D1F70>
[2024-10-30 06:43:13,672] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2024-10-30 06:43:13,672] [28940] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2024-10-30 06:43:13,673] [28940] [MainThread] [INFO] (uvicorn.error:62) - Application startup complete.
[2024-10-30 06:43:13,673] [28940] [MainThread] [INFO] (uvicorn.error:214) - Uvicorn running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
[2024-10-30 06:43:35,581] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:44) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2024-10-30 06:43:35,582] [28940] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: Output Volume1; Weight: 582
[2024-10-30 06:43:35,587] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:60) - Next sample: {'id': 'Output Volume1', 'weight': 582, 'path': 'C:\ML_dataset\khoury\Output Volume1.nii.gz', 'ts': 1729741578, 'name': 'Output Volume1.nii.gz', 'strategy': {'random': {'ts': 1730288615, 'client_id': 'user-xyz'}}}
[2024-10-30 06:43:44,811] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:44) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2024-10-30 06:43:44,812] [28940] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: spleen_53; Weight: 591
[2024-10-30 06:43:44,816] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:60) - Next sample: {'id': 'spleen_53', 'weight': 591, 'path': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'ts': 1729716311, 'name': 'spleen_53.nii.gz', 'strategy': {'random': {'ts': 1730288624, 'client_id': 'user-xyz'}}}
[2024-10-30 06:43:57,815] [28940] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'vertebra_pipeline', 'image': 'spleen_53', 'device': 'NVIDIA RTX A6000:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'}
[2024-10-30 06:43:57,816] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda:0', 'model': 'vertebra_pipeline', 'image': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'Combines three stage for vertebra segmentation', 'image_path': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'pipeline_mode': True}
[2024-10-30 06:43:57,818] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2024-10-30 06:43:57,819] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode']
[2024-10-30 06:44:05,293] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 7.472; image: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 06:44:05,371] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.078; image: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 06:44:05,372] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 06:44:05,375] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.002; image: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 06:44:06,471] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 1.096; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:06,472] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:07,414] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.941; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:07,478] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.063; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:07,478] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None}
[2024-10-30 06:44:07,480] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_spine.pt
You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
[2024-10-30 06:44:14,611] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s)
[2024-10-30 06:44:14,611] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode', 'latencies', 'image_cached', 'pred']
[2024-10-30 06:44:14,613] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:14,622] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.009; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:14,627] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.005; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:19,571] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 4.943; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:19,572] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (BinaryMaskd): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 06:44:19,574] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 06:44:19,574] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 21.7583; Pre: 9.6610; Inferer: 7.1322; Invert: 0.0000; Post: 4.9640; Write: 0.0000
[2024-10-30 06:44:19,575] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['label_names', 'latencies', 'centroids']
[2024-10-30 06:44:19,576] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:24) - **************************** DATA ********************************************
[2024-10-30 06:44:19,577] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: device = cuda:0
[2024-10-30 06:44:19,577] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: model = vertebra_pipeline
[2024-10-30 06:44:19,578] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image = torch.Size([1, 512, 512, 156])
[2024-10-30 06:44:19,578] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_extension = .nrrd
[2024-10-30 06:44:19,579] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_dtype = uint8
[2024-10-30 06:44:19,580] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: client_id = user-xyz
[2024-10-30 06:44:19,580] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: description = Combines three stage for vertebra segmentation
[2024-10-30 06:44:19,581] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image_path = C:\ML_dataset\khoury\spleen_53.nii.gz
[2024-10-30 06:44:19,582] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: label = torch.Size([512, 512, 156])
[2024-10-30 06:44:19,584] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: pipeline_mode = True
[2024-10-30 06:44:19,585] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:38) - ******************************************************************************
monai.transforms.croppad.dictionary CropForegroundd.init:allow_smaller: Current default value of argument allow_smaller=True has been deprecated since version 1.2. It will be changed to allow_smaller=False in version 1.5.
[2024-10-30 06:44:19,586] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2024-10-30 06:44:19,587] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode']
[2024-10-30 06:44:19,588] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 06:44:19,589] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 06:44:19,881] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.292; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 06:44:19,882] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 06:44:19,993] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.11; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 06:44:20,030] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.036; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 06:44:20,038] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CropForegroundd): Time: 0.008; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32)
[2024-10-30 06:44:20,039] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': device(type='cpu'), 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None}
[2024-10-30 06:44:20,039] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_vertebra.pt
[2024-10-30 06:44:20,717] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - INV - Run Transform(s)
[2024-10-30 06:44:20,717] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - INV - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred']
[2024-10-30 06:44:21,064] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (CropForegroundd): Time: 0.346; image: torch.Size([25, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32)
[2024-10-30 06:45:01,703] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (Spacingd): Time: 40.6367; image: torch.Size([25, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32)
[2024-10-30 06:45:01,704] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s)
[2024-10-30 06:45:01,705] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred']
[2024-10-30 06:45:01,706] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32)
[2024-10-30 06:45:02,111] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.404; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32)
[2024-10-30 06:45:03,797] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 1.686; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 06:45:10,570] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 6.772; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 06:45:10,571] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 06:45:11,412] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (VertebraLocalizationSegmentation): Time: 0.838; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 06:45:11,412] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 51.8357; Pre: 0.4530; Inferer: 0.6380; Invert: 41.0257; Post: 9.7090; Write: 0.0000
[2024-10-30 06:45:11,413] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['centroids', 'label_names', 'latencies']
0it [00:00, ?it/s]
[2024-10-30 06:45:11,416] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST(P) - Run Transform(s)
[2024-10-30 06:45:11,416] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST(P) - Input Keys: ['model', 'image', 'device', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pred']
[2024-10-30 06:45:11,417] [28940] [MainThread] [ERROR] (uvicorn.error:411) - Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 406, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\fastapi\applications.py", line 1054, in call
await super().call(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\errors.py", line 187, in call
raise exc
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\errors.py", line 165, in call
await self.app(scope, receive, _send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\cors.py", line 85, in call
await self.app(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app
raise exc
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 735, in app
await route.handle(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 288, in handle
await self.app(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app
raise exc
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 73, in app
response = await f(request)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\fastapi\routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\fastapi\routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\endpoints\infer.py", line 206, in api_run_inference
return run_inference(background_tasks, model, image, session_id, params, file, label, output)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\endpoints\infer.py", line 172, in run_inference
result = instance.infer(request)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\interfaces\app.py", line 307, in infer
result_file_name, result_json = task(request)
File "C:\MLapp008\radiology\lib\infers\vertebra_pipeline.py", line 158, in call
data = run_transforms(data, [Restored(keys="pred", ref_image="image")], log_prefix="POST(P)", use_compose=False)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\interfaces\utils\transform.py", line 106, in run_transforms
data = t(data)
File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\transform\post.py", line 127, in call
current_size = result.shape[1:] if self.has_channel else result.shape
AttributeError: 'NoneType' object has no attribute 'shape'

To Reproduce
Steps to reproduce the behavior:

conda create -n monailabel02 python=3.9
conda activate monailabel02
monailabel apps --download --name radiology --output MLapp008
monailabel apps --download --name monaibundle --output MLapp008
monailabel start_server --app MLapp008/radiology --studies ML_dataset/khoury --conf models all --conf bundles wholeBody_ct_segmentation,pancreas_ct_dints_segmentation
or:
monailabel start_server --app MLapp008/radiology --studies ML_dataset/khoury --conf models all --conf bundles wholeBody_ct_segmentation,pancreas_ct_dints_segmentation --log_level debug
run Monailabel in 3D slicer
run vertebra_pipeline in auto segmentation
@linfanluntan
Copy link
Author

I modified vertebra_pipeline.py

def call(self, request):
start = time.time()
request.update({"image_path": request.get("image")})

device = name_to_device(request.get("device", "cuda"))
request["device"] = device

try:
    # Run first stage
    logging.info("Starting locate_spine stage")
    d1, r1, l1 = self.locate_spine(request)
    image = d1["image"]
    label = d1["pred"]
    logging.info(f"locate_spine completed. Image shape: {image.shape}, Label shape: {label.shape}")

    # Run second stage
    logging.info("Starting locate_vertebra stage")
    d2, r2, l2 = self.locate_vertebra(request, image, label)
    centroids = r2["centroids"]
    logging.info(f"locate_vertebra completed. Centroids: {centroids}")

    # Run third stage
    logging.info("Starting segment_vertebra stage")
    result_mask, l3 = self.segment_vertebra(request, image, centroids)
    logging.info(f"segment_vertebra completed. Result mask shape: {result_mask.shape if result_mask is not None else 'None'}")

    if result_mask is None:
        raise ValueError("segment_vertebra returned None result_mask")

    # Finalize the mask/result
    data = copy.deepcopy(request)
    data.update({"pred": result_mask, "image": image})
    
    logging.info("Starting post-processing")
    data = run_transforms(data, [Restored(keys="pred", ref_image="image")], log_prefix="POST(P)", use_compose=False)
    
    if "pred" not in data or data["pred"] is None:
        raise ValueError("Post-processing resulted in None prediction")

    begin = time.time()
    result_file, _ = Writer(label="pred")(data)
    latency_write = round(time.time() - begin, 2)

    total_latency = round(time.time() - start, 2)
    result_json = {
        "label_names": self.task_seg_vertebra.labels,
        "centroids": centroids,
        "latencies": {
            "locate_spine": l1,
            "locate_vertebra": l2,
            "segment_vertebra": l3,
            "write": latency_write,
            "total": total_latency,
        },
    }

    logging.info(f"Result Mask (aggregated/pre-restore): {result_mask.shape}; total_latency: {total_latency}")
    return result_file, result_json

except Exception as e:
    logging.error(f"Error in vertebra pipeline: {str(e)}")
    raise

After running, we got:
File "C:\MLapp008\radiology\lib\infers\vertebra_pipeline.py", line 162, in call
raise ValueError("segment_vertebra returned None result_mask")
ValueError: segment_vertebra returned None result_mask

@linfanluntan
Copy link
Author

modified segment_vertebra

def segment_vertebra(self, request, image, centroids):
    original_size = list(image.shape)
    result_mask = None
    l = None
    image_cached = None
    count = 0

    logging.info(f"Starting vertebra segmentation for {len(centroids)} centroids")
    logging.info(f"Initial memory usage: {psutil.virtual_memory().percent}%")

    try:
        for centroid in tqdm(centroids):
            logging.info(f"Processing centroid: {centroid}")
            
            req = copy.deepcopy(request)
            req.update({
                "image": image,
                "image_cached": image_cached,
                "original_size": original_size,
                "centroids": [centroid],
                "pipeline_mode": True,
                "logging": "ERROR" if count > 1 else "INFO",
            })

            d, r = self.task_seg_vertebra(req)
            l = self._latencies(r, l)

            if "image" not in d or d["image"] is None:
                raise ValueError(f"Segmentation failed for centroid {centroid}")

            image = d["image"]
            image_cached = image

            # Paste/Merge each mask
            v = d["current_label"]
            s = d["slices_cropped"]
            m = d["pred"]
            
            if m is None:
                logging.warning(f"Null prediction for centroid {centroid}")
                continue

            m[m > 0] = v

            mask = torch.zeros_like(image)
            mask[:, s[-3][0]:s[-3][1], s[-2][0]:s[-2][1], s[-1][0]:s[-1][1]] = m

            if result_mask is None:
                result_mask = mask
            else:
                result_mask = result_mask + mask
                result_mask[result_mask > v] = v

            count += 1

            if count % 5 == 0:  # Log memory usage every 5 iterations
                logging.info(f"Memory usage after {count} centroids: {psutil.virtual_memory().percent}%")

        if result_mask is None:
            raise ValueError("No valid segmentation produced")

        logging.info(f"Vertebra segmentation completed. Result mask shape: {result_mask.shape}")
        return result_mask, l

    except Exception as e:
        logging.error(f"Error in vertebra segmentation: {str(e)}")
        raise

    finally:
        logging.info(f"Final memory usage: {psutil.virtual_memory().percent}%")

[2024-10-30 09:53:23,907] [26228] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'vertebra_pipeline', 'image': 'spleen_53', 'device': 'NVIDIA RTX A6000:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'}
[2024-10-30 09:53:23,908] [26228] [MainThread] [INFO] (root:166) - Starting locate_spine stage
[2024-10-30 09:53:23,909] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda:0', 'model': 'vertebra_pipeline', 'image': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'Combines three stage for vertebra segmentation', 'image_path': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'pipeline_mode': True}
[2024-10-30 09:53:23,910] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2024-10-30 09:53:23,910] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode']
[2024-10-30 09:53:28,391] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 4.48; image: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 09:53:28,541] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.15; image: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 09:53:28,542] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 09:53:28,543] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.001; image: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 09:53:28,567] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.024; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:28,568] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:28,808] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.24; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:28,847] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.038; image: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:28,847] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None}
[2024-10-30 09:53:28,848] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_spine.pt
[2024-10-30 09:53:35,816] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s)
[2024-10-30 09:53:35,816] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode', 'latencies', 'image_cached', 'pred']
[2024-10-30 09:53:35,817] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:35,826] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.008; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:35,827] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:40,819] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 4.992; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:40,820] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (BinaryMaskd): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32)
[2024-10-30 09:53:40,821] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 16.9131; Pre: 4.9370; Inferer: 6.9682; Invert: 0.0000; Post: 5.0070; Write: 0.0000
[2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['label_names', 'latencies', 'centroids']
[2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (root:170) - locate_spine completed. Image shape: torch.Size([1, 512, 512, 156]), Label shape: torch.Size([512, 512, 156])
[2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (root:173) - Starting locate_vertebra stage
[2024-10-30 09:53:40,823] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:24) - **************************** DATA ********************************************
[2024-10-30 09:53:40,823] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: device = cuda:0
[2024-10-30 09:53:40,823] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: model = vertebra_pipeline
[2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image = torch.Size([1, 512, 512, 156])
[2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_extension = .nrrd
[2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_dtype = uint8
[2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: client_id = user-xyz
[2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: description = Combines three stage for vertebra segmentation
[2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image_path = C:\ML_dataset\khoury\spleen_53.nii.gz
[2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: label = torch.Size([512, 512, 156])
[2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: pipeline_mode = True
[2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:38) - ******************************************************************************
[2024-10-30 09:53:40,826] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s)
[2024-10-30 09:53:40,826] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode']
[2024-10-30 09:53:40,827] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 09:53:40,827] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 09:53:41,185] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.357; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 09:53:41,186] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 09:53:41,297] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.111; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 09:53:41,334] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.036; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32)
[2024-10-30 09:53:41,342] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CropForegroundd): Time: 0.008; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32)
[2024-10-30 09:53:41,342] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': device(type='cpu'), 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None}
[2024-10-30 09:53:41,342] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_vertebra.pt
[2024-10-30 09:53:41,885] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - INV - Run Transform(s)
[2024-10-30 09:53:41,885] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - INV - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred']
[2024-10-30 09:53:42,233] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (CropForegroundd): Time: 0.347; image: torch.Size([25, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32)
[2024-10-30 09:54:23,117] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (Spacingd): Time: 40.8844; image: torch.Size([25, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32)
[2024-10-30 09:54:23,162] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s)
[2024-10-30 09:54:23,163] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred']
[2024-10-30 09:54:23,164] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.001; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32)
[2024-10-30 09:54:23,584] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.42; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32)
[2024-10-30 09:54:25,283] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 1.698; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 09:54:32,057] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 6.774; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32)
[2024-10-30 09:54:32,058] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 09:54:32,921] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (VertebraLocalizationSegmentation): Time: 0.862; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32)
[2024-10-30 09:54:32,922] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 52.0994; Pre: 0.5160; Inferer: 0.5010; Invert: 41.3184; Post: 9.7610; Write: 0.0000
[2024-10-30 09:54:32,922] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['centroids', 'label_names', 'latencies']
[2024-10-30 09:54:32,922] [26228] [MainThread] [INFO] (root:176) - locate_vertebra completed. Centroids: {}
[2024-10-30 09:54:32,923] [26228] [MainThread] [INFO] (root:179) - Starting segment_vertebra stage
[2024-10-30 09:54:32,923] [26228] [MainThread] [INFO] (root:93) - Starting vertebra segmentation for 0 centroids
[2024-10-30 09:54:32,930] [26228] [MainThread] [INFO] (root:94) - Initial memory usage: 9.7%
[2024-10-30 09:54:32,931] [26228] [MainThread] [ERROR] (root:151) - Error in vertebra segmentation: No valid segmentation produced
[2024-10-30 09:54:32,937] [26228] [MainThread] [INFO] (root:155) - Final memory usage: 9.7%

@SachidanandAlle
Copy link
Collaborator

@diazandr3s can you please help

@diazandr3s
Copy link
Collaborator

Thanks for the ping, @SachidanandAlle

@linfanluntan, many thanks for detailed information and sorry for the late reply.

From the error logs you've shared, it seems the localization_vertebra model didn't detect any vertebra:

File "C:\MLapp008\radiology\lib\infers\vertebra_pipeline.py", line 162, in call
raise ValueError("segment_vertebra returned None result_mask")
ValueError: segment_vertebra returned None result_mask

This log is more clear:

[2024-10-30 09:54:32,923] [26228] [MainThread] [INFO] (root:93) - Starting vertebra segmentation for 0 centroids

Could you please share more details of the volume you used to run inference? What do you see after running localization_spine only?

Let us know

@junxiant
Copy link

Hi,

I might be facing a similar issue.

Error when running vertebra_pipeline model
image

I have attached the log for the error when i ran:

  1. localization_spine
  2. localization_vertebra
  3. vertebra_pipeline

vertebra_pipeline_error.txt

The sample data was taken from verse: https://osf.io/t43px

@diazandr3s
Copy link
Collaborator

Hi @junxiant,

Thanks for sharing logs. Yes, this seems like a similar issue. Could you please share what's the output of the localization spine model? You can run inference using only that model instead of the pipeline.

It lokks like that's the source of the issue. The transfiorn CropForegroundd in localization_vertebra model crops the image based on the spine predicted mask.

From the logs you've shared:

[2024-11-30 07:03:10,441] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.0007; image: torch.Size([1, 124, 260, 93])(torch.float32); label: torch.Size([1, 124, 260, 92])(torch.float32)
[2024-11-30 07:03:10,444] [3310] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CropForegroundd): Time: 0.0028; image: torch.Size([1, 0, 0, 0])(torch.float32);

@junxiant
Copy link

junxiant commented Dec 2, 2024

Hi @diazandr3s ,
app.log

I have attached the updated log that only include the run of localization_spine. I dont see any segmentation on the sample data.
image

I forgot to include the version previously, here:
monailabel = 0.8.3
monai = 1.3.2
monai-weekly = 1.4.dev2427

Appreciate your help in this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@linfanluntan @SachidanandAlle @diazandr3s @junxiant