triton server EXECUTION_ENV, no module named ' '

[daewoung]

Hello!
The server I'm using is version cuda 11.6, and I want to use multiple models.

To achieve this, I've set up a Conda environment and packed it using conda-pack, aiming for a higher version of diffusers. Here's the list of the Conda environment I intend to use.

`

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
accelerate 0.24.1 pypi_0 pypi
asttokens 2.4.1 pyhd8ed1ab_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
ca-certificates 2023.7.22 hbcca054_0 conda-forge
certifi 2023.7.22 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
cmake 3.27.7 pypi_0 pypi
comm 0.1.4 pyhd8ed1ab_0 conda-forge
conda-pack 0.7.1 pypi_0 pypi
cudatoolkit 11.7.0 hd8887f6_11 conda-forge
cudnn 8.9.2.26 cuda11_0 anaconda
debugpy 1.8.0 py39h3d6467e_1 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
diffusers 0.22.3 pypi_0 pypi
exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge
executing 2.0.1 pyhd8ed1ab_0 conda-forge
filelock 3.13.1 pypi_0 pypi
fsspec 2023.10.0 pypi_0 pypi
huggingface-hub 0.19.0 pypi_0 pypi
idna 3.4 pypi_0 pypi
importlib-metadata 6.8.0 pyha770c72_0 conda-forge
importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge
ipykernel 6.26.0 pyhf8b6a83_0 conda-forge
ipython 8.17.2 pyh41d4057_0 conda-forge
jedi 0.19.1 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pypi_0 pypi
jupyter_client 8.6.0 pyhd8ed1ab_0 conda-forge
...
`
Afterwards, I designated this environment as the EXECUTION_ENV PATH and launched the Triton server.

`

=============================
== Triton Inference Server ==

NVIDIA Release 22.02 (build 32400308)

Copyright (c) 2018-2021, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

I1109 10:33:50.029835 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/pytorch/libtriton_pytorch.so
I1109 10:33:50.203003 1 libtorch.cc:1306] TRITONBACKEND_Initialize: pytorch
I1109 10:33:50.203015 1 libtorch.cc:1316] Triton TRITONBACKEND API version: 1.8
I1109 10:33:50.203017 1 libtorch.cc:1322] 'pytorch' TRITONBACKEND API version: 1.8
I1109 10:33:50.203040 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/tensorflow1/libtriton_tensorflow1.so
2023-11-09 10:33:50.285234: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-11-09 10:33:50.305578: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I1109 10:33:50.305619 1 tensorflow.cc:2176] TRITONBACKEND_Initialize: tensorflow
I1109 10:33:50.305630 1 tensorflow.cc:2186] Triton TRITONBACKEND API version: 1.8
I1109 10:33:50.305634 1 tensorflow.cc:2192] 'tensorflow' TRITONBACKEND API version: 1.8
I1109 10:33:50.305638 1 tensorflow.cc:2216] backend configuration:
{}
I1109 10:33:50.305686 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so
I1109 10:33:50.306612 1 onnxruntime.cc:2319] TRITONBACKEND_Initialize: onnxruntime
I1109 10:33:50.306620 1 onnxruntime.cc:2329] Triton TRITONBACKEND API version: 1.8
I1109 10:33:50.306622 1 onnxruntime.cc:2335] 'onnxruntime' TRITONBACKEND API version: 1.8
I1109 10:33:50.306624 1 onnxruntime.cc:2365] backend configuration:
{}
I1109 10:33:50.311639 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/openvino_2021_2/libtriton_openvino_2021_2.so
I1109 10:33:50.316714 1 openvino.cc:1234] TRITONBACKEND_Initialize: openvino
I1109 10:33:50.316721 1 openvino.cc:1244] Triton TRITONBACKEND API version: 1.8
I1109 10:33:50.316723 1 openvino.cc:1250] 'openvino' TRITONBACKEND API version: 1.8
I1109 10:33:50.414583 1 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fbe70000000' with size 268435456
I1109 10:33:50.414774 1 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864
I1109 10:33:50.415551 1 model_config_utils.cc:667] Server side auto-completed config: name: "animate_diff"
input {
name: "text_input"
data_type: TYPE_STRING
dims: 1
}
output {
name: "text_output"
data_type: TYPE_STRING
dims: -1
}
parameters {
key: "EXECUTION_ENV_PATH"
value {
string_value: "$$TRITON_MODEL_DIRECTORY/animdiff.tar.gz"
}
}
backend: "python"

I1109 10:33:50.415721 1 model_repository_manager.cc:700] AsyncLoad() 'animate_diff'
I1109 10:33:50.415753 1 model_repository_manager.cc:939] TriggerNextAction() 'animate_diff' version 1: 1
I1109 10:33:50.415759 1 model_repository_manager.cc:975] Load() 'animate_diff' version 1
I1109 10:33:50.415763 1 model_repository_manager.cc:994] loading: animate_diff:1
I1109 10:33:50.516341 1 model_repository_manager.cc:1052] CreateModel() 'animate_diff' version 1
I1109 10:33:50.516536 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/python/libtriton_python.so
I1109 10:33:50.520291 1 python.cc:1679] 'python' TRITONBACKEND API version: 1.8
I1109 10:33:50.520321 1 python.cc:1701] backend configuration:
{}
I1109 10:33:50.520350 1 python.cc:1811] Shared memory configuration is shm-default-byte-size=67108864,shm-growth-byte-size=67108864,stub-timeout-seconds=30
I1109 10:33:50.520576 1 python.cc:1859] TRITONBACKEND_ModelInitialize: animate_diff (version 1)
I1109 10:33:50.522155 1 model_config_utils.cc:1616] ModelConfig 64-bit fields:
I1109 10:33:50.522160 1 model_config_utils.cc:1618] ModelConfig::dynamic_batching::default_queue_policy::default_timeout_microseconds
I1109 10:33:50.522162 1 model_config_utils.cc:1618] ModelConfig::dynamic_batching::max_queue_delay_microseconds
I1109 10:33:50.522164 1 model_config_utils.cc:1618] ModelConfig::dynamic_batching::priority_queue_policy::value::default_timeout_microseconds
I1109 10:33:50.522166 1 model_config_utils.cc:1618] ModelConfig::ensemble_scheduling::step::model_version
I1109 10:33:50.522167 1 model_config_utils.cc:1618] ModelConfig::input::dims
I1109 10:33:50.522169 1 model_config_utils.cc:1618] ModelConfig::input::reshape::shape
I1109 10:33:50.522171 1 model_config_utils.cc:1618] ModelConfig::instance_group::secondary_devices::device_id
I1109 10:33:50.522172 1 model_config_utils.cc:1618] ModelConfig::model_warmup::inputs::value::dims
I1109 10:33:50.522174 1 model_config_utils.cc:1618] ModelConfig::optimization::cuda::graph_spec::graph_lower_bound::input::value::dim
I1109 10:33:50.522176 1 model_config_utils.cc:1618] ModelConfig::optimization::cuda::graph_spec::input::value::dim
I1109 10:33:50.522178 1 model_config_utils.cc:1618] ModelConfig::output::dims
I1109 10:33:50.522179 1 model_config_utils.cc:1618] ModelConfig::output::reshape::shape
I1109 10:33:50.522181 1 model_config_utils.cc:1618] ModelConfig::sequence_batching::direct::max_queue_delay_microseconds
I1109 10:33:50.522183 1 model_config_utils.cc:1618] ModelConfig::sequence_batching::max_sequence_idle_microseconds
I1109 10:33:50.522185 1 model_config_utils.cc:1618] ModelConfig::sequence_batching::oldest::max_queue_delay_microseconds
I1109 10:33:50.522186 1 model_config_utils.cc:1618] ModelConfig::sequence_batching::state::dims
I1109 10:33:50.522188 1 model_config_utils.cc:1618] ModelConfig::sequence_batching::state::initial_state::dims
I1109 10:33:50.522190 1 model_config_utils.cc:1618] ModelConfig::version_policy::specific::versions
I1109 10:33:50.522270 1 python.cc:1618] Using Python execution env model_repository/animate_diff/animdiff.tar.gz
I1109 10:33:50.522435 1 python.cc:1903] TRITONBACKEND_ModelInstanceInitialize: animate_diff (GPU device 0)
I1109 10:33:50.522499 1 backend_model_instance.cc:105] Creating instance animate_diff on GPU 0 (8.6) using artifact ''
I1109 10:34:18.054521 48 python.cc:1206] Starting Python backend stub: source /tmp/python_env_BwfCQh/0/bin/activate && exec env LD_LIBRARY_PATH=/tmp/python_env_BwfCQh/0/lib:$LD_LIBRARY_PATH /opt/tritonserver/backends/python/triton_python_backend_stub model_repository/animate_diff/1/model.py triton_python_backend_shm_region_1 67108864 67108864 1 /opt/tritonserver/backends/python 56 animate_diff
1109 10:34:18.719397 48 pb_stub.cc:369] Failed to initialize Python stub: ModuleNotFoundError: No module named 'diffusers'

At:
/opt/tritonserver/model_repository/animate_diff/1/model.py(12):
(219): _call_with_frames_removed
(848): exec_module
(686): _load_unlocked
(975): _find_and_load_unlocked
(991): _find_and_load

I1109 10:34:18.719505 1 python.cc:1993] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I1109 10:34:18.719646 1 python.cc:1882] TRITONBACKEND_ModelFinalize: delete model state
I1109 10:34:18.719658 1 triton_backend_manager.cc:101] unloading backend 'python'
I1109 10:34:18.719660 1 python.cc:1839] TRITONBACKEND_Finalize: Start
I1109 10:34:20.614596 1 python.cc:1844] TRITONBACKEND_Finalize: End
E1109 10:34:20.615449 1 model_repository_manager.cc:1152] failed to load 'animate_diff' version 1: Internal: ModuleNotFoundError: No module named 'diffusers'
`
However, I'm getting the error mentioned above. What should I do? Thank you.
Before packing the Conda environment, I executed 'export PYTHONNOUSERSITE=True.