Custom Nodes as Python Dependencies

[marclove]

I recently ran into an issue where a ComfyUI change broke a custom node. The only way to fix it was to rollback ComfyUI but that rollback broke other custom nodes.

Fortunately the custom node fixed the problem quickly, but this is a classic dependency management problem and I wondered if there's been any discussion of making custom nodes Python dependencies to make it easier to handle and prevent such issues.

I'm not an expert in pip's dependency resolver, but this should also speed up the installation process since you'd do a single pip install that would do just one dependency resolution process, checks, and required installs. Right now, if you have several custom nodes, many of them do yet another pip install, which triggers that whole process again and a lot of duplicate work that slows down spinning up cloud machines.

I'm interested in contributing to make that possible, but wanted to get a temperature check before investing the time.

[phineas-pta]

my workaround is combine requirements of all nodes into 1 big file then pip install

[andcarefree]

My solution was similar to yours, but I ran into another problem. How do you solve the problem of dependency conflicts between customized nodes? Some nodes depend on specified versions and have conflicts, but I hope to be able to execute different workflows in the same environment.

[ltdrdata]

This is a challenging aspect. It's the case where dependencies conflict between custom nodes. If we manage dependencies strictly, depending on the installation order, it can become impossible to resolve the dependencies.

On the other hand, if we manage them loosely, even if conflicts occur, ignoring those parts and proceeding with the installation often allows for usage without major issues in most cases.

Custom node developers can verify basic dependency issues between ComfyUI and custom nodes, but verifying dependencies between custom nodes and other custom nodes is practically impossible.

For these reasons, in the case of ComfyUI-Manager, the initial method of installing dependencies for custom nodes using pip install -r requirements.txt was changed to individually installing dependencies for packages listed in requirements.txt. This change was made because even when there were only about 100 custom nodes, attempting to install all requirements at once from requirements.txt often resulted in failure with high probability.

The most challenging aspect is that code created by researchers often consists of requirements frozen to specific versions. And recently, a significant proportion of custom nodes are simply ported versions of these research codes.

[AaronLnoe]

I've recently encountered this issue myself.

My issue started in a weird way though, I tried to load a custom node and despite cleanly installing the node, the node was unable to import successfully no matter what I did. Manager failed, git clone failed, installing reqs worked - but nodes still failed to be imported. Restarted the server several times, but then I downloaded a clean version of swarm and did all three with the same processes and it worked fine. So something was wrong with my primary install.

I've tried with multiple versions of python, multiple versions of pip, all attempts were done in clean venvs, multiple installation methods ranging from installing independent dependencies, to req files, to concating and cleaning 1 super req file and I noticed that reqs for custom nodes are all over the place and not standardized. Some nodes includes things like Git links for examples, screenshots, and other miscellaneous data that "advertises" things that their nodes can do, other reqs pull from user git hubs and require specific file paths, some try to make assumptions around the machine type that is being used and install deps depending on what is detected (which I appreciate, but feel can cause issues in certain scenarios). I noticed that numpy and timm are prone to version conflicts.

I appreciate your position though. Relying on community code can have the issue of enforcing a standardization. The alternative would be to make the product capable of handling these issues gracefully. Maybe scan requirements files for conflicts and add that as a flag in the UI to draw attention to requirement conflicts. From this perhaps a profiler to load and unload sets of nodes to avoid conflicts in requirements.

I do think this issue would go a long way to making the overall product more stable and usable. Anyways, I appreciate your work, I hope my commentary is constructive in some way.

Thank you!