create setup.py and support system wide installation

[blshkv]

This is a common problem that many tools which we want to distribute do not support system wide installation. Some of these tools provide their own "install.sh" or "setup.sh" or "requirements.txt" with a distro specific commands (sudo apt-install) to install dependencies instead of a proper setup.py.
With a proper setup.py script, an end user can install it using the following single command:
python3 setup.py install --user. It also a distro friendly and makes life for maintainers easier.

The end result, a FHS compliant directory structure should be created.
In case if software produce any output, a local user directory should be used, such as ~.//

pentoo/pentoo-overlay#596

[blshkv]

Please apply the following patch:
https://github.com/pentoo/pentoo-overlay/blob/master/app-forensics/spiderfoot/files/spiderfoot-3.3_add_support_user_homedir.patch

[bcoles]

Please apply the following patch:
https://github.com/pentoo/pentoo-overlay/blob/master/app-forensics/spiderfoot/files/spiderfoot-3.3_add_support_user_homedir.patch

There's a lot of work going on at the moment to refactor SpiderFoot as a package. Unfortunately that's still a long way off.

The patch above is going to break a bunch of things. But I agree something similar should be implemented. I'll notify you in this thread when complete.

[blshkv]

yeah, we adopted that patch a few times for each new version, but enough is enough ;-) I masked spiderfoot as "broken" until the refactoring is done, and we have a proper setup.py with user home support.

Thanks!

[bcoles]

yeah, we adopted that patch a few times for each new version, but enough is enough ;-) I masked spiderfoot as "broken" until the refactoring is done, and we have a proper setup.py with user home support.

Thanks!

As a stop gap measure, in theory you could force the path for the cache and database by setting the SPIDERFOOT_CACHE and SPIDERFOOT_DATA environment variables respectively. This will work for the cache, but as you suggest in pentoo/pentoo-overlay#739 SPIDERFOOT_DATA is currently broken. This is a separate issue which will be fixed long before SpiderFoot is packaged.

[bcoles]

The database file is now loaded from $HOME/.spiderfoot/spiderfoot.db. Fixed on master branch in #1343.

Support for the SPIDERFOOT_DATA environment variable has also been fixed. The environment variable takes priority. The home directory is used if the environment variable is not set.

This should be sufficient for your use case and remove the need for your patches.

The next official release should not be far away. Unfortunately, it will not support installation with setup.py.

[blshkv]

pentoo/pentoo-overlay#740
Please make sure that logs are also handled properly in the next version

[bcoles]

pentoo/pentoo-overlay#740
Please make sure that logs are also handled properly in the next version

The logs are now stored in $HOME/.spiderfoot/logs. Fixed on master branch in #1356.

The log directory may be overwritten by SPIDERFOOT_LOGS environment variable.

[bcoles]

@blshkv SpiderFoot 3.5 has been released. This should address the issues with the database, cache data, and logs. These are all now loaded/stored in ~/.spiderfoot/. Hopefully this is sufficient to remove the "broken" status from the Pentoo SpiderFoot port.

[blshkv]

Almost there. setup.py or pyproject.toml (PEP 517) are still missing.

[ShutdownRepo]

Any update on having a setup.py file? The requirements of the projects are very restrictive (since they are versioned), which creates conflicts with other tools. The great alternative to dockerised things is python virtual environments (like pipx for instance), and a setup.py file would be needed for pipx to be able to install spiderfoot

[bcoles]

Any update on having a setup.py file?

No. There has been no progress on creating a setup.py file. This is still desirable and the preferred approach for deployment.

Progress towards refactoring the libraries is ongoing at a glacial pace.

The requirements of the projects are very restrictive

The test requirements are restrictive and locked to specific versions; however, these are optional for development purposes only. They aren't required for installation and operation.

The required dependencies defined in requirements.txt are loosely versioned and not locked. They specify a range: a minimum version and sometimes specify a maximum version.

The vast majority of the dependencies (with a few inconvenient exceptions) use the latest major version branch. If these are causing conflicts with other software, the other software should be updated.

In instances where a minimum version is enforced, this is usually because older versions are not compatible or contain security vulnerabilities.

In instances where a maximum version is enforced, this is usually because newer versions are not compatible. Admittedly, we're a bit slow on checking whether the newest versions of dependencies are compatible. The maximum version, when set, may be safe to extend.