Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip-compile doesn't provide hashes for wheels hosted by simple index servers #1536

Open
stefansjs opened this issue Nov 15, 2021 · 7 comments · May be fixed by #1556
Open

pip-compile doesn't provide hashes for wheels hosted by simple index servers #1536

stefansjs opened this issue Nov 15, 2021 · 7 comments · May be fixed by #1556
Labels
bug Something is not working hashes Related to hashes generated via --generate-hashes

Comments

@stefansjs
Copy link

pip-compile doesn't compute hashes for all index servers, which leads to pip install failure. Specifically, any index server that provides hashes from its json API, will prevent pip-compile for checking if other installation candidates are available from other index servers.

Environment Versions

  1. OS Type
  2. Python version: $ python -V
  3. pip version: $ pip --version
  4. pip-tools version: $ pip-compile --version

Steps to replicate

  1. host a simple index server
  2. add wheels for source packages in pypi, e.g. avro-python3==1.9.2.1
  3. list this requirement in requirements.in
  4. run pip-compile --generate-hashes -i 'http://<my local index server>.com/' --extra-index-url https://pypi.org/simple
  5. run pip install -r requirements.txt

Expected result

The hashes should include the wheel in the local index server.

Actual result

Only hashes from pypi.org are listed in requirements.txt.

Looking at the implementation, it looks like piptools first tries to get hashes from the first index server that implements the json API. The first server that responds with a json blob is taken as the only hash candidates. Files are not manually hashed for any missing files.

In this case, because some files are on pypi, which implements the json API, other index servers are able to provide valid installation candidates to pip, but piptools does not bother to hash them, thus causing a pip installation failure due to the missing hashes.

stefansjs added a commit to stefansjs/pip-tools that referenced this issue Nov 16, 2021
Get json from all index servers instead of just the first, then computes
hashes on all remaining index links not already returned from servers.
@stefansjs
Copy link
Author

I've got a PR underway. I'm just trying to make sure I add enough test coverage first.

@snmishra
Copy link

@stefansjs I'm running into this limitation as well. Any updates on progress? Could I offer assistance?

@stefansjs
Copy link
Author

@snmishra I just got back to this PR. It sat on my back burner for a little longer than I'd like to admit. So far my feature is implemented on https://github.com/stefansjs/pip-tools/tree/all_hashes but I don't think it has enough test coverage to ensure that my implementation is right in all cases.

If it's easier, maybe I should start the PR and we can discuss changes in that CR discussion. Do you think that's the best way to discuss?

@snmishra
Copy link

@stefansjs Sounds good

@stefansjs stefansjs linked a pull request Jan 25, 2022 that will close this issue
4 tasks
@atugushev atugushev added bug Something is not working hashes Related to hashes generated via --generate-hashes labels Apr 6, 2022
@jedie
Copy link

jedie commented Nov 20, 2022

Any News here?!?

@atugushev
Copy link
Member

Any News here?!?

@jedie The work is in progress, see related PR #1556.

jedie added a commit to jedie/victron-ble2mqtt that referenced this issue Sep 22, 2024
I run into: piwheels/packages#473 (Missing hashes from piwheel) and there
is jazzband/pip-tools#1536 for the real problem.

But, just add piwheel as extra-index fix the problem and hashes from there packages just added.
Fine.
@jedie
Copy link

jedie commented Sep 22, 2024

Hm. Strange, i have just add piwheels as extra urls in pip-compile call to add the package hashes from there, too?!?

See: jedie/victron-ble2mqtt@972fe33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is not working hashes Related to hashes generated via --generate-hashes
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants