Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix least duration algo with random ordering #109

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

kohutgergely
Copy link

@kohutgergely kohutgergely commented Jan 19, 2025

Description

Hi!

I found a rather critical bug in the least duration algorithm when running together with random ordering and when durations do not exist. The issue arises from the way pytest represents a test case item. It uses only the test case name. In the least duration algorithm you use the string representation of the test item to sort the test cases to ensure deterministic groups even when random order is applied. Due to the fact that only the name is used for ordering, the split groups become undeterministic (depending on the initial order) in those cases when there are multiple tests with same name in different modules and durations does not exist. This leads to repeated and skipped tests when used in CI for example.

A very simple solution would be to use the nodeid to sort the tests which I tested and implemented in this PR. I hope you find it OK but if modifications needed I am happy to do that.

Checklist

  • [✅ ] Tests covering the new functionality have been added
  • [❌ ] Documentation has been updated OR the changes are too minor to be documented
  • [✅ ] The Changes are listed in the CHANGELOG.md OR the changes are insignificant

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant