fix least duration algo with random ordering #109
+54
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Hi!
I found a rather critical bug in the least duration algorithm when running together with random ordering and when durations do not exist. The issue arises from the way pytest represents a test case item. It uses only the test case name. In the least duration algorithm you use the string representation of the test item to sort the test cases to ensure deterministic groups even when random order is applied. Due to the fact that only the name is used for ordering, the split groups become undeterministic (depending on the initial order) in those cases when there are multiple tests with same name in different modules and durations does not exist. This leads to repeated and skipped tests when used in CI for example.
A very simple solution would be to use the nodeid to sort the tests which I tested and implemented in this PR. I hope you find it OK but if modifications needed I am happy to do that.
Checklist
CHANGELOG.md
OR the changes are insignificant