-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SDTEST-437] Retry failed tests for cucumber #212
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #212 +/- ##
==========================================
+ Coverage 98.84% 98.87% +0.02%
==========================================
Files 253 256 +3
Lines 11630 11868 +238
Branches 521 533 +12
==========================================
+ Hits 11496 11734 +238
Misses 134 134 ☔ View full report in Codecov by Sentry. |
…nteger - whether retries enabled and max attempts number
…retries into account
…st passed after retries
… was already executed at least once in the current test suite
…figured by Datadog user
anmarchenko
changed the title
[SDTEST-437] WIP: flaky test retries for cucumber
[SDTEST-437] Flaky test retries for cucumber
Aug 5, 2024
anmarchenko
changed the title
[SDTEST-437] Flaky test retries for cucumber
[SDTEST-437] Retry failed tests for cucumber
Aug 5, 2024
juan-fernandez
approved these changes
Aug 5, 2024
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds auto test retries support for cucumber test framework. The feature is enabled if
flaky_test_retries_enabled
field is set in remote settings.As Cucumber has its own built-in retries mechanism, it is enough for us to override the framework configuration and enable test retries when this feature is enabled on Datadog side.
The following customer-facing settings are added:
DD_CIVISIBILITY_RETRY_FAILED_TESTS_MAX_ATTEMPTS
- number of test retries per each failed test (default: 5)DD_CIVISIBILITY_RETRY_FAILED_TESTS_TOTAL_LIMIT
- total number of failed tests that are going to be retried (default: 100)Also this PR fixes retries support for test suite and test sessions status: now test suite and test session are reported as passed if tests succeed on retries.
How to test the change?
Unit tests are provided for all the possible cases when retrying failed tests.