Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDTEST-437] Retry failed tests for cucumber #212

Merged
merged 11 commits into from
Aug 5, 2024

Conversation

anmarchenko
Copy link
Member

@anmarchenko anmarchenko commented Aug 1, 2024

What does this PR do?
This PR adds auto test retries support for cucumber test framework. The feature is enabled if flaky_test_retries_enabled field is set in remote settings.

As Cucumber has its own built-in retries mechanism, it is enough for us to override the framework configuration and enable test retries when this feature is enabled on Datadog side.

The following customer-facing settings are added:

  • DD_CIVISIBILITY_RETRY_FAILED_TESTS_MAX_ATTEMPTS - number of test retries per each failed test (default: 5)
  • DD_CIVISIBILITY_RETRY_FAILED_TESTS_TOTAL_LIMIT - total number of failed tests that are going to be retried (default: 100)

Also this PR fixes retries support for test suite and test sessions status: now test suite and test session are reported as passed if tests succeed on retries.

How to test the change?
Unit tests are provided for all the possible cases when retrying failed tests.

@codecov-commenter
Copy link

codecov-commenter commented Aug 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.87%. Comparing base (f00caf6) to head (5e2a950).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #212      +/-   ##
==========================================
+ Coverage   98.84%   98.87%   +0.02%     
==========================================
  Files         253      256       +3     
  Lines       11630    11868     +238     
  Branches      521      533      +12     
==========================================
+ Hits        11496    11734     +238     
  Misses        134      134              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@anmarchenko anmarchenko changed the title [SDTEST-437] WIP: flaky test retries for cucumber [SDTEST-437] Flaky test retries for cucumber Aug 5, 2024
@anmarchenko anmarchenko changed the title [SDTEST-437] Flaky test retries for cucumber [SDTEST-437] Retry failed tests for cucumber Aug 5, 2024
@anmarchenko anmarchenko marked this pull request as ready for review August 5, 2024 10:49
@anmarchenko anmarchenko requested review from a team as code owners August 5, 2024 10:49
@anmarchenko anmarchenko requested a review from tonyredondo August 5, 2024 10:49
@anmarchenko anmarchenko merged commit 2eb656d into main Aug 5, 2024
28 checks passed
@anmarchenko anmarchenko deleted the anmarchenko/flaky_test_retries_cucumber branch August 5, 2024 13:56
@github-actions github-actions bot added this to the 1.4.0 milestone Aug 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants