Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDTEST-173] early flake detection support for rspec and minitest #229

Merged
merged 26 commits into from
Sep 10, 2024

Conversation

anmarchenko
Copy link
Member

@anmarchenko anmarchenko commented Sep 3, 2024

What does this PR do?
Adds a strategy to retry new tests using early flake detection backend. This automatically makes the product work for RSpec and Minitest (but not Cucumber, will be added in subsequent PRs).

Motivation
Early flake detection for Ruby is to be released next

Additional Notes
The TestRetries::Component clearly breaks single responsibility principle: it must be refactored and it will be done in the separate PR.

How to test the change?
Unit tests are provided.

Tested using sidekiq project on staging.

Tests are marked as new by EFD:
image

They can be found with test.is_retry tag:
image

Test session has tag early_flake.enabled:
image

@anmarchenko anmarchenko changed the title move the test retries callback to fire after test span is finished [SDTEST-173] move the test retries callback to fire after test span is finished Sep 3, 2024
@codecov-commenter
Copy link

codecov-commenter commented Sep 4, 2024

Codecov Report

Attention: Patch coverage is 97.77778% with 11 lines in your changes missing coverage. Please review.

Project coverage is 98.89%. Comparing base (e2151ad) to head (6c16d02).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
lib/datadog/ci/test_retries/component.rb 83.07% 11 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #229      +/-   ##
==========================================
- Coverage   98.95%   98.89%   -0.06%     
==========================================
  Files         266      268       +2     
  Lines       12754    13171     +417     
  Branches      574      597      +23     
==========================================
+ Hits        12621    13026     +405     
- Misses        133      145      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@anmarchenko anmarchenko changed the title [SDTEST-173] move the test retries callback to fire after test span is finished [SDTEST-173] add retry strategy for new tests retry Sep 4, 2024
@anmarchenko anmarchenko changed the title [SDTEST-173] add retry strategy for new tests retry [SDTEST-173] early flaked detection support for rspec and minites Sep 10, 2024
@anmarchenko anmarchenko marked this pull request as ready for review September 10, 2024 10:35
@anmarchenko anmarchenko requested review from a team as code owners September 10, 2024 10:35
@anmarchenko anmarchenko changed the title [SDTEST-173] early flaked detection support for rspec and minites [SDTEST-173] early flaked detection support for rspec and minitest Sep 10, 2024
@anmarchenko anmarchenko changed the title [SDTEST-173] early flaked detection support for rspec and minitest [SDTEST-173] early flake detection support for rspec and minitest Sep 10, 2024
@anmarchenko anmarchenko merged commit c842a90 into main Sep 10, 2024
30 checks passed
@anmarchenko anmarchenko deleted the anmarchenko/test_retries_add_testvis_events branch September 10, 2024 12:23
@anmarchenko anmarchenko added this to the 1.5.0 milestone Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants