-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SDTEST-437] Auto test retries for minitest #214
Conversation
… make it accessible in Minitest hooks
…ncapsulate all the retries logic inside TestRetries::Component
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #214 +/- ##
==========================================
- Coverage 98.89% 98.88% -0.01%
==========================================
Files 261 261
Lines 12096 12243 +147
Branches 544 552 +8
==========================================
+ Hits 11962 12107 +145
- Misses 134 136 +2 ☔ View full report in Codecov by Sentry. |
|
||
class FlakyTestSuite2 < Minitest::Test | ||
@@max_flaky_test_failures = 4 | ||
@@flaky_test_failures = 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 Code Quality Violation
Do not use class variables (...read more)
The rule "Avoid class variables" refers to the practice of refraining from using class variables (variables starting with '@@') in Ruby. Class variables are shared between a class and all of its descendants, which can lead to unexpected behavior and bugs that are difficult to trace. This is because if a class variable is changed in a subclass, that change will also affect the superclass and all other subclasses.
This rule is crucial for maintaining clean, predictable, and easy-to-debug code. It also helps to prevent unintentional side effects that can occur when class variables are manipulated in different parts of a program.
To adhere to this rule, consider using class instance variables or constants instead. Class instance variables belong solely to the class they are defined in, and their value does not get shared with subclasses. Constants, on the other hand, are a good option when the value is not meant to change. For example, in the given non-compliant code, the class variable @@class_var
could be replaced with a class instance variable @class_var
or a constant CLASS_VAR
, depending on the intended use.
i_suck_and_my_tests_are_order_dependent! | ||
|
||
@@max_flaky_test_failures = 4 | ||
@@flaky_test_failures = 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 Code Quality Violation
Do not use class variables (...read more)
The rule "Avoid class variables" refers to the practice of refraining from using class variables (variables starting with '@@') in Ruby. Class variables are shared between a class and all of its descendants, which can lead to unexpected behavior and bugs that are difficult to trace. This is because if a class variable is changed in a subclass, that change will also affect the superclass and all other subclasses.
This rule is crucial for maintaining clean, predictable, and easy-to-debug code. It also helps to prevent unintentional side effects that can occur when class variables are manipulated in different parts of a program.
To adhere to this rule, consider using class instance variables or constants instead. Class instance variables belong solely to the class they are defined in, and their value does not get shared with subclasses. Constants, on the other hand, are a good option when the value is not meant to change. For example, in the given non-compliant code, the class variable @@class_var
could be replaced with a class instance variable @class_var
or a constant CLASS_VAR
, depending on the intended use.
What does this PR do?
Adds auto test retries support for minitest
How to test the change?
Feature is covered by unit tests in
minitest/instrumentation_spec.rb
Tested using Sidekiq in this commit: anmarchenko/sidekiq@020ca88
Local test run ends successfully:
![image](https://private-user-images.githubusercontent.com/426400/356569938-bd1e745e-b2b2-43e6-907f-ca7389860ed7.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzg5NDU2NzQsIm5iZiI6MTczODk0NTM3NCwicGF0aCI6Ii80MjY0MDAvMzU2NTY5OTM4LWJkMWU3NDVlLWIyYjItNDNlNi05MDdmLWNhNzM4OTg2MGVkNy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjA3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIwN1QxNjIyNTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iZDZmMWE0OGU1MTdjNWVjMmQ0NjM0ZjYyMTE0NDkyODZhMGFkOTk0YWQxODM2ZGY0YTE1MGY3NTc5YmRjMzBhJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.r91I72kmz4gT2OU3P9ogee-JcRiqQFcZDUWQt_7a3mI)
New flaky test is detected by Datadog:
![image](https://private-user-images.githubusercontent.com/426400/356570006-e9e699d9-203e-4880-ae04-71b031473da8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzg5NDU2NzQsIm5iZiI6MTczODk0NTM3NCwicGF0aCI6Ii80MjY0MDAvMzU2NTcwMDA2LWU5ZTY5OWQ5LTIwM2UtNDg4MC1hZTA0LTcxYjAzMTQ3M2RhOC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjA3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIwN1QxNjIyNTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01MzRjNTg1ZjM1MDRiNTI5ODM3MmM4OTE5ZjQ3ODFhZTkwM2ZjZGExMTcyNDU4NjQwNzcyYmQ4YThmZWVlNjYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.W3zKhvkAJHtvxmRq7-PMN4h2s-Q3mW8VVTcDIVz5NiA)
Correct test runs are traced:
![image](https://private-user-images.githubusercontent.com/426400/356570176-c32a324e-707d-44f3-86e5-54ea687902eb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzg5NDU2NzQsIm5iZiI6MTczODk0NTM3NCwicGF0aCI6Ii80MjY0MDAvMzU2NTcwMTc2LWMzMmEzMjRlLTcwN2QtNDRmMy04NmU1LTU0ZWE2ODc5MDJlYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjA3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIwN1QxNjIyNTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT03Y2ZiYzRhNTZkNWM0YTYwMGI4NDY3YzE1MThmY2RkYTJiMzI3MmQxMzljN2Q4OTY4ODhkNzRiMWFiODIwZThjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.dzQS_DG2S8Tdie2kdKK1fa5l88_QjpSkuKYXCxa24ts)
Retries can be filtered using
![image](https://private-user-images.githubusercontent.com/426400/356570337-38a6ecef-62f9-4eed-b8b1-c7d1feb3e83e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzg5NDU2NzQsIm5iZiI6MTczODk0NTM3NCwicGF0aCI6Ii80MjY0MDAvMzU2NTcwMzM3LTM4YTZlY2VmLTYyZjktNGVlZC1iOGIxLWM3ZDFmZWIzZTgzZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjA3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIwN1QxNjIyNTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jOTg0ZGE4NzZmMzI3YTkxYmUxYjUyZTQ4YzM5Njc0NTA3MWE0MjRkZjA0YmIwN2FkNDkwMzI1YjczNjNmNThkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.RuvKzZl9T5Nl0WzLAslOchI7wF1nel0e7xJ9W7dXQNk)
test.is_retry
tag:Test session is reported as passed:
![image](https://private-user-images.githubusercontent.com/426400/356570414-3349d8ed-332f-4973-b96a-53ece79d8a31.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzg5NDU2NzQsIm5iZiI6MTczODk0NTM3NCwicGF0aCI6Ii80MjY0MDAvMzU2NTcwNDE0LTMzNDlkOGVkLTMzMmYtNDk3My1iOTZhLTUzZWNlNzlkOGEzMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwMjA3JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDIwN1QxNjIyNTRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1kZjdkNjExN2M2YzczM2NhZWI2ODE4MTlhODI4N2FhODlkMzEwNWY2OTg1OTUwY2JjNTVhZGFlODY2ZDJhODZmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.NoGHxzsqXXvAB_atXrOE6pOEWyIljOI0JfgW4v6NYL4)