Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA 18238 - fIdentify the first commit where a test became flaky #18334

Draft
wants to merge 6 commits into
base: trunk
Choose a base branch
from

Conversation

santhoshct
Copy link
Contributor

Added script to find the commit that would have caused the flakiness.

Flaky Pattern Detection:

For each test execution in chronological order:

  • When a FAILED or FLAKY status is encountered, opens a window of size M
  • Counts failures/flaky occurrences within that window
  • If count ≥ N, marks this as a potential flaky pattern
  • Records detailed information about the pattern including:
    • Initial failing commit
    • Build information
    • All test executions within the window

Example

Given:

  • Window size (M) = 6
  • Required failures (N) = 2

Timeline:
[PASS] -> [PASS] -> [FAIL] -> [PASS] -> [FAIL] -> [PASS] -> [PASS]
^
|
Window starts here

Window Analysis:
[FAIL], [PASS], [FAIL], [PASS], [PASS] = 2 failures in 5 executions
Result: Pattern detected (2 failures ≥ N within window of size 6)

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@github-actions github-actions bot added triage PRs from the community build Gradle build or GitHub Actions labels Dec 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Gradle build or GitHub Actions ci-approved triage PRs from the community
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants