Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate benchmarks reliability #1239

Closed
crisidev opened this issue Mar 7, 2022 · 3 comments
Closed

Validate benchmarks reliability #1239

crisidev opened this issue Mar 7, 2022 · 3 comments
Labels
question Further information is requested server Rust server SDK testing

Comments

@crisidev
Copy link
Contributor

crisidev commented Mar 7, 2022

Since #1230 we run a benchmark using wrk against origin/main and any pull request and calculate the deviation.

We need to validate if the benchmark is reliable when run by the Github action runner. If it is not reliable we can

  • Run the benchmark on hardware we own
  • Change the benchmark to count instruction instead of using time
@crisidev crisidev added question Further information is requested testing server Rust server SDK labels Mar 7, 2022
@david-perez
Copy link
Contributor

I've been keeping an eye on these. Usually the deviation does not go over 50%, but here's a PR in which we had 94% in one message, 16% in the next one #1368.

@david-perez
Copy link
Contributor

Change the benchmark to count instruction instead of using time

Instruction count is is a useful metric for a compiler but not for what we're building. We could introduce severe performance degradation without noticeably increasing instruction count (think e.g. replacing a linear-time query string parsing function with a quadratic one).

@david-perez
Copy link
Contributor

Closing since benchmarks were removed years ago because of unreliability, we'll use #214 as the tracking issue to re-add them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested server Rust server SDK testing
Projects
None yet
Development

No branches or pull requests

2 participants