Wild variances in v3

[Snugug]

I'm using v3, running a live url 3 times. The URLs are all hitting Firebase's hosting CDN, so they're very stable performance-wise. In my latest run, my Performance score across 3 runs was 24, 73, 73. The exact same site, with identical code, run from the same infra, but at a different URL is reporting values of 89, 74, and 72. When running Lighthouse from Chrome, the same URLs consistently have a performance score of between 85-87. I'm curious as to why there is so much variance in the actions version of Lighthouse as compared to the browser version, and if there's any way to stabilize the results.

Thanks!

[exterkamp]

Do you have the benchmark indices of those variant runs? Or know the time the github-action took to run?

I know in the past sometimes github had bad days and actions VMs were really flaky. #14

[alekseykulikov]

Yes, would be interesting to see benchmarkIndex values. They should be around 750 to be trustable.
We need to enable PSI as an alternative runner GoogleChrome/lighthouse-ci#251

[alekseykulikov]

Let's keep the track of variance in #14