[POC] per test code coverage using built-in Coverage.resume/suspend

[anmarchenko]

Ruby has built-in Coverage class to track code coverage. It is used by all major code coverage libraries in Ruby right now. This object provides a way to track total code coverage per process. There is some trick to use it for per test code coverage: use Coverage.resume and Coverage.suspend to start/stop coverage tracing and Coverage.results(stop: false, clear: true) to get intermediate results.

This approach is not suitable for us because of the following disadvantages:

• It interferes with existing code coverage setup our users might have
• It does not work with multithreaded test execution
• On every step it returns not only files interesting to us but all of the Ruby files loaded and we have to do expensive filtering to get only the files that were actually covered by the current test
• resume and suspend are only available for Ruby 3.1+ (we need to support 2.7+)

Nevertheless, this PR explores performance of this approach to have a baseline for comparisons.

Performance evaluation

Projects used to evaluate performance:

• sidekiq: very small and fast test suite
• rubocop: very big but quite fast test suite (20 000 tests under 2 minutes)
• vagrant: very big and slow test suite (5 000 tests in about 14 minutes)

In the following table are durations of the whole test session in seconds, in brackets overhead percentage compared to run with datadog-ci.

Coverage type	Sidekiq	Rubocop	Vagrant
No coverage	~6,8	111	827
Simplecov (total coverage)	~7	137	864
Per test coverage, lines	7,8 (14%)	460 (314%)	915 (10%)

* overhead here is compared to "Simplecov (total coverage)"

Relevant code

# to filter results only taking the the files in git repository's root
@regex = /\A#{Regexp.escape(Utils::Git.root + File::SEPARATOR)}/i.freeze

::Coverage.setup(lines: true)

# when starting test
::Coverage.resume

# when stopping test
result = ::Coverage.result(stop: false, clear: true)
::Coverage.suspend if ::Coverage.running?

result.select do |path, coverage|
  path =~ @regex && coverage[:lines].any? { |line| !line.nil? && line > 0 }
end