minitest: fix rails parallel test runner #115
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes rails parallel test executor instrumentation that was broken since I added ci-queue support to minitest framework.
Why was this broken?
In the previous episode of "Datadog figures out how to instrument all possible ways one can use Minitest" we watched minitest-reporters interfering with
datadog-ci
plugin which caused me to remove plugin and instrument Minitest::CompositeReporter to mark start and end of test sessions.But... (suspenseful music) there is a twist!
Here is the
Minitest.run
method (important parts only):Unfortunately as you can see
reporter.start
is called after theself.parallel_executor.start
. This is usually not that important, but not for Rails parallel executor, ladies and gentlemen! When#start
is called, it forks processes copying memory and creating an independent process to run your tests.This worked fine when we used plugin to start session: datadog-ci stores test session in global context that is copied on fork to all child processes. But now, if we start session after workers are forked, then the session is stored only in main test process and not available to workers that actually execute tests.
This PR fixes this issue by moving session start (again) to the
init_plugins
method but instead of using official Minitest plugin system we are instrumenting this method directly because it is the only way to make it work for all cases discovered so far.How to test the change?
Manual testing via https://github.com/anmarchenko/openstreetmap-website is required (this is how the issue was discovered). I am happy to pair if anyone would like to see that it works.
This project will be added to Datadog's integration testing environment later.
Why don't you have unit tests for rails parallel executor?
It is extremely tricky to get any data out of forked processes in unit tests, I created a task to investigate it and try figure out a method of creating test environment suitable for unit tests.