Test randomly hangs in CI

[RaphaelMarcouxCTRL]

Hi,

I've been using BUnit for a few months now and things are mostly going great. However, I'm now facing a weird issue that's hard to understand.

One of my tests seems to sometimes not be able to finish executing. It seems to hang on WaitForState or WaitForAssertionCalls.
Here what the test looks like:

public void CliquerSurResultat_FermeRecherche() {
        AfficherEdition();

        RechercheApi.RechercherProjet(default).ReturnsForAnyArgs(new RechercheAvecProfilResponse() {
            Items = { new() { Code = "X" } }
        });

        Edition.WaitForState(() => Edition.EditionSpinner() == null);
        Edition.Projet().Click();
        Recherche.WaitForState(() => Recherche.Resultats().Count > 0);
        Recherche.Resultats()[0].Click();
        Recherche.WaitForAssertion(() => Recherche.EstFermee());
    }

What the test does is as follows :
Render page which contains a list of items
Click on one item to open a "detailed" view for modification (a form)
Cllick on a field of the form, which opens a pop-up panel that displays a list of items that can be selected for this field
Click on one of those items
Check that the pop-up panel has been hidden

We have a lot of these kind of tests and I can't see anything special with it but for some reason, only this one seems to be causing deadlocks or something of that nature when executing in CI.

I have been able to reproduce it using the dotnet.exe CLI with a powershell script that runs the tests in a loop until it fails due to
--blame-hang-timeout being activated:

for ($i=1; $i -le 50; $i++) {dotnet test --blame-hang-timeout 20000ms; if($LASTEXITCODE -ne 0){throw "WOOPS! $i"}}
I've attached the whole project source with only the test that is causing problem.
CTRL.zip

[linkdotnet]

That might be related to #710 .

Can you try the latest preview release? See #209 how to get the latest preview

[RaphaelMarcouxCTRL]

@nguyenhthai indeed, in the test I have problems with, I have the same thing, a placeholder is being displayed during loading of the paga data and the WaitForState call waiting for the data to be displayed is causing the hang.

@linkdotnet gonna try the disposeAnsync and get back to you

[nguyenhthai]

@nguyenhthai indeed, in the test I have problems with, I have the same thing, a placeholder is being displayed during loading of the paga data and the WaitForState call waiting for the data to be displayed is causing the hang.

@linkdotnet gonna try the disposeAnsync and get back to you

Base on #622 and using the current build (not the preview), I was able to run the tests and pass them more consistently. Basically, I wrapped the WaitForState inside the component InvokeAsync and before that, I used additional Task.Delay().

[RaphaelMarcouxCTRL]

Okay I might need a little help on this one.

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace FeuilleTemps.Interface.Commun;

[TestClass]
public class TestsBase : Bunit.TestContext, IAsyncDisposable
{
    protected Bunit.TestContext Context => this;

    public async ValueTask DisposeAsync() {
        await Services.DisposeAsync();
        Services.Dispose();
    }

    [TestInitialize]
    public virtual void _Initialize() {
        Context.Services.AddTestLogging();
    }

    protected override void Dispose(bool disposing) {
    }
}

Am I doing this right?
Am I supposed to be calling Dispose explicitely somewhere?

[linkdotnet]

I am not an expert on MsTest and whether or not it supports

a) IAsyncDisposable as pattern or
b) Cleanup via ClassCleanupAttribute which can invoke asynchronous stuff.

I do think we have to go with the ClassCleanup and call DisposeAsync synchronously, but again I don't use MsTest.

[egil]

Just out of curiosity, can you try with xUnit instead. I don't think mstest has a synchronization context, and that might actually cause some problems (long shot).

[egil]

I tried to run it locally, and this is the exception I received after a few iteractions:

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Exception: An existing connection was forcibly closed by the remote host.
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Span`1 buffer)
   at System.Net.Sockets.NetworkStream.ReadByte()
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.

The active Test Run was aborted because the host process exited unexpectedly. Please inspect the call stack above, if available, to get more information about where the exception originated from.
The test running when the crash occurred:
FeuilleTemps.Interface.Pages.Horodateur.Edition.RechercheTests.CliquerSurResultat_FermeRecherche

This test may, or may not be the source of the crash.

Are the test opening collections to a database or something similar? It might be running out of connections if the connections are probably released after each test finishes.

Nope, inspected the memory dump and this is indeed related to #577.

[egil]

Took a more detailed look at you test code.

In EditerTestsBase you have two properties, Edition and Recherche, that you use multiple times in your test. However, because you write them like this:

protected IRenderedComponent<FeuilleTempsEdition> Edition => Horodateur.FindComponent<FeuilleTempsEdition>();

You are calling FindComponent everytime you access the property, which is not only wasteful, but probably not what you want.

I changed the top of EditerTestsBase to this:

    protected IRenderedComponent<FeuilleTempsEdition> Edition() 
        => Horodateur.FindComponent<FeuilleTempsEdition>();
    protected IGestionChronometre GestionChronometre { get; private set; }

    protected IRenderedComponent<ModalConfirmerQuitterEdition> ModalConfirmerQuitterEdition() =>
        Horodateur.FindComponent<ModalConfirmerQuitterEdition>();

    protected IRenderedComponent<FenetreRecherche> Recherche() 
        => Horodateur.FindComponent<FenetreRecherche>();

and the test to this:

    [TestMethod("Cliquer sur un résultat ferme le panneau de recherche")]
    public void CliquerSurResultat_FermeRecherche() {
        AfficherEdition();

        RechercheApi.RechercherProjet(default).ReturnsForAnyArgs(new RechercheAvecProfilResponse() {
            Items = { new() { Code = "X" } }
        });

        var edition = Edition();
        var recherche = Recherche();

        edition.WaitForState(() => edition.EditionSpinner() == null);
        edition.Projet().Click();
        recherche.WaitForState(() => recherche.Resultats().Count > 0);
        recherche.Resultats()[0].Click();
        recherche.WaitForAssertion(() => recherche.EstFermee());
    }

And now the test seems to be running without a problem (current at 75 iterations without fail, before it only lasted three iterations).

So that does solve your particular problem, but besides that, we do have a deadlock possibility if people use FindComponent inside a WaitFor..., and that is something I am looking at.

Thanks for the sample code, it will make it much easier for me to create a fix.

[RaphaelMarcouxCTRL]

Thank you for checking it out!

I also kept working on my side and concluded that the cause was indeed the use of CUT.FindX inside WaitForX.
I will inform my team to stop doing this.

For anyone who has such a problem, if you want to wait until an element has appeared or disapeared from the render tree, make use of the IRefreshableCollection.Refresh() method instead of calling find inside your WaitFor it looks like it was what fixed it for me.

Here's the code after modification :

`
public void CliquerSurResultat_FermeRecherche() {
AfficherEdition();

    RechercheApi.RechercherProjet(default).ReturnsForAnyArgs(new RechercheAvecProfilResponse() {
        Items = { new() { Code = "X" } }
    });

    var elements = Edition.FindAll("#edition-spinner");

    Edition.WaitForState(() => {
        elements.Refresh();
        return elements.Count == 0;
    });

    Edition.Projet().Click();

    elements = Edition.FindAll(".resultat");

    Recherche.WaitForState(() => {
        elements.Refresh();
        return elements.Count > 0;
    });

    Recherche.Resultats()[0].Click();

    var element = Recherche.Find("section[test-id=pop-up-recherche]");

    Recherche.WaitForAssertion(() => element.ClassList.Should().NotContain("visible"));
}

`

[egil]

You can also just specify auto refresh via the FindAll method, see https://bunit.dev/api/Bunit.RenderedFragmentExtensions.html#Bunit_RenderedFragmentExtensions_FindAll_Bunit_IRenderedFragment_System_String_System_Boolean_.

Also, there is a WaitForElements that you could also benefit from.

[RaphaelMarcouxCTRL]

Aaaaah I see! I missed that autoRefresh parameter earlier, thanks for pointing it out :)