Two TrioInternalErrors

[jtrakk]

The exception is raised when I try to do the threaded-apply trick followed by another function. For example, apply mario.testing.atools.wrapped_drop_falsy followed by map list. There's a lot of non-Trio code here, and I don't expect anybody to wade through it all for a diagnosis, but perhaps at least the tracebacks are useful. Anyway, here's what I do to make it fail:

git clone [email protected]:jtrakk/mario.git
cd mario
git checkout 805551473e6fdb4f3496233a5eae5283d0e1c143
virtualenv -p python3.7 venv
venv/bin/pip install -e '.[dev]'

There are two different TrioInternalError cases, depending on how i run it. The main code is in sync_apply.

$ venv/bin/mario   map int apply mario.testing.atools.wrapped_drop_falsy  map list <<<$'1\n2\n0\n3\n4'
Traceback (most recent call last):
  File "venv/bin/mario", line 11, in <module>
    load_entry_point('mario', 'console_scripts', 'mario')()
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 1164, in invoke
    return _process_result(rv)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 1102, in _process_result
    **ctx.params)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/user/Documents/mario/src/mario/cli.py", line 46, in cli_main
    app.main(pairs, **kwargs)
  File "/home/user/Documents/mario/src/mario/app.py", line 115, in main
    trio.run(functools.partial(async_main, pairs, **kwargs))
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1430, in run
    run_impl(runner, async_fn, args)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1579, in run_impl
    runner.task_exited(task, final_outcome)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1064, in task_exited
    raise TrioInternalError
trio.TrioInternalError
Exception ignored in: <async_generator object sync_apply at 0x7f42e8d84f28>
RuntimeError: async generator ignored GeneratorExit
Exception ignored in: <function Nursery.__del__ at 0x7f42ea2ab598>
Traceback (most recent call last):
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 638, in __del__
AssertionError: 

$ venv/bin/mario   map int apply 'mario.testing.atools.wrapped_drop_falsy'  apply mario.testing.atools.wrapped_max <<<$'1\n2\n0\n3\n4'
hello
Traceback (most recent call last):
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1430, in run
    run_impl(runner, async_fn, args)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1579, in run_impl
    runner.task_exited(task, final_outcome)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1051, in task_exited
    self.tasks.remove(task)
KeyError: <Task "<method-wrapper '__anext__' of async_generator object at 0x7fbb1d3150d0>" at 0x7fbb1d29fc88>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "venv/bin/mario", line 11, in <module>
    load_entry_point('mario', 'console_scripts', 'mario')()
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 1164, in invoke
    return _process_result(rv)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 1102, in _process_result
    **ctx.params)
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/user/Documents/mario/src/mario/cli.py", line 46, in cli_main
    app.main(pairs, **kwargs)
  File "/home/user/Documents/mario/src/mario/app.py", line 115, in main
    trio.run(functools.partial(async_main, pairs, **kwargs))
  File "/home/user/Documents/mario/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1436, in run
    ) from exc
trio.TrioInternalError: internal error in trio - please file a bug!

[zthompson47]

I tested the first case and was able to conjure up the original exception that led to the TrioInternalError:

Traceback (most recent call last):
  File "/Users/zach/venv/default/lib/python3.7/site-packages/trio/_core/_run.py", line 1069, in task_exited
    self.main_task_outcome.unwrap()
  File "/Users/zach/venv/default/lib/python3.7/site-packages/outcome/_sync.py", line 111, in unwrap
    raise captured_error
  File "/Users/zach/github/mario/src/mario/app.py", line 110, in async_main
    async for item in items:
  File "/Users/zach/github/mario/src/mario/traversals.py", line 202, in <genexpr>
    yield (await function(item) async for item in iterable)
  File "<string>", line 2, in _mario_runner
  File "/Users/zach/github/mario/src/mario/testing/atools.py", line 8, in drop_falsy
    for x in items:
  File "/Users/zach/github/mario/src/mario/traversals.py", line 95, in _pull_values_from_async_iterator
    yield in_trio.run(ait.__anext__)
  File "/Users/zach/venv/default/lib/python3.7/site-packages/trio/_threads.py", line 105, in run
    return self._do_it(self._run_cb, afn, *args)
  File "/Users/zach/venv/default/lib/python3.7/site-packages/trio/_threads.py", line 82, in _do_it
    "this is a blocking function; call it from a thread"
RuntimeError: this is a blocking function; call it from a thread

It looks like you're calling in_trio.run(ait.__anext__), a trio.BlockingTrioPortal method, from within the main Trio task. As discussed on the chat, BlockingTrioPortal.run() needs to run in a separate thread from Trio so it can block.

In general, it seems like Trio should be showing this traceback but here it gets swallowed. To show it, I edited trio._core._run.task_exited:

    def task_exited(self, task, outcome):
        ...
        elif task is self.init_task:
            # If the init task crashed, then something is very wrong and we
            # let the error propagate. (It'll eventually be wrapped in a
            # TrioInternalError.)
            outcome.unwrap()
            # the init task should be the last task to exit. If not, then
            # something is very wrong.
            if self.tasks:  # pragma: no cover
                # BEGIN ADDED CODE
                if self.main_task_outcome:
                    try:
                        self.main_task_outcome.unwrap()
                    except Exception as err:
                        raise TrioInternalError(err)
                # END ADDED CODE
                raise TrioInternalError
        ...

Do we want to add something like that to Trio?

[jtrakk]

It looks like you're calling in_trio.run(ait.anext), a trio.BlockingTrioPortal method, from within the main Trio task.

Ah you're completely right. Thanks!

[njsmith]

Do we want to add something like that to Trio?

It would certainly be good if we could give a better error here. I don't know if this is the right solution though. How are we getting into a situation where the init task is exiting early? It seems like there must be some other bug to let that happen...

[zthompson47]

At first glance, I can see that there's a trio.run_sync_in_worker_thread task which doesn't exit before the init task exits. That sync thread is calling back to the main Trio thread with a BlockingTrioPortal to send a message through a MemorySendChannel. Also, that sync thread is created in a task in the main Trio thread, via nursery.start_soon.

I haven't figured out the "how" yet, but I'll dig back in when I get some time.

[zthompson47]

Here's a boiled down version of a situation where the init task exits while the trio.run_sync_in_worker_thread task is still running. The Exception is not reported, but we get a trio.TrioInternalError.

import time
from functools import partial
import trio

def work_sync(portal, snd_ch):
    portal.run(snd_ch.send, "hello")
    time.sleep(1)  # Run longer that it takes to raise in main()

async def rcv_msgs():
    portal = trio.BlockingTrioPortal()
    snd_ch, rcv_ch = trio.open_memory_channel(0)

    async with trio.open_nursery() as nursery:
        nursery.start_soon(
            trio.run_sync_in_worker_thread,
            partial(work_sync, portal, snd_ch)
        )

        #async with rcv_ch:  # Creates error calling current_task()
                             # from CancelScope._close(), and init
                             # task still exits early
        async for i in rcv_ch:
            yield i

async def main():
    async for _ in rcv_msgs():
        raise Exception

trio.run(main)

It looks like making rcv_msgs an async generator is causing the problem. Using trio.run(rcv_msgs) and replacing the yield in rcv_msgs with raise Exception will generate a traceback for Exception instead of TrioInternalError (without the init task exiting early).

I'll be back in there later to figure out what's going on, but I thought I'd post this now in case anyone had any pointers on what to look for.

[njsmith]

     async with trio.open_nursery() as nursery:
         [...]
            yield i

Ah yeah this won't work – you can't yield inside a nursery block. That's a common trap. See #264.

From the discussion in chat, though, it sounded like it was concluded that this wasn't an example of yield-inside-nursery? (CC @oremanj)

[oremanj]

My conclusion was that this was not the most typical manifestation of misnesting of cancel scopes (which usually raises ValueError when removing a task from the cancel scope's _tasks member, or did before my recent changes to detect and recover from it). I'll need to dig some to figure out how we can detect the nursery-misnesting issue described here, but I'm on vacation the rest of this week, so it won't be immediate.

[zthompson47]

Yeah, both of the errrors that @jtrakk reported run code with a yield inside a nursery block (mario.traversals.sync_apply()).

[oremanj]

Closing this in favor of #1056, subpart "better support catching misnesting of nurseries".