Full scope of "task work" and IORING_SETUP_DEFER_TASKRUN

[cmazakas]

Hey everyone, I'm trying to update my runtime to take advantage of this flag and I had a few questions.

       IORING_SETUP_DEFER_TASKRUN
              By default, io_uring will process all outstanding work at the end of any system call or thread interrupt. This can de‐
              lay the application from making other progress.  Setting this flag will hint to io_uring that it should defer work un‐
              til an io_uring_enter(2) call with the IORING_ENTER_GETEVENTS flag set. This allows the application to request work to
              run  just  before  it wants to process completions.  This flag requires the IORING_SETUP_SINGLE_ISSUER flag to be set,
              and also enforces that the call to io_uring_enter(2) is called from the same thread  that  submitted  requests.   Note
              that if this flag is set then it is the application’s responsibility to periodically trigger work (for example via any
              of the CQE waiting functions) or else completions may not be delivered.  Available since 6.1.

and also the passage here seem to imply that "task work" is pushed back until io_uring_enter(2) is called with a GETEVENTS flag.

I have code that relies on io_uring_submit(ring); to clear the SQ so that io_uring_get_sqe() can succeed. Am I supposed to replace these calls with io_uring_submit_and_get_events(ring)? I tried playing around with a minimal example and the OP_TIMEOUT but it seemed like io_uring_submit() triggered work just fine and the only thing I noticed was a difference in when CQEs populated the CQ.

When I use this flag, what actually gets delayed? What's the full scope of this change? One thing I did notice is that this flag seemed to delay submission of a socket() op I had prepared and I wound up actually catching a memory leak in some of my exception-based tests.

[isilence]

and also the passage here seem to imply that "task work" is pushed back until io_uring_enter(2) is called with a GETEVENTS flag.

I have code that relies on io_uring_submit(ring); to clear the SQ so that io_uring_get_sqe() can succeed. Am I supposed to replace these calls with io_uring_submit_and_get_events(ring)? I tried playing around with a minimal example and the OP_TIMEOUT but it seemed like io_uring_submit() triggered work just fine and the only thing I noticed was a difference in when CQEs populated the CQ.

The flag doesn't affect submissions, so submit should try to consume all nr_submit SQEs and queue up requests regardless ...

When I use this flag, what actually gets delayed? What's the full scope of this change? One thing I did notice is that this flag seemed to delay submission of a socket() op I had prepared and I wound up actually catching a memory leak in some of my exception-based tests.

... but the flag affect completion of requests. If a request completes "inline", i.e. while in the syscall in the submission path, the CQE will be available, DEFER_TASKRUN doesn't change that. If a request is executed asynchronously, however, let's say to complete the request and get the result you need to do GETEVENTS (or CQ waiting). Even more, you might need to do it multiple time for a single request without them posting CQEs (and even more true for multishots), e.g. to process a part of data, after which it'll continue waiting (short recv handling). Without the flag all that happens asynchronously from the userspace perspective.

Note, that and_get_events is literally wait(0), that's why there is no

wait_and_get_events();

Which should be just wait(). And this snippet ...

submit_and_get_events();
wait();

.. can be just

submit();
wait();
// or submit_and_wait();

submit_and_get_events() is usually used as an optimisation, e.g. when you know that there is probably a bunch of requests should've completed by the time the submit returns but you don't start waiting immediately (any time soon). But there are more inventive scenarios, e.g. when you never wait and don't really care about latency. You just submit everything and add get_events to get completions from requests from previous submissions.