problem causing message Block the processing of good messages

[neildsouth]

Description

if a message causes a RequeueWithDelay() to be called, with the defaults this delay the NAK for 30 seconds, because the channel is setup like so

_channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);

this blocks the delivery of another message until the first one is cleared, meaning all processing will stop for potentially several minutes.

There are a few potential solutions that occur to me ATM.

1/ have the prefetch count a config item, but depending on the c# rabbitMq SDK this will need to be tested to ensure it doesnt introduce threading issues.

2/ use the newish plugin for rabbit that allow exchanges to have a message delivery delay, this is set in the message header, this would involve acking the message before a resend with the header set for delay, + setting the retry/failed header count. again this would need testing

3/ like the above you could just ack the message and then resend after a Task.Delay() setting the right message header for the retry/failed count. <--- this doesn't work, Ive tried changing the message header but x-delivery-count isnt use by rabbitMq for the tracking.

Steps to reproduce

publish a message that causes a RequeueWithDelay() the publish one that will not. the second is not delivered until the first is DeadLettered.

Expected behavior

The second should be picked up immediately and processed before the first is on its second round.

Actual behavior

A potential large delay in processing the good message.