Skip to content

Commit

Permalink
Shoryuken config (#1092)
Browse files Browse the repository at this point in the history
## Purpose
Current shoryuken workers sit idle waiting for work.

We were using groups and concurrency limits as a proxy for priority/weighted queue.

There are other mechanisms for that.
https://github.com/ruby-shoryuken/shoryuken/wiki/Polling-strategies

closes: _Add github issue that originated this PR_

## Approach
Changes the concurrency configuration for the shroyuken workers.

 - The default polling strategy is weighted round robin
 - Drops groups and individual concurrency limits and adds weights to the queues.
 - Adds option to set total concurrency for all queues as an environment variable `SHORYUKEN_CONCURRENCY`.
 - Default `SHORYUKEN_CONCURRENCY` is 15 (the same as the previous concurrency limit).


## Learning
Using the [shoryuken docs](https://github.com/ruby-shoryuken/shoryuken/wiki/shroyuken-options) as a reference.
Specifically, the [load balancing](https://github.com/ruby-shoryuken/shoryuken/wiki/Shoryuken-options#load-balancing), [concurrency](https://github.com/ruby-shoryuken/shoryuken/wiki/Shoryuken-options#concurrency) and [polling strategies](https://github.com/ruby-shoryuken/shoryuken/wiki/Polling-strategies#weightedroundrobin) sections


## Types of changes

- [ ] Bug fix (non-breaking change which fixes an issue)

- [x] New feature (non-breaking change which adds functionality)

- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Reviewer, please remember our [guidelines](https://datacite.atlassian.net/wiki/spaces/TEC/pages/1168375809/Pull+Request+Guidelines):

- Be humble in the language and feedback you give, ask don't tell.
- Consider using positive language as opposed to neutral when offering feedback. This is to avoid the negative bias that can occur with neutral language appearing negative.
- Offer suggestions on how to improve code e.g. simplification or expanding clarity.
- Ensure you give reasons for the changes you are proposing.
  • Loading branch information
jrhoads authored Jan 24, 2024
2 parents de8a420 + 3e30347 commit 813194b
Showing 1 changed file with 6 additions and 20 deletions.
26 changes: 6 additions & 20 deletions config/shoryuken.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,9 @@
concurrency: 15
concurrency: <%= ENV.fetch('SHORYUKEN_CONCURRENCY', 30) %>
delay: 0
pidfile: tmp/pids/shoryuken.pid
queues:
- lupo

groups:
import:
concurrency: 10
queues:
- lupo_import
import_other_doi:
concurrency: 10
queues:
- lupo_import_other_doi
transfer:
concurrency: 10
queues:
- lupo_transfer
background:
concurrency: 3
queues:
- lupo_background
- [lupo, 8]
- [lupo_import, 8]
- [lupo_import_other_doi, 8]
- [lupo_transfer, 4]
- [lupo_background, 2]

0 comments on commit 813194b

Please sign in to comment.