Hacky benchmark of history reindex #6055
Draft
+273
−112
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for oasisprotocol-oasis-core canceled.
|
You can set parallel writting by increasing number of writeWorkers. Note that if you set batchSize=1 and writeWorkers=1 you mimic current behaviour from main. I can confirm becnmarks match. Finally we also added more granular benchmarking of reads and writes.
We create a read worker pool per worker, that we use during batch processing. E.g. if you set writeWorkers=1, then batching is happening sequentially, but withing a batch we are fetching (reading) data in parallel.
martintomazic
force-pushed
the
martin/benchmark/history-reindex
branch
from
c24a60d to
5764e2d
Compare
martintomazic
changed the base branch from
master
to
martin/feature/batch-history-reindex-writes
martintomazic
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hacky Bench-marking only:
Warning
Ignore code quality etc, this is benchmark only. Corner case logic is also probably off sometimes, logs are also often just copied.
prune: strategy: nonebatching has minor bug if you are doing prunning.
How to benchmark
Change:
writeWorkersif you want to write in parallelbatchSizeif you want to batch`readWorkersif withing a batch processing you want to fetch consensus blocks and data in parallel.**
writeWorkers=1+batchSize=1should give you same benchmark as is on themain.**grepping for
history reindex0should output relevant logs.