Small improvements to import performance #131
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ings to make that useful
small tweaks, but it redunce time to import the listings on my machine from 40 minutes to 23
|
Just got home from work, running a comparison check of |
Live should not be true on this one. |
|
36 minutes to completion with this branch, 11.0.3 is still at 88% on the main listings file |
|
This branch: listings: ~35 minutes Total: ~37 minutes listings: ~46 minutes Total: ~49 minutes Total time savings: 12 minutes |
|
I'm going to push a couple changes, check I didn't break anything, and then do the merge. |
Have all the methods external to the class have matching naming convention
Don't use self.execute() or self.commit() in some cases and db.execute() or db.commit() in others, use db.* in all cases.
Need to include `self.dataPath` or it will always return False, which is not what we want.
|
Nice! | remove unused methods mmmmm - I love it when you talk dirty |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There's a bunch of linting/fixing first, which helped me spot some actual problems. After that there's some tidy-up of eddbplugin that made it easier for me to reason about, and enabled me to find a balance between building big lists in memory to attack the db with vs making better use of the dbs own inevitable cache anyway.
this reduces import time from 40 minutes to 23 minutes on my windows pc.
please run this branch on other hardware yourself before merging.