A modular rewrite featuring numerous enhancements and bug fixes #3
Conversation
This commit also gives more consideration to rate limiting and fixes some bugs like posts on meta sites weren't being backed up, as well as the issue at mhdadk#1. Filters are now baked in, following the official doc's recommendation. A few CLI options are also added.
|
Hey @9ao9ai9ar, thank you very much for this PR! It seems like you really cleaned up the code. Adding tests was an especially nice touch. Before merging this PR, because it is quite large (49 files changed), I'd like to review it in more detail to make sure it is OK. Unfortunately, I'm not sure when I will have the time in the near future to do this. I'm going to keep this PR open until I do so. Nevertheless, I wanted to write to you to give you some positive feedback on your work. |
|
Thanks for your words of encouragement. I will be pushing out minor commits in the meantime, like updating the README and upgrading dependencies (Pydantic v2.9.0 is just around the corner, promising to fix the high memory consumption). I do have an idea to improve the output format, but I will withhold work on that until next year. The idea is to first move all non-body fields of the posts into the (non-standard but widely supported) YAML metadata section at the beginning of the Markdown files. Secondly, individual answers should probably be separated into individual files to keep them as close to the original as possible without modifying the Markdown content by inserting the section markers. Lastly, the licenses of the posts (currently there are 3 versions of CC in use) should be included and linked. I can open an issue if you're interested in further discussion. |
|
All these changes sounds good to me. I can have a look at them as well when you push them (and when I have some more time). |
1. Fix filter. 2. Add `--no-meta` option. 3. Update README. 4. Other minor adjustments.
9ao9ai9ar
changed the title
9ao9ai9ar
changed the title
The only downsides of my rewrite that I can think of are the higher memory usage (by about 50%) and increased dependencies.