Can I merge download directories?

[jeff-geoff]

I have one download directory, let’s call it “Directory 1”.

several weeks ago, I moved it off my local machine and started a new, fresh download directory called “Directory 2”

Directory 2 is smaller than Directory 1 because some of my subscriptions expired over the last several weeks.

I want to combine Directory 1 and Directory 2 in to the same directory, let’s call it Directory 3, and continue scraping in to Directory 3.

I can compare and merge the directories just fine, but the databases will not reflect the contents.

How can I proceed to merge Directory 1 and Directory 2 in to Directory 3, and use Directory 3 as my new download directory?

[Nick05491]

I'm going to try and give you an answer since no one else has yet... I'd still like to hear from someone with a better understanding of the code and how the script functions.

I think you would want to manually merge the folders for each creator. And then delete the Metadata folder (or don't copy transfer one of them over from Directory 1 or 2). My understanding is that next time you scrape this user, the script will check for files that you already have downloaded and skip them so that they aren't downloaded again.

I think the way to avoid this going forward is to change the metadata to be stored elsewhere. I've seen mention of this in these discussions. Going to go looking for this now. I've also struggled with storage space, and have had to spread the accounts I've scraped across multiple drives. But just bought a new external drive and was able to consolidate them to one place. I hadn't deleted the metadata for each user to force it to be regenerated. Now I'm wondering if this is necessary after changing the directory location of the scraped account. Will update on what I learn.

[jeff-geoff]

I’ll try this with some old files. Btw the repo has moved to DIGITALCRIMINALS