Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perform dechunking at indexing time #592

Closed
ibnesayeed opened this issue Oct 18, 2018 · 3 comments
Closed

Perform dechunking at indexing time #592

ibnesayeed opened this issue Oct 18, 2018 · 3 comments
Labels
enhancement ipwb indexer ipwb replay

Comments

@ibnesayeed
Copy link
Member

Currently we are dechunking responses at replay time which means we perform the same task repeatedly as many times as a resource is requested and also losing the potential deduplication benefits. We should move this functionality in the indexer before pushing the content to the IPFS.
@machawk1
Copy link
Member

It might be good to extract extractResponseFromChunkedData() from replay.py, move it to util, and reuse the same method from both the indexer and replay.

@machawk1
Copy link
Member

Also, this is a duplicate of #126. Closing.

@ibnesayeed
Copy link
Member Author

... and reuse the same method from both the indexer and replay.

Unless we want to play nice with already indexed (and pushed to the IPFS) records, we do not need to reuse it from the replay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement ipwb indexer ipwb replay
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ibnesayeed @machawk1