Generator implementation for extractResponseFromChunkedData

[ibnesayeed]

I think we got a use case to implement a generator function to extract chunks from chunked encoded payload. This will enable iteration over chunks transparently. May be we want to revisit this when we move dechunking on indexing time.

[machawk1]

@ibnesayeed You are correct in that a generator probably would have been a better way to accomplish this. The current implementation (still in the indexer per #126) is fairly succinct.

https://github.com/oduwsdl/ipwb/blob/master/ipwb/replay.py#L268-L284

When we do move to using a generator, we ought to benchmark this implementation and the new one. Further, to reinforce your previous points, the dechunking code ought to exist in a script independent of both the replay and indexing code and reused as needed. A potential use case for this resides in still-chunked responses already existing in IPFS (through some other means than IPWB, perhaps).

[ibnesayeed]

We can perhaps utilize https://github.com/webrecorder/warcio to get decunked payload as buffered stream directly.