Feature request: upload byte ranges

[OYTAdmin]

After a day of tinkering, it seems that background_downloader could be the perfect tool for doing multipart S3 uploads with HTTP PUT requests and presigned URLs. In a multipart upload, a file is uploaded in several parts (hence the name) and the final file is stitched together on the server side. From my experiments so far, background_downloader could be the only tool needed to do this if it could support startByte and endByte parameters in the UploadTask class.

It sure would be nice for my use case if this feature could be supported....

[781flyingdutchman]

Thanks for raising - this has certainly come up. The lack of a formal standard for what is confusingly also called multipart upload (different from the form/multi-part) makes it difficult to come up with a good implementation. That said, a good first step would be specifying a byte range, so the question is how best to do this. I would very much prefer not to introduce additional properties to the Task class, so my proposal would be to define a header that accomplishes this. The obvious candidate is Range, similar to how it is defined for GET requests (see here). The second decision is whether to pass the header on to server (i.e. leave it in place) or to remove it (and use it only to inform the client side that the request needs to upload only a portion of the specified file). The precedent we have is that for downloads it is possible to set the content length on the request side (e.g. if the server does not provide it) using the Known-Content-Length header, which we do not remove.

So, let me know if you are ok with 1) using the Range header to specify the desired range within the file to upload, and 2) to pass that header on to the server.

Another thing to be aware of is that because of the way uploads are done on iOS, we would need to make a copy of the sub-range of the original file into a new file, and update that, then delete it once the upload is complete. In practice this can of course lead to a significant incremental memory use, and it may iin fact turn out to be not possible on iOS.

[OYTAdmin]

I'd prefer additional properties on the task; a header is something that I envision would be forwarded to the server which would not be the case here. But I'd be happy with any solution. I think that developers will understand the iOS limitations you mention and find ways to work with it. If that becomes too much of an issue, perhaps the range uploads can be done serially rather than in parallel, so that only one range sub-file has to exist on the filesystem at any one time.

[781flyingdutchman]

Adding a property to the Task adds overhead in all the comms between app and plugin, which is why I don't prefer it, but I'll look into it. The fallback is to use the Range header and strip it before making the upload request, so it is not passed onto the server.

[781flyingdutchman]

Implemented in V8.6.0