DVC can not push file larger than 100MiB due to an upstream bug #10643
Labels
A: data-sync
Related to dvc get/fetch/import/pull/push
bug
Did we break something?
fs: oss
Related to the Alibaba Cloud OSS filesystem
Bug Report
Description
Since a bug in the
ossfs
dependency,dvc
can not push file larger than100MiB
to theoss
remote.Reproduce
The output report the file is pushed to remote which is false.
Expected
The file should be pushed to the oss, or if something unexpected happen, the CLI should report an error.
Environment information
Output of
dvc doctor
:Additional Information (if any):
The root cause of this issue is a bug in the
ossfs
package. According to the warning info, this line is used to upload large filehttps://github.com/fsspec/ossfs/blob/224e98868f32018fabacdac1eb5daddb16ce419c/src/ossfs/async_oss.py#L388
and finally this line(L159) will invoke the underlying method to perform uploading.
https://github.com/fsspec/ossfs/blob/224e98868f32018fabacdac1eb5daddb16ce419c/src/ossfs/async_oss.py#L159
when the
method_name
is , themethod(service, *args, **kwargs)
returns a future that is not awaited, which cause this problem.A small and quick fix is applying this patch:
The text was updated successfully, but these errors were encountered: