-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(python): Support lakefs://
URI for delta scanner
#20757
base: main
Are you sure you want to change the base?
feat(python): Support lakefs://
URI for delta scanner
#20757
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #20757 +/- ##
==========================================
- Coverage 79.74% 79.74% -0.01%
==========================================
Files 1561 1561
Lines 222012 222015 +3
Branches 2530 2531 +1
==========================================
- Hits 177049 177041 -8
- Misses 44381 44391 +10
- Partials 582 583 +1 ☔ View full report in Codecov by Sentry. |
@nameexhaustion could you take a look? Can't seem to add reviewer requests myself : ) |
Can you confirm that you've tested it works? After that I think it's ready to merge. I think it's ok to have it in the Python side for now, we can add this rewrite to the path expansion step on the rust side later if we want to add support for other I/O scan functions. |
lakefs://
URI for delta scanner
@nameexhaustion yes it works :) |
In deltalake 0.24 I've added support for LakeFS, this means users can use
lakefs://
uri's to read and write delta tables in LakeFS.Internally in delta-rs, we just construct an S3 ObjectStore and some other things which I won't bother you with :). Point is polars can safely read
lakefs://
uri's ass3://
uris. If we find lakefs:// uri's we just rewrite them as s3://.If you want I can also move this in to rust, but I want to mostly keep this for the delta reader/scanner.