Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python): Support lakefs:// URI for delta scanner #20757

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ion-elgreco
Copy link
Contributor

@ion-elgreco ion-elgreco commented Jan 16, 2025

In deltalake 0.24 I've added support for LakeFS, this means users can use lakefs:// uri's to read and write delta tables in LakeFS.

Internally in delta-rs, we just construct an S3 ObjectStore and some other things which I won't bother you with :). Point is polars can safely read lakefs:// uri's as s3:// uris. If we find lakefs:// uri's we just rewrite them as s3://.

If you want I can also move this in to rust, but I want to mostly keep this for the delta reader/scanner.

@ion-elgreco ion-elgreco changed the title chore: replace lakefs uri with s3 for delta scanner chore: Replace lakefs uri with s3 for delta scanner Jan 16, 2025
@github-actions github-actions bot added internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars and removed title needs formatting labels Jan 16, 2025
Copy link

codecov bot commented Jan 16, 2025

Codecov Report

Attention: Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 79.74%. Comparing base (725c960) to head (4add192).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
py-polars/polars/io/delta.py 33.33% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #20757      +/-   ##
==========================================
- Coverage   79.74%   79.74%   -0.01%     
==========================================
  Files        1561     1561              
  Lines      222012   222015       +3     
  Branches     2530     2531       +1     
==========================================
- Hits       177049   177041       -8     
- Misses      44381    44391      +10     
- Partials      582      583       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ion-elgreco ion-elgreco changed the title chore: Replace lakefs uri with s3 for delta scanner chore(python): Replace lakefs uri with s3 for delta scanner Jan 17, 2025
@ion-elgreco
Copy link
Contributor Author

@nameexhaustion could you take a look? Can't seem to add reviewer requests myself : )

@nameexhaustion
Copy link
Collaborator

Can you confirm that you've tested it works? After that I think it's ready to merge.

I think it's ok to have it in the Python side for now, we can add this rewrite to the path expansion step on the rust side later if we want to add support for other I/O scan functions.

@nameexhaustion nameexhaustion changed the title chore(python): Replace lakefs uri with s3 for delta scanner feat(python): Support lakefs:// URI for delta scanner Jan 17, 2025
@github-actions github-actions bot added the enhancement New feature or an improvement of an existing feature label Jan 17, 2025
@nameexhaustion nameexhaustion removed rust Related to Rust Polars internal An internal refactor or improvement labels Jan 17, 2025
@ion-elgreco
Copy link
Contributor Author

@nameexhaustion yes it works :)
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants