Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use public direct links for object for S3 and Azure #761

Open
shcheklein opened this issue Dec 29, 2024 · 2 comments
Open

Use public direct links for object for S3 and Azure #761

shcheklein opened this issue Dec 29, 2024 · 2 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@shcheklein
Copy link
Member

Followup #755

Less critical of implementation since it affects only public, no credentials buckets and Studio teams. It works already for Google Storage since @dreadatour fixed it a while ago.

Public S3 and Azure client.url() code. Similar to GS that already has a check for anon in it - we need to generate and return direct URL to the cloud storage.

Make sure along the way:

  • Endpoint URLs are supported, especially for AWS
  • On the Studio side pass ms header to signed URL to get a public URL that actually works (see some SO discussions)
  • add tests
@shcheklein shcheklein added enhancement New feature or request bug Something isn't working labels Dec 29, 2024
@dreadatour
Copy link
Contributor

Quick note: I have checked AWS S3 and it returns public URL out of the box if no credentials found:

In [1]: from datachain.catalog import get_catalog

In [2]: catalog = get_catalog()

In [3]: catalog.signed_url('s3://fast-ai-nlp', 'ag_news_csv.tgz')
Out[3]: 'https://fast-ai-nlp.s3.amazonaws.com/ag_news_csv.tgz'

This URL is actually works: https://fast-ai-nlp.s3.amazonaws.com/ag_news_csv.tgz

We still need to check all possible options for S3 and Azure.

@shcheklein
Copy link
Member Author

Thing to check for S3 if it works for versioned files (when you pass version_id)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants