Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.pyiceberg.yaml config files should be loaded from current dir instead of home folder #1333

Open
anentropic opened this issue Nov 18, 2024 · 5 comments · May be fixed by #1464
Open

.pyiceberg.yaml config files should be loaded from current dir instead of home folder #1333

anentropic opened this issue Nov 18, 2024 · 5 comments · May be fixed by #1464

Comments

@anentropic
Copy link

Feature Request / Improvement

https://py.iceberg.apache.org/configuration/#catalogs

There are three ways to pass in configuration:

  • Using the ~/.pyiceberg.yaml configuration file
  • Through environment variables
  • By passing in credentials through the CLI or the Python API

I don't want a global iceberg config for my whole machine, I want a file that lives in my project directory that could be potentially checked into git

there is already a search order and fall-back logic for home dir vs PYICEBERG_HOME env value

https://github.com/apache/iceberg-python/blob/main/pyiceberg/utils/config.py#L69-L74

...so it just needs to do the logical thing and try the current working dir before either of those

@kevinjqliu
Copy link
Contributor

kevinjqliu commented Nov 20, 2024

I don't want a global iceberg config for my whole machine, I want a file that lives in my project directory that could be potentially checked into git

that's an interesting use case. what do you think about this search order?

  • PYICEBERG_HOME
  • home directory
  • current directory

Or do you think the order should be reversed?

@anentropic
Copy link
Author

It should check the current dir first since it is the most specific

e.g. then you could use the global home dir file for most things but override that on a particular project with a project-local config file instead

@Fokko
Copy link
Contributor

Fokko commented Nov 20, 2024

I don't want a global iceberg config for my whole machine, I want a file that lives in my project directory that could be potentially checked into git

Aren't you afraid of checking in credentials?

@anentropic
Copy link
Author

true I probably wouldn't check it in

but I'd never put it in my home folder either way, it seems like a project file

@Fokko
Copy link
Contributor

Fokko commented Nov 20, 2024

It is inspired by how AWS handles credentials in ~/.aws/credentials with the different profiles.

That said, I'm not against adding the current directory as the last fallback. Are you interested in adding this to the codebase?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants