Integration of Polars with Unity Catalog

[lukland]

Description

Context

In our organization, we are adopting Unity Catalog, managed by Databricks, as a unified governance solution for managing data access and compliance. Simultaneously, we utilize Polars as part of our medium-scale data processing stack, which operates independently of Databricks.

Problem Statement

Currently, there is no straightforward way to integrate Polars with Unity Catalog. This limitation makes it challenging for us to take advantage of the rich governance, access control, and metadata management capabilities of Unity Catalog while using Polars for data processing tasks.

[Bidek56]

I think this is a UC issue and there's a discussion on this already.

[ion-elgreco]

We will soon have unity client in delta-rs, polars could probably use this crate