You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Task for #765
Three notebooks that use mount paths to process data potentially need updating to work with Unity Catalog. Research and testing is required for this task. Currently each of the notebooks uses a mount path that is in ADLS to perform transformation/data engineering tasks.
DoD
Explore.py notebook works with Unity Catalog
Standardize.py notebook works with Unity Catalog
Transform.py notebook works with Unity Catalog
The text was updated successfully, but these errors were encountered:
So far these notebooks function as expected with Unity Catalog as the mounts are created properly in the setup.py
thesqlpro
changed the title
[Unity Catalog] - Change/test Standardize and Transform notebooks
[Unity Catalog] - Change/test Explore, Standardize and Transform notebooks
Dec 26, 2024
For explore.py no changes are needed. The mount points work fine as long as the ADF pipeline runs and stages the data in the proper paths.
For the other two notebooks, just a simple fix of adding USE CATALOG sensordata at the beginning of the notebook. Additionally to that, there needs to be some user permissions changes.
These are the permissions needed:
Because the SPN from ADF is going to be generated it will be difficult to put together the permissions needed. This is because the Data Control Language (DCL) is SQL based in the notebook and has some difficulties with dynamic SQL (having to inject the SPN somehow into the GRANT syntax for granting privileges). The APIs for setting permissions currently not supporting Unity Catalog actions (must be done through UI or SQL).
My solution is to grant these permissions to All account users since this catalog is only for testing purposes. This can be added to the setup.py (Sql) or the readme (via UI).
Task for #765
Three notebooks that use mount paths to process data potentially need updating to work with Unity Catalog. Research and testing is required for this task. Currently each of the notebooks uses a mount path that is in ADLS to perform transformation/data engineering tasks.
DoD
The text was updated successfully, but these errors were encountered: