Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade Spark version to 15.4 LTS #896

Open
2 of 3 tasks
ydaponte opened this issue Nov 27, 2024 · 0 comments · May be fixed by #941
Open
2 of 3 tasks

Upgrade Spark version to 15.4 LTS #896

ydaponte opened this issue Nov 27, 2024 · 0 comments · May be fixed by #941
Assignees
@thesqlpro
Labels
e2e: parking-sensors-databricks P1 High Priority
Milestone
E2E - parking_sen...

Comments

@ydaponte
Copy link
Collaborator

ydaponte commented Nov 27, 2024 •
edited by thesqlpro
Loading

Spark 15.4 runtime in Azure Databricks doesn't allow to use dbfs anymore.
By upgrading the Spark version, the library import process needs to be upgraded to be imported either from the workspace or from an adls gen2 storage account.

DoD

  • Upgrade the Cluster deployment to use 15.4 LTS
  • Upgrade the Cluster template file
  • Modify deployment scripts to import ddo whl library from the Databricks workspace or from ADLS Gen 2 (open for discussions)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thesqlpro thesqlpro

Labels
e2e: parking-sensors-databricks P1 High Priority
Projects
None yet
Milestone
E2E - parking_sensor_databricks - V1
Development

Successfully merging a pull request may close this issue.

update for cluster config and template Azure-Samples/modern-data-warehouse-dataops
2 participants
@thesqlpro @ydaponte