Merge branch 'main' into hshahconsulting-add-tags-support

.github/workflows/tagging.yml

@@ -0,0 +1,42 @@
+name: tagging
+
+on:
+  workflow_dispatch:
+  # Enable for automatic tagging
+  #schedule:
+  #  - cron: '0 0 * * TUE'
+
+# Ensure that only a single instance of the workflow is running at a time.
+concurrency: 
+  group: "tagging"
+
+
+jobs:
+  tag:
+    environment: "release-is"
+    runs-on:
+      group: databricks-deco-testing-runner-group
+      labels: ubuntu-latest-deco
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Generate GitHub App Token
+        id: generate-token
+        uses: actions/create-github-app-token@v1
+        with:
+          app-id: ${{ secrets.DECO_SDK_TAGGING_APP_ID }}
+          private-key: ${{ secrets.DECO_SDK_TAGGING_PRIVATE_KEY }}
+
+      # NOTE: actions/create-github-app-token automatically masks the output token,
+      # so the token is not exposed in the logs even as part of the remote.
+      - name: Set up Git configuration
+        run: |
+          git config user.name "Databricks SDK Release Bot"
+          git remote set-url origin https://x-access-token:${{ steps.generate-token.outputs.token }}@github.com/${{ github.repository }}.git
+
+      - name: Run Deco Taggging script
+        run: |
+          python tagging.py

.package.json

@@ -0,0 +1,5 @@
+{
+    "version": {
+        "common/version.go": "version = \"$VERSION\""
+    }
+}

.release_metadata.json

@@ -0,0 +1,3 @@
+{
+    "timestamp": "2025-01-16 13:55:53"
+}

CHANGELOG.md

@@ -1,5 +1,66 @@
 # Version changelog
 
+## Release v1.63.1
+
+### Exporter
+
+ * Refactor jobs implementation to use only Go SDK
+
+### Internal Changes
+
+ * Introduced new automated tagging workflow. At the time of this commit,
+   the workflow must be run manually.
+
+
+## [Release] Release v1.63.0
+
+### New Features and Improvements
+
+ * Add `fallback` to `databricks_external_location` ([#4372](https://github.com/databricks/terraform-provider-databricks/pull/4372)).
+
+
+### Bug Fixes
+
+ * Send only what is required in Update of `databricks_credential` ([#4349](https://github.com/databricks/terraform-provider-databricks/pull/4349)).
+
+
+### Documentation
+
+ * Add `MANAGE` privilege to the relevant UC resources ([#4371](https://github.com/databricks/terraform-provider-databricks/pull/4371)).
+ * Add a note about schema evolution and `databricks_sql_table` ([#4352](https://github.com/databricks/terraform-provider-databricks/pull/4352)).
+ * Add description of `clean_rooms_notebook_task` in `databricks_job` ([#4368](https://github.com/databricks/terraform-provider-databricks/pull/4368)).
+ * Add docs for cluster attributes for clusters assigned to groups ([#4359](https://github.com/databricks/terraform-provider-databricks/pull/4359)).
+ * Add note about use of `azurerm_databricks` for enhanced security settings ([#4393](https://github.com/databricks/terraform-provider-databricks/pull/4393)).
+ * Clarify views sharing in `databricks_share` ([#4378](https://github.com/databricks/terraform-provider-databricks/pull/4378)).
+ * Document job edit_mode ([#4354](https://github.com/databricks/terraform-provider-databricks/pull/4354)).
+
+
+### Internal Changes
+
+ * Auto generated customizable schemas ([#4356](https://github.com/databricks/terraform-provider-databricks/pull/4356)).
+ * Disable `terraform fmt` in exporter to workaround build issues ([#4394](https://github.com/databricks/terraform-provider-databricks/pull/4394)).
+ * Fixed redundant optional owners in share resource and bumped Go SDK ([#4396](https://github.com/databricks/terraform-provider-databricks/pull/4396)).
+ * Load auto-generated resources and data sources ([#4367](https://github.com/databricks/terraform-provider-databricks/pull/4367)).
+ * Migrate workflows that need write access to use hosted runners ([#4377](https://github.com/databricks/terraform-provider-databricks/pull/4377)).
+ * Remove TF annotations on TFSDK models ([#4383](https://github.com/databricks/terraform-provider-databricks/pull/4383)).
+ * Restart Cluster before Library Installation ([#4384](https://github.com/databricks/terraform-provider-databricks/pull/4384)).
+
+
+### Dependency Updates
+
+ * Bump github.com/zclconf/go-cty from 1.15.1 to 1.16.0 ([#4374](https://github.com/databricks/terraform-provider-databricks/pull/4374)).
+ * Bump github.com/zclconf/go-cty from 1.16.0 to 1.16.1 ([#4397](https://github.com/databricks/terraform-provider-databricks/pull/4397)).
+
+
+### Exporter
+
+ * Add exporting of `for_each_task` in jobs ([#4347](https://github.com/databricks/terraform-provider-databricks/pull/4347)).
+ * Add references for missing permissions/grants types ([#4390](https://github.com/databricks/terraform-provider-databricks/pull/4390)).
+ * Support for group assigned clusters ([#4361](https://github.com/databricks/terraform-provider-databricks/pull/4361)).
+ * Use Go SDK to list clusters ([#4385](https://github.com/databricks/terraform-provider-databricks/pull/4385)).
+ * Use `Workspace.Export` from Go SDK for notebooks/files ([#4391](https://github.com/databricks/terraform-provider-databricks/pull/4391)).
+
+
 ## [Release] Release v1.62.1
 
 ### Bug Fixes

NEXT_CHANGELOG.md

@@ -0,0 +1,14 @@
+# NEXT CHANGELOG
+
+## Release v1.64.0
+
+### New Features and Improvements
+
+### Bug Fixes
+
+### Documentation
+
+### Exporter
+
+### Internal Changes
+

common/version.go

@@ -3,7 +3,7 @@ package common
 import "context"
 
 var (
-	version = "1.62.1"
+	version = "1.63.1"
 	// ResourceName is resource name without databricks_ prefix
 	ResourceName contextKey = 1
 	// Provider is the current instance of provider

docs/guides/experimental-exporter.md

@@ -163,7 +163,7 @@ For security reasons, [databricks_secret](../resources/secret.md) cannot contain
 To speed up export, Terraform Exporter performs many operations, such as listing & actual data exporting, in parallel using Goroutines.  Built-in defaults control the parallelism, but it's also possible to tune some parameters using environment variables specific to the exporter:
 
 * `EXPORTER_WS_LIST_PARALLELISM` (default: `5`) controls how many Goroutines are used to perform parallel listing of Databricks Workspace objects (notebooks, directories, workspace files, ...).
-* `EXPORTER_DIRECTORIES_CHANNEL_SIZE` (default: `100000`) controls the channel's capacity when listing workspace objects. Please ensure that this value is big enough (greater than the number of directories in the workspace; default value should be ok for most cases); otherwise, there is a chance of deadlock.
+* `EXPORTER_DIRECTORIES_CHANNEL_SIZE` (default: `300000`) controls the channel's capacity when listing workspace objects. Please ensure that this value is big enough (greater than the number of directories in the workspace; default value should be ok for most cases); otherwise, there is a chance of deadlock.
 * `EXPORTER_DEDICATED_RESOUSE_CHANNELS` - by default, only specific resources (`databricks_user`, `databricks_service_principal`, `databricks_group`) have dedicated channels - the rest are handled by the shared channel.  This is done to prevent throttling by specific APIs.  You can override this by providing a comma-separated list of resources as this environment variable.
 * `EXPORTER_PARALLELISM_NNN` - number of Goroutines used to process resources of a specific type (replace `NNN` with the exact resource name, for example, `EXPORTER_PARALLELISM_databricks_notebook=10` sets the number of Goroutines for `databricks_notebook` resource to `10`).  There is a shared channel (with name `default`) for handling of resources for which there are no dedicated channels - use `EXPORTER_PARALLELISM_default` to increase its size (default size is `15`).   Defaults for some resources are defined by the `goroutinesNumber` map in `exporter/context.go` or equal to `2` if there is no value.  *Don't increase default values too much to avoid REST API throttling!*
 * `EXPORTER_DEFAULT_HANDLER_CHANNEL_SIZE` is the size of the shared channel (default: `200000`). You may need to increase it if you have a huge workspace.

docs/guides/unity-catalog-azure.md

@@ -152,6 +152,12 @@ resource "azurerm_role_assignment" "ext_storage" {
   role_definition_name = "Storage Blob Data Contributor"
   principal_id         = azurerm_databricks_access_connector.ext_access_connector.identity[0].principal_id
 }
+
+resource "azurerm_role_assignment" "ext_storage" {
+  scope                = azurerm_storage_account.ext_storage.id
+  role_definition_name = "Storage Queue Data Contributor"
+  principal_id         = azurerm_databricks_access_connector.ext_access_connector.identity[0].principal_id
+}
 ```
 
 Then create the [databricks_storage_credential](../resources/storage_credential.md) and [databricks_external_location](../resources/external_location.md) in Unity Catalog.
@@ -247,9 +253,9 @@ resource "databricks_grants" "things" {
 
 ## Configure Unity Catalog clusters
 
-To ensure the integrity of ACLs, Unity Catalog data can be accessed only through compute resources configured with strong isolation guarantees and other security features. A Unity Catalog [databricks_cluster](../resources/cluster.md) has a  ‘Security Mode’ set to either **User Isolation** or **Single User**.
+To ensure the integrity of ACLs, Unity Catalog data can be accessed only through compute resources configured with strong isolation guarantees and other security features. A Unity Catalog [databricks_cluster](../resources/cluster.md) has the access mode set to either **Shared** or **Single User**.
 
-- **User Isolation** clusters can be shared by multiple users, but has certain [limitations](https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog)
+- **Shared** clusters can be shared by multiple users, but has certain [limitations](https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog)
 
 ```hcl
 data "databricks_spark_version" "latest" {

docs/guides/unity-catalog-gcp.md

@@ -236,9 +236,9 @@ resource "databricks_grants" "things" {
 
 ## Configure Unity Catalog clusters
 
-To ensure the integrity of ACLs, Unity Catalog data can be accessed only through compute resources configured with strong isolation guarantees and other security features. A Unity Catalog [databricks_cluster](../resources/cluster.md) has a  ‘Security Mode’ set to either **User Isolation** or **Single User**.
+To ensure the integrity of ACLs, Unity Catalog data can be accessed only through compute resources configured with strong isolation guarantees and other security features. A Unity Catalog [databricks_cluster](../resources/cluster.md) has the access mode set to either **Shared** or **Single User**.
 
-- **User Isolation** clusters can be shared by multiple users, but has certain [limitations](https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog)
+- **Shared** clusters can be shared by multiple users, but has certain [limitations](https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog)
 
 ```hcl
 data "databricks_spark_version" "latest" {

docs/guides/unity-catalog.md

@@ -35,7 +35,7 @@ To get started with Unity Catalog, this guide takes you through the following hi
 
 ## Provider initialization
 
-Initialize [provider with `mws` alias](https://www.terraform.io/language/providers/configuration#alias-multiple-provider-configurations) to set up account-level resources. See [provider authentication](../index.md#authenticating-with-service-principal) for more details.
+Initialize [provider with `mws` alias](https://www.terraform.io/language/providers/configuration#alias-multiple-provider-configurations) to set up account-level resources. See [provider authentication](../index.md#authenticating-with-databricks-managed-service-principal) for more details.
 
 ```hcl
 terraform {
@@ -343,9 +343,9 @@ resource "databricks_grants" "things" {
 
 ## Configure Unity Catalog clusters
 
-To ensure the integrity of ACLs, Unity Catalog data can be accessed only through compute resources configured with strong isolation guarantees and other security features. A Unity Catalog [databricks_cluster](../resources/cluster.md) has a  ‘Security Mode’ set to either **User Isolation** or **Single User**.
+To ensure the integrity of ACLs, Unity Catalog data can be accessed only through compute resources configured with strong isolation guarantees and other security features. A Unity Catalog [databricks_cluster](../resources/cluster.md) has the access mode set to either **Shared** or **Single User**.
 
-- **User Isolation** clusters can be shared by multiple users, but has certain [limitations](https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog)
+- **Shared** clusters can be shared by multiple users, but has certain [limitations](https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog)
 
 ```hcl
 data "databricks_spark_version" "latest" {

docs/resources/sql_permissions.md

@@ -48,7 +48,7 @@ resource "databricks_sql_permissions" "foo_table" {
 
 ## Argument Reference
 
-* `cluster_id` - (Optional) Id of an existing [databricks_cluster](cluster.md), where the appropriate `GRANT`/`REVOKE` commands are executed. This cluster must have the appropriate data security mode (`USER_ISOLATION` or `LEGACY_TABLE_ACL` specified). If no `cluster_id` is specified, a single-node TACL cluster named `terraform-table-acl` is automatically created.
+* `cluster_id` - (Optional) Id of an existing [databricks_cluster](cluster.md), where the appropriate `GRANT`/`REVOKE` commands are executed. This cluster must have the appropriate access mode (`USER_ISOLATION` or `LEGACY_TABLE_ACL` specified). If no `cluster_id` is specified, a single-node TACL cluster named `terraform-table-acl` is automatically created.
 
 ```hcl
 resource "databricks_sql_permissions" "foo_table" {