[OpenLineage] Added Openlineage support for DatabricksCopyIntoOperator

[rahul-madaan]

---

This PR adds support for DatabricksCopyIntoOperator (/providers/databricks/operators/databricks_sql.py)
Taking reference from CopyFromExternalStageToSnowflakeOperator which already has OL support.

tested using this DAG - click to open

"""
Example DAG demonstrating the usage of DatabricksCopyIntoOperator with OpenLineage support.
"""

import logging
logging.getLogger('databricks.sql').setLevel(logging.DEBUG)

from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.databricks.operators.databricks_sql import DatabricksCopyIntoOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 0,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'databricks_copy_into_example',
    default_args=default_args,
    description='Example DAG for DatabricksCopyIntoOperator with OpenLineage',
    schedule_interval=None,
    schedule=None,
    start_date=datetime(2024, 12, 13),
    catchup=False,
    tags=['example', 'databricks', 'openlineage'],
) as dag:

    # Example with S3
    copy_from_s3 = DatabricksCopyIntoOperator(
        task_id='copy_from_s3',
        databricks_conn_id='databricks_default',
        table_name='wide_world_importers.astronomer_assets.sample',
        file_location='s3a://kreative360/yoyo/sample.csv',
        file_format='CSV',
        format_options={
            "header": "true",
            "inferSchema": "true",
            "delimiter": ","
        },
        copy_options={
            "force": "true",
            "mergeSchema": "true"
        },
        http_path='/sql/1.0/warehouses/ca43e87568a0b22e',
        credential={
            "AWS_ACCESS_KEY": "<redacted>",
            "AWS_SECRET_KEY": "<redacted>",
            "AWS_SESSION_TOKEN": "<redacted>",
            "AWS_REGION": "ap-south-1"
        }
    )

    # Example with Azure Blob Storage using wasbs protocol
    copy_from_azure = DatabricksCopyIntoOperator(
        task_id='copy_from_azure',
        databricks_conn_id='databricks_default',  
        table_name='wide_world_importers.astronomer_assets.sample',
        file_location='wasbs://[email protected]/sample.csv',
        file_format='CSV',
        # Using Azure storage credential
        credential={
            "AZURE_SAS_TOKEN": "<redacted>", # Replace with actual SAS token
        },
        format_options={
            "header": "true",
            "inferSchema": "true"
        },
        copy_options={
            "force": "true",
            "mergeSchema": "true"
        },
        http_path='/sql/1.0/warehouses/ca43e87568a0b22e'
    )

    # Example with GCS
    copy_from_gcs = DatabricksCopyIntoOperator(
        task_id='copy_from_gcs',
        databricks_conn_id='databricks_default',
        table_name='wide_world_importers.astronomer_assets.sample',
        file_location='gs://kreative360/yoyo/sample.csv',
        file_format='CSV',
        format_options={
            "header": "true",
            "inferSchema": "true",
            "delimiter": ","
        },
        copy_options={
            "force": "true",
            "mergeSchema": "true"
        },
        http_path='/sql/1.0/warehouses/ca43e87568a0b22e',
    )

    [copy_from_s3, copy_from_azure, copy_from_gcs]

note - tests have been performed using s3 object on aws only, other cloud providers (azure and gcs) have been tested only using FAIL events.

OL events:

• aws(s3) - aws.json
• azure - azure.json
• gcs - gcs.json

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[rahul-madaan]

@kacpermuda @potiuk could you please take a look at the PR and approve?