• Adding Scheduler HA
• Testing with 2.6.3

Current body of work:

Ansible Airflow TDP Extra

This role deploys the Apache Airflow release. In general it installs and/or configures:

• Airflow: Installs packaged airflow 2.2.5 or 2.6.2, and all its dependencies, through a virtual-environment and pip-install
• Scheduler: Responsible for monitoring and ensuring scheduled execution of all tasks. Decides where and when tasks are run.
• Webserver: A Flask server that serves the Airflow UI. Helps monitor, trigger and debug DAGs.
• Broker: Uses Redis. This Facilitates communication between the Airflow Scheduler and the Workers, handling task messages and their status updates.
• Executor: Uses Celery. Responsible for determining how tasks are run, in parallel or sequentially, locally or distributed.
• Flower: Used to monitor task progress and history in celery executors.
• Workers: Execute the tasks. They pick up and run tasks sent to the queue by the executor.
• Database (Metadata DB): Uses postgresql. Stores metadata about the state of tasks and workflows, and assists in recovery in case of failures.
• Dag Directory: Folder of DAG files. It is read by the scheduler and executor, and has to exist in every worker as well.
• Services: Creates services for web-server, scheduler, workers, redis, flower.

Prerequisites

• python3 and python3-pip installed on all nodes.
• Airflow version = 2.2.5 requires python >= 3.6 & Airflow version > 2.6.3 requires python >= 3.8
• Hadoop TDP release .tar.gz (hadoop_dist_file role variable) file available in files
• Groups airflow_webserver, airflow_scheduler, airflow_broker and airflow_worker defined in the Ansible hosts file
• Postegres user and databse to store metadata. This is done in prerequisites.
• Certificate files {{ fqdn }}.key and {{ fqdn }}.pem for every node available in files
• Admin access to a KDC with the realm, kadmin_principal and kadmin_password role vars provided
• Extra steps done for hadoop and hive configuration. This is currently added in this collection through tdp-cluster.yml.

Extra Work (Utils)

These tasks are intended for some extra tasks the user might or might not want to include in the installation. They include:

• Policies: Currently used to limit what a user (ie. tdp_user) can do when editing a dag.
• Roles: Used to limit what a user/group can view or do within a dag.
• Dag Directory structure: Working on dag-folder structure to handle groups/users.
• Creates connectors with impersonation capabilities for Hive, Spark and HDFS
• Dags examples that can be directly used on top of TDP (given the previous steps have been included)

Security implementations:

• Policies: Currently used to limit what a user (ie. tdp_user) can do when editing a dag.
• Roles: Used to limit what a user/group can view or do within a dag.
• Dag Directory structure: Working on dag-folder structure to handle groups/users.

Airflow 2.7 is out. Relevant changes:

• Cluster Activity UI: New overview page for the cluster. May render flower useless.
• Deferrable Mode Enablement: Enable for all deferrable tasks with 1 config setting.
• OpenLineage Integration: Now a built-in feature for reliable lineage publishing.
• Executors Moved into Providers: Improved bug-fix release process. May need reconfig in roles
• Required Provider Versions:
  • Celery provider 3.3.0+ for Celery executors.
  • Kubernetes provider 7.4.0+ for Kubernetes executor.

Inactivity