Skip to content

Commit

Permalink
Let's try calculating column diffs
Browse files Browse the repository at this point in the history
  • Loading branch information
allejo committed Dec 19, 2023
1 parent 61d7f39 commit 5ebe70f
Showing 1 changed file with 26 additions and 2 deletions.
28 changes: 26 additions & 2 deletions .github/workflows/provider-map-jobs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,38 @@ jobs:
with open(geojson_file, 'w') as f:
json.dump(geojson, f, indent=2)
- name: Setup yq
uses: vegardit/gha-setup-yq@v1
with:
use-cache: true
version: 4.40.5

- name: Check for column updates
id: column-updates
run: |
curl -O https://raw.githubusercontent.com/cal-itp/data-infra/main/warehouse/models/mart/transit_database/_mart_transit_database.yml
columns_from_dbt=$(yq '.models[] | select(.name == "dim_mobility_mart_providers") | .columns[].name' _mart_transit_database.yml)
columns_from_repo=$(yq '.[].column' src/metadata/providers/dictionary.csv)
column_diff=$(diff <( printf '%s\n' "$columns_from_dbt" ) <( printf '%s\n' "$columns_from_repo" ))
echo "column_diff=$column_diff" >> "$GITHUB_OUTPUT"
- name: Create Pull Request
uses: peter-evans/create-pull-request@v5
with:
title: Provider Map Data Auto Update
body: |
It's that time again! The warehouse has delivered new data for us to use.
It's that time again! The warehouse has delivered new data for us to use. This is an automatic pull request created by the `provider-map-jobs.yml` workflow; it is triggered via a cron that runs every Sunday at midnight UTC.
## Changed Columns
These are columns that differ between the warehouse and the repository. If you see descrepancies here, please update the `src/metadata/providers/dictionary.csv` file to match the warehouse.
This is an automatic pull request created by the `provider-map-jobs.yml` workflow. It is triggered by a cron job that runs every Sunday at midnight UTC.
```diff
${{ steps.column-updates.outputs.column_diff }}}
```
commit-message: Auto-update provider data from warehouse
add-paths: |
src/metadata/providers/providers.csv
Expand Down

0 comments on commit 5ebe70f

Please sign in to comment.