Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Integration Testing Workflow #14

Merged
merged 11 commits into from
Sep 22, 2022
Merged

Fix Integration Testing Workflow #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8c2280a
Fix indentation
dfornika Sep 21, 2022
3c7fc72
Add logic to handle exit codes for timeout
dfornika Sep 21, 2022
ffa6258
Capture symlinks db in artifacts
dfornika Sep 22, 2022
ab2e141
Use tee to redirect to file
dfornika Sep 22, 2022
a5bda81
Create output dirs before running simulator
dfornika Sep 22, 2022
c51775e
Adjust timeouts
dfornika Sep 22, 2022
01c3ef7
Adjust workflow
dfornika Sep 22, 2022
bf91b3c
Add badge to README
dfornika Sep 22, 2022
74e5d5b
Can't collect symlinks in artifacts
dfornika Sep 22, 2022
b0ecee0
Update projects config
dfornika Sep 22, 2022
44b6d9c
Generate debug logs
dfornika Sep 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 15 additions & 8 deletions .github/workflows/integration_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,19 @@ jobs:
run: |
mkdir -p artifacts
- name: Setup Java
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '11'
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '11'
- name: Download Run Simulator
run: |
wget https://github.com/dfornika/illumina-run-simulator/releases/download/v0.1.0/illumina-run-simulator-0.1.0-standalone.jar -O illumina-run-simulator.jar
- name: Create simulated run output directories
run: |
mkdir -p artifacts/simulated_runs/{M00123,M00456,VH00123}/22
- name: Run Run Simulator
run: |
timeout 10 java -jar illumina-run-simulator.jar --config test/data/illumina-run-simulator/config.edn 2> >(tee artifacts/illumina-run-simulator.log.jsonl)
timeout 10 java -jar illumina-run-simulator.jar --config test/data/illumina-run-simulator/config.edn 2> >(tee artifacts/illumina-run-simulator.log.jsonl) || code=$?; if [[ $code -ne 124 && $code -ne 0 ]]; then exit $code; fi
- name: Setup Python
uses: actions/setup-python@v4
with:
Expand All @@ -41,12 +44,16 @@ jobs:
mkdir alembic/versions
alembic revision --autogenerate -m 'init'
alembic upgrade head
- name: Create symlink output directories
run: |
mkdir -p test_output/symlinks_by_project
- name: Run Auto Symlinker
run: |
timeout 30 auto-fastq-symlink --config test/data/config/integration_testing/config.json 2> >(artifacts/auto-fastq-symlink.log.jsonl)
timeout 15 auto-fastq-symlink --config test/data/config/integration_testing/config.json --log-level debug 2> >(tee artifacts/auto-fastq-symlink.log.jsonl) || code=$?; if [[ $code -ne 124 && $code -ne 0 ]]; then exit $code; fi
- name: Copy db to Artifacts
run: |
cp symlinks.db artifacts
- uses: actions/upload-artifact@v1
with:
name: Artifacts
path: artifacts


6 changes: 4 additions & 2 deletions .github/workflows/unit_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,10 @@ jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/setup-python@v4
- name: Checkout Repo
uses: actions/checkout@v1
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Create artifacts directory
Expand Down
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
[![Unit Tests](https://github.com/BCCDC-PHL/auto-fastq-symlink/actions/workflows/unit_tests.yml/badge.svg?branch=main)](https://github.com/BCCDC-PHL/auto-fastq-symlink/actions/workflows/unit_tests.yml)
[![Integration Tests](https://github.com/BCCDC-PHL/auto-fastq-symlink/actions/workflows/integration_tests.yml/badge.svg?branch=main)](https://github.com/BCCDC-PHL/auto-fastq-symlink/actions/workflows/integration_tests.yml)

# auto-fastq-symlink
Automated symlinking of illumina fastq files, organized by project.
Expand Down
6 changes: 3 additions & 3 deletions test/data/config/integration_testing/projects.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
project_id,fastq_symlinks_dir,excluded_runs_list,excluded_libraries_list,simplify_symlink_filenames
assay_development,artifacts/symlinks_by_project/assay_development,test/data/config/integration_testing/assay_development_excluded_runs.csv,test/data/config/integration_testing/assay_development_excluded_libraries.csv,False
viral_outbreak,artifacts/symlinks_by_project/viral_outbreak,test/data/config/integration_testing/viral_outbreak_excluded_runs.csv,test/data/config/integration_testing/viral_outbreak_excluded_libraries.csv,True
antibiotic_resistance,artifacts/symlinks_by_project/antibiotic_resistance,test/data/config/integration_testing/antibiotic_resistance_excluded_runs.csv,test/data/config/integration_testing/antibiotic_resistance_excluded_libraries.csv,True
assay_development,test_output/symlinks_by_project/assay_development,test/data/config/integration_testing/assay_development_excluded_runs.csv,test/data/config/integration_testing/assay_development_excluded_libraries.csv,False
viral_outbreak,test_output/symlinks_by_project/viral_outbreak,test/data/config/integration_testing/viral_outbreak_excluded_runs.csv,test/data/config/integration_testing/viral_outbreak_excluded_libraries.csv,True
antibiotic_resistance,test_output/symlinks_by_project/antibiotic_resistance,test/data/config/integration_testing/antibiotic_resistance_excluded_runs.csv,test/data/config/integration_testing/antibiotic_resistance_excluded_libraries.csv,True