Update R #242

alexvpickering · 2022-06-02T03:37:26Z

Description

Update R to 4.2 using included script to minimize package updates as much as possible.

https://ui-alex-upgrade-r-p242-w274.scp-staging.biomage.net/

Notes:

updated Seurat to make RunUMAP(..., umap.method = 'umap-learn') reproducible
- see umap not set up to be reproducible in all methods satijalab/seurat#5194
- NOTE: our embeddings still differ with sample order
move to rocker/r-ver as base image (see advantages in link)
only install system runtime dependencies for final image (smaller than build deps)
general minimization of layer sizes to keep final image relatively light (see below)
- delete renv download cache
- strip debug code from compiled shared libraries
- further reductions in size may be possible with alpine based r-hub/r-minimal but I couldn't get BiocParallel to compile.
reverted to default BLAS/LAPACK: seems to be working correctly with Seurat::FindIntegrationAnchors
- test with more datasets using Seurat integration
- ask Oliver or look on slack to find dataset that caused issue

Image sizes differences:

Worker (↓ ~ 50%):

Pipeline (↓ ~ 40%):

Details

Integration test branch

master

Merge checklist

Your changes will be ready for merging after all of the steps below have been completed.

Code updates

Have best practices and ongoing refactors being observed in this PR

Migrated any selector / reducer used to the new format.

Manual/unit testing

Tested changes using InfraMock locally or no tests required for change, e.g. Kubernetes chart updates.
Validated that current unit tests for code work as expected and are sufficient for code coverage or no unit tests required for change, e.g. documentation update.
Unit tests written or no unit tests required for change, e.g. documentation update.

Integration testing

You must check the box below to run integration tests on the latest commit on your PR branch.
Integration tests have to pass before the PR can be merged. Without checking the box, your PR
will not pass the required status checks for merging.

Started end-to-end tests on the latest commit.

Documentation updates

Relevant Github READMEs updated or no GitHub README updates required.
Relevant Wiki pages created/updated or no Wiki updates required.

Optional

Staging environment is unstaged before merging.
Photo of a cute animal attached to this PR.

codecov · 2022-06-03T17:24:55Z

Codecov Report

Merging #242 (81b062e) into master (be5c36f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #242   +/-   ##
=======================================
  Coverage   76.55%   76.55%           
=======================================
  Files          19       19           
  Lines        1557     1557           
=======================================
  Hits         1192     1192           
  Misses        365      365

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be5c36f...81b062e. Read the comment docs.

gerbeldo

I have some questions because I'm curious about a few things, but I tested everything I could think of and everything worked as expected. The platform locally feels snappier (when starting).

I think it's good to go!

gerbeldo · 2022-06-27T15:22:10Z

pipeline-runner/setup/sysdeps_build_debian.txt

+liblzma-dev
+libbz2-dev


Why are we adding these? they are compression libraries right?

gerbeldo · 2022-06-27T15:30:36Z

pipeline-runner/Dockerfile

 RUN R -e "install.packages('remotes', repos = c(CRAN = 'https://cloud.r-project.org'))" && \
-    R -e "remotes::install_github('rstudio/[email protected]')"
+    R -e "remotes::install_github('rstudio/[email protected]')" && \
+    R -e "renv::init(bare = TRUE, settings = list(use.cache = FALSE))"


why do we not use the renv cache? I suppose when building the image there is no cache available. but what happens when rebuilding locally?

gerbeldo · 2022-06-27T15:35:28Z

pipeline-runner/Dockerfile

+    R -e 'root <- renv::paths$root(); unlink(root, recursive = TRUE)' && \
+    strip --strip-debug $RENV_LIB/*/libs/*.so
+
+RUN R -e "renv::deactivate()"


why is this necessary?

gerbeldo · 2022-06-27T15:44:00Z

pipeline-runner/Dockerfile

+    pip install PyYAML==3.10 && \
+    pip install awscli==1.25.0


why do need pyYAML? is it a dep of awscli?

gerbeldo · 2022-06-27T15:45:36Z

pipeline-runner/Dockerfile

+RUN echo "python3-pip" >> sysdeps_run.txt && \
+    echo "wget" >> sysdeps_run.txt && \
+    cat sysdeps_run.txt | xargs ./install_debian_packages.sh && \
+    rm -rf *


I do not understand this docker magic. what does this remove?

gerbeldo · 2022-06-27T15:46:33Z

pipeline-runner/Dockerfile

 ENTRYPOINT ["Rscript", "init.R"]


 # ---------------------------------------------------
 # DEVELOPMENT BUILD
 # ---------------------------------------------------
-FROM builder AS dev
+FROM common AS dev

 # install Radian for interactive R shell
 # also install watchdog to automatically restart
 # when source files change
 RUN pip install -U jedi radian PyYAML watchdog[watchmedo]


isn't pyYAML already installed?

alexvpickering added 11 commits June 1, 2022 17:55

upgrade renv

56e6c4a

init R deps update

c45acf1

update Dockerfile to R 4.2

acc65b2

sync renv.lock.init

8e5dedf

fix awscli install

453a9a0

roxygenize

56ded89

remove renv artifacts

f387ea6

simplify cleanup

7c49742

remove redundancy

1d4f490

add dockerignore

db17895

add comment

b44adba

alexvpickering mentioned this pull request Jun 3, 2022

Update R hms-dbmi-cellenics/worker#274

Merged

9 tasks

alexvpickering added 8 commits June 5, 2022 19:46

optimize dockerfile

b8ff6ab

update fs and Seurat

98c469d

Merge branch 'master' into update-R

766f7ca

update init lockfile

4a4bba6

sync lockfiles

91cd130

fix copy

4792d7b

FastMNN requires remotes; don't delete R packages

85097ba

clean setup files

3126741

alexvpickering marked this pull request as ready for review June 8, 2022 21:15

alexvpickering added 4 commits June 8, 2022 14:17

fix dockerignore

dcfd6a2

fix covr path

35e9dc3

fix typo

a07c42d

Merge branch 'master' into update-R

7982ef3

alexvpickering requested a review from gerbeldo June 9, 2022 18:38

avoid blanket upgrades

85c6006

alexvpickering requested a review from kafkasl June 10, 2022 23:26

Merge branch 'master' into update-R

81b062e

gerbeldo approved these changes Jun 27, 2022

View reviewed changes

gerbeldo merged commit 33cfada into master Jun 27, 2022

gerbeldo deleted the update-R branch June 27, 2022 18:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update R #242

Update R #242

alexvpickering commented Jun 2, 2022 •

edited by gerbeldo

Loading

codecov bot commented Jun 3, 2022 •

edited

Loading

gerbeldo left a comment

gerbeldo Jun 27, 2022

gerbeldo Jun 27, 2022

gerbeldo Jun 27, 2022

gerbeldo Jun 27, 2022

gerbeldo Jun 27, 2022

gerbeldo Jun 27, 2022

		liblzma-dev
		libbz2-dev

Update R #242

Update R #242

Conversation

alexvpickering commented Jun 2, 2022 • edited by gerbeldo Loading

Description

Notes:

Image sizes differences:

Worker (↓ ~ 50%):

Pipeline (↓ ~ 40%):

Details

URL to issue

Link to staging deployment URL (or set N/A)

Links to any PRs or resources related to this PR

Integration test branch

Merge checklist

Code updates

Manual/unit testing

Integration testing

Documentation updates

Optional

codecov bot commented Jun 3, 2022 • edited Loading

Codecov Report

gerbeldo left a comment

Choose a reason for hiding this comment

gerbeldo Jun 27, 2022

Choose a reason for hiding this comment

gerbeldo Jun 27, 2022

Choose a reason for hiding this comment

gerbeldo Jun 27, 2022

Choose a reason for hiding this comment

gerbeldo Jun 27, 2022

Choose a reason for hiding this comment

gerbeldo Jun 27, 2022

Choose a reason for hiding this comment

gerbeldo Jun 27, 2022

Choose a reason for hiding this comment

alexvpickering commented Jun 2, 2022 •

edited by gerbeldo

Loading

codecov bot commented Jun 3, 2022 •

edited

Loading