Add a task for Rmarkdown notebook rendering

[bkkkk]

We use Rmarkdown notebooks in our team to create nice data reports and portable dashboards. I would like to build a Task to render these kinds of notebooks.

I'm trying to figure out what the right approach is here, from Python we run R commands via the shell as a subprocess. I'm not sure what the right approach is in the context of a task. A few issues to clarify:

• How to run the R code to render the notebook (subprocess library)?
• How to capture the screen output from R correctly?
• How do we handle errors since we are running through the shell?
• How to handle complicated parameters to user notebook?

[joshmeek]

Hi @bkkkk while I haven't used Rmarkdown notebooks before with Prefect I could answer your questions generally:

How to run the R code to render the notebook (subprocess library)?

I think you would have success doing something similar to the DbtShellTask (which uses Dbt through the ShellTask) https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/dbt/dbt.py OR if there is a R library for python bindings that you like such as rpy2.

How to capture the screen output from R correctly?

Not familiar with that the screen output is but if it's an image you could write it to a file either directly or through a Result

How do we handle errors since we are running through the shell?

Shell commands return status codes that you could reason about depending on what is returned from running the command

How to handle complicated parameters to user notebook?

If running the notebooks through the shell allows arguments you could pass them there or you could write some logic in your notebooks to read these parameters from places like files or env vars