Target Markdown

targets 0.5.0

Bug fixes

• Export in-memory config settings from _targets.yaml to parallel workers.

New features

• Add a limited-scope exclude argument to tar_watch() and tar_watch_server() (#458, @gorkang).
• Write a .gitignore file to ignore everything in _targets/meta/ except .gitignore and _targets/meta/meta.
• Target Markdown: add knitr engines for pipeline construction and prototyping from within literate programming documents (#469, @cderv, @nviets, @emilyriederer, @ijlyttle, @gshotwell, @gadenbuie, @tomsing1). Huge thanks to @cderv on this one for answering my deluge of questions, helping me figure out what was and was not possible in knitr, and ultimately circling me back to a successful approach.
• Add an RStudio R Markdown template for Target Markdown (#469).
• Implement use_targets(), which writes the Target Markdown template to the project root (#469).
• Implement tar_unscript() to clean up scripts written by Target Markdown.

Enhancements

• Enable priorities in tar_make() and tar_manifest().
• Show the priority in the print method of stem and pattern targets.
• Throw informative errors if the secondary arguments to pattern = slice() or pattern = sample() are invalid.
• In tar_target_raw(), assert that commands have length 1 when converted to expressions.
• Handle errors and post failure artifacts in the Github Actions YAML file.
• Rewrite the documentation on invalidation rules in tar_cue() (@maelle).
• Drop dplyr groups and "grouped_df" class in tar_group() (tarchetypes discussion #53, @kendonB).
• Assign branch names to dynamic branching return values produced by tar_read() and tar_read_raw().

CRAN hotfix: _targets.yaml

targets 0.4.2

Bug fixes

• Do not use time stamps to monitor the config file (e.g. _targets.yaml). Fixes CRAN check errors from version 0.4.1.

CRAN hotfix

targets 0.4.1

• Fix CRAN test error on Windows R-devel.
• Do not inherit roxygen2 docstrings from shiny.
• Handle more missing Suggests: packages.
• Unset the config lock before reading targets.yaml in the callr process.

HPC performance and monitoring

targets 0.4.0

Bug fixes

• Avoid file.rename() errors when migrating staged temporary files (#410).
• Return correct error messages from feather and parquet formats (#388). Now calling assert_df() from store_assert_format() instead of store_cast_object(). And now those last two functions are not called at all if the target throws an error.
• Retry writing lines to database files so Windows machines can run tar_poll() at the same time as the pipeline (#393).
• Rename file written by tar_renv() to _targets_packages.R (#397).
• Ensure metadata is loaded to compute labels properly when outdated = FALSE in tar_visnetwork().

New features

• Implement tar_timestamp() and tar_timestamp_raw() to get the last modified timestamp of a target's data (#378).
• Implement tar_progress_summary() to compactly summarize all pipeline progress (#380).
• Add a characters argument of tar_traceback() to cap the traceback line lengths (#383).
• Add new "summary" and "about" views to tar_watch() (#382).
• Implement tar_poll() to repeatedly poll runtime progress in the R console (#381). tar_poll() is a lightweight alternative to tar_watch().
• Change the color of the "dormant" status in the graph.
• Add a tar_envvar() function to list values of special environment variables supported in targets. The help file explains each environment variable in detail.
• Support extra project-level configuration settings with _targets.yaml (#297). New functions tar_config_get() and tar_config_set() interact with the _targets.yaml file. Currently only supports the store field to set the data store path to something other than _targets/.

Performance

• Shut down superfluous persistent workers earlier in dynamic branching and when all remaining targets have deployment = "main" (#398, #399, #404, @pat-s).

Enhancements

• Attempt to print only the useful part of the traceback in tar_traceback() (#383).
• Add a line break at the end of the "summary" reporter so warnings do not mangle the output.
• In tar_watch(), use shinybusy instead of shinycssloaders and keep current output on display while new output is rendering (#386, @rcorty).
• Right-align the headers and counts in the "summary" and "forecast" reporters.
• Add a timestamp to the "summary" reporter.
• Make the reporters show when a target ends (#391, @mattwarkentin).
• Make the reporters show when a pattern ends if the pattern built at least one target and none of the targets errored or canceled.
• Use words "start" and "built" in reporters.
• Use the region of the AWS S3 bucket instead of the local AWS_DEFAULT_REGION environment variable (check_region = TRUE; #400, @tomsing1).
• In tar_meta(), return POSIXct times in the time zone of the calling system (#131).
• Throw informative error messages when a target's name or command is missing (#413, @liutiming).
• Bring back ALTREP in qs::qread() now that qs 0.24.1 requires stringfish >= 1.5.0 (#147, @glep).
• Relax dynamic branching checks so pattern = slice(...) can take multiple indexes (#406, #419, @djbirke, @alexgphayes)

CRAN hotfix: arrow formats

targets 0.3.1

Bug fixes

• queue$enqueue() is now queue$prepend() and always appends to the front of the queue (#371).

Enhancements

• Throw a warning if devtools::load_all() or similar is detected inside _targets.R (#374).

CRAN

• Skip feather and parquet tests on CRAN.

Arrow formats, HPC performance, infrastructure refactor, GHA

targets 0.3.0

Bug fixes

• Fix the "write target at cursor" RStudio addin and move cursor between the parentheses.

New features

• Add a backoff option in tar_option_set() to set the maximum upper bound (seconds) for the polling interval (#333).
• Add a new tar_github_actions() function to write a GitHub Actions workflow file for continuous deployment of data analysis pipelines (#339, @jaredlander).
• Add a new TAR_MAKE_REPORTER environment variable to globally set the reporter of the tar_make*() functions (#345, @alexpghayes).
• Support new storage formats "feather", "parquet", "aws_feather", and "aws_parquet" (#355, @riazarbi).

Performance

• Implement an exponential backoff algorithm for polling the priority queue in tar_make_clustermq() and tar_make_future() (#333).
• In tar_make_future(), try to submit a target every time a worker is polled.
• In tar_make_future(), poll workers in order of target priority.
• Avoid the time delay in exiting on error (from r-lib/callr#185).
• Clone target objects for the pipeline and scrape more targets internal objects out of the environment in order to avoid accidental massive data transfers to workers.

Enhancements

• Use rlang::check_installed() inside assert_package() (#331, @malcolmbarrett).
• Allow tar_destroy(destroy = "process").
• In tar_watch(), increase default seconds to 15 (previously 5).
• In tar_watch(), debounce instead of throttle inputs.
• In tar_watch(), add an action button to refresh the outputs.
• Always deduplicate metadata after tar_make(). Will help compute a cache key on GitHub Actions and similar services.
• Deprecate tar_deduplicate() due to the item above.
• Reorder information in timestamped messages.
• Document RNG seed generation in tar_target_raw(), tar_meta(), and tar_seed() (#357, @alexpghayes).
• Switch meaning of %||% and %|||% to conform to historical precedent.
• Only show a command line spinner if reporter = "silent" (#364, @matthiasgomolka).
• Target and pipeline objects no longer have an envir element.

Nicer arguments and messages

Bug fixes

• In tar_load(), subset metadata to avoid accidental attempts to load global objects in tidyselect calls.
• Do not register a pattern as running unless an actual branch is about to start (#304).
• Use a name spec in vctrs::vec_c() (#320, @joelnitta).

New features

• Add a new names argument to tar_objects() and tar_workspaces() with tidyselect functionality.
• Record info on the main process (PID, R version, targets version) in _targets/meta/process and write new functions tar_process() and tar_pid() to retrieve the data (#291, #292).
• Add a new targets_only argument to tar_meta().
• Add new functions tar_helper() and tar_helper_raw() to write general-purpose R scripts, using tidy evaluation for as a template mechanism (#290, #291, #292, #306).
• Export functions to check the existence of various pieces of local storage: tar_exist_meta(), tar_exist_objects(), tar_exist_progress(), tar_exist_progress(), tar_exist_script() (#310).
• Add a new supervise argument to tar_watch().
• Add a new complete_only argument to tar_meta() to optionally return only complete rows (no NA values).
• Catch callr errors and refer users to the debugging chapter of the manual.

Enhancements

• Improve error messages of invalid arguments (#298, @brunocarlin). Removes partial argument matching in most cases.
• By default, locally enable crayon if an only if the calling process is interactive (#302, @ginolhac). Can still be disabled with options(crayon.enabled = FALSE) in _targets.R.
• Improve error handling and message for format = "url" when the HTTP response status code is not 200 (#303, @petrbouchal).
• Add more extras packages to tar_renv() (to support tar_watch()).
• Show informative message instead of error in tar_watch() if _targets.R does not exist.
• Clear up the documentation of the names argument of tar_load() (#314, @jameelalsalam).
• Do not override nobody in custom curl handles (#315, @riazarbi).
• Rename "running" to "started" in the progress metadata. This avoids the implicit claim that targets is somehow actively monitoring each job, e.g. through a connection or heartbeat (#318).
• Set errormode = "warn" in getVDigest() for files to work around eddelbuettel/digest#49 for network drives on Windows. targets already runs those file checks anyway. (#316, @boshek).
• If a package fails to load, print the library paths targets tried to load from.

CRAN hotfix and app improvements

Bug fixes

• tar_test() now skips all tests on Solaris in order to fix the problems shown on the CRAN check page.
• Enable allow and exclude to work on imports in tar_visnetwork() and tar_glimpse().
• Put visNetwork legends on right to deal with how legend nodes always align right within the legend space.

Performance

• Call force() on subpipeline objects to eliminate high-memory promises in target objects. Allows targets to be deployed to workers much faster when retreival is "main" (#279).

New features

• Add a new box to the tar_watch() app to tabulate progress on dynamic branches (#273, @mattwarkentin).
• Store type, parent, and branches in progress data for tar_watch() (#273, @mattwarkentin).
• Add a fields argument in tar_progress() and default to "progress" for back compatibility (#273, @mattwarkentin).
• Add a new tar_progress_branches() function to tabulate branch progress (#273, @mattwarkentin).
• Add new "refresh" switch to tar_watch() to toggle automatic refreshing and force a refresh.

Enhancements

• Exclude .Random.seed by default in tar_visnetwork().
• Spelling: "cancelled" changed to "canceled".
• Enhance controls and use of space in the tar_watch() app.
• Centralize internal path management utilities.

Configuration

• Skip clustermq tests on Solaris.

Address CRAN feedback

• Avoid starting the description with the package name.
• Remove if(FALSE) blocks from help files to fix "unexecutable code" warnings (tar_glimpse(), tar_visnetwork(), and tar_watch()).
• Remove commented code in the examples (tar_edit(), tar_watch_ui(), and tar_watch_server()).
• Ensure that all examples, tests, and vignettes do not write to the user's home file space. (Fixed an example of tar_workspace().)

First production release

Enhancements

• Accept lists of target objects at the end of _targets.R (#253).
• Deprecate tar_pipeline() and tar_bind() because of the above (#253).
• Always show a special message when the pipeline finishes (#258, @petrbouchal).
• Disable visNetwork stabilization (#264, @mattwarkentin).
• Use default visNetwork font size.
• Relay errors as condition messages if error is "continue" (#267, @liutiming).