Merge pull request #2200 from iterative/ref/exp/main

ref: main exp workflow (run,show,diff,apply)

content/docs/api-reference/make_checkpoint.md

@@ -20,12 +20,31 @@ while True:
 ## Description
 
 To track successive steps in a longer <abbr>experiment</abbr>, you can write
-your code so it registers checkpoints with DVC during runtime. This function
-should be called by the code in stages executes by `dvc exp run` (see `cmd`
-field of `dvc.yaml`).
+your code so it registers checkpoints with DVC during runtime (similar to a
+logger). This function can be called by the code in stages executed by
+`dvc exp run`.
 
-> Note that for non-Python code, the alternative is to write a
-> `.dvc/tmp/DVC_CHECKPOINT` signal file.
+`make_checkpoint()` does the following:
+
+1. Check that the `DVC_ROOT` env var is set. It means this code is being
+   executed via `dvc exp run`, and it contains the location to the correct
+   `.dvc/` directory for this experiment (which can vary when `-j` is used)
+2. Creates an empty `$DVC_ROOT/tmp/DVC_CHECKPOINT` signal file so DVC knows that
+   a checkpoint should be captures now.
+3. Blocks the execution of any further code (that changes the state of the
+   <abbr>workspace</abbr>) until the signal file is deleted, which means that
+   DVC has finished caching all the data, calculating hashes, etc. (see
+   `dvc commit`).
+
+💡 Note that for non-Python code, the way to register checkpoints with DVC is to
+implement the steps above yourself.
+
+The stage definition in `dvc.yaml` should contain at least one
+<abbr>output</abbr> with the `checkpoint: true` value set, so that DVC registers
+its checkpoints.
+
+⚠️ Using the `checkpoint` field in `dvc.yaml` is only compatibly with
+`dvc exp run`, `dvc repro` will abort if any stage contains it.
 
 ## Example: Every 100th iteration
 
@@ -86,8 +105,8 @@ Experiment results have been applied to your workspace.
 
 In this example we kill the process (with Ctrl + C) after 3 checkpoints (at 0,
 100, and 200). The <abbr>cache</abbr> will contain those 3 versions of
-`int.txt`. And DVC applies the last checkpoint to the <abbr>workspace</abbr>
-even when more cycles happened before the interrupt:
+`int.txt`. DVC applies the last checkpoint to the <abbr>workspace</abbr> even if
+more cycles happened after that:
 
 ```dvc
 $ cat int.txt
@@ -113,9 +132,9 @@ $ dvc exp show
 # Press q to exit this screen.
 ```
 
-Now if we use `dvc exp run` again, the process will start from 200. The
-`--reset` option of that command drops the existing checkpoints and restarts the
-experiment from scratch. Or, to restart from a previous checkpoint, you can use
-use `dvc exp apply` instead.
+If we use `dvc exp run` again, the process will start from 200 (since that's
+what the workspace reflects). The `--reset` option of that command drops the
+existing checkpoints and restarts the experiment from scratch. Or, to restart
+from a previous checkpoint, you can use use `dvc exp apply` instead.
 
 See `dvc experiments` for more info on managing <abbr>experiments</abbr>.

content/docs/command-reference/dag.md

@@ -54,15 +54,17 @@ restore one or more pipelines later (see `dvc repro`).
 
 - `-v`, `--verbose` - displays detailed tracing information.
 
-## Paging the output
+## Paginating the output
 
 This command's output is automatically piped to
-[Less](<https://en.wikipedia.org/wiki/Less_(Unix)>), if available in the
-terminal. (The exact command used is `less --chop-long-lines --clear-screen`.)
-If `less` is not available (e.g. on Windows), the output is simply printed out.
+[less](<https://en.wikipedia.org/wiki/Less_(Unix)>) if available in the terminal
+(the exact command used is `less --chop-long-lines --clear-screen`). If `less`
+is not available (e.g. on Windows), the output is simply printed out.
 
 > It's also possible to
-> [enable Less paging on Windows](/doc/user-guide/running-dvc-on-windows#enabling-paging-with-less).
+> [enable `less` on Windows](/doc/user-guide/running-dvc-on-windows#enabling-paging-with-less).
+
+> Note that this also applies to `dvc exp show`.
 
 ### Providing a custom pager
 

content/docs/command-reference/exp/apply.md

@@ -0,0 +1,40 @@
+# exp apply
+
+Apply the results from any [experiment](/doc/command-reference/exp) to the
+<abbr>workspace</abbr>.
+
+## Synopsis
+
+```usage
+usage: dvc exp apply [-h] [-q | -v] experiment
+
+positional arguments:
+  experiment     Experiment to be applied
+```
+
+## Description
+
+Rolls back/forward the workspace to reflect the results of a given `experiment`
+(name or hash are accepted, see `dvc exp run` for details). This means changing
+the appropriate [metafiles](/doc/user-guide/project-structure),
+<abbr>parameter</abbr> files, <abbr>metrics</abbr>, <abbr>plots</abbr>, and
+corresponding DVC-tracked data.
+
+> This is similar to `dvc checkout`, but for `dvc experiments`.
+
+This is typically used after using `dvc exp show` or `dvc exp diff` to find the
+best experiment, and before committing to Git in order to make it
+[persistent](/doc/user-guide/experiment-management#persistent-experiments).
+
+Note that this command will fail if the target `experiment` was not derived from
+the current Git commit.
+
+## Options
+
+- `-h`, `--help` - shows the help message and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
+  problems arise, otherwise 1.
+
+- `-v`, `--verbose` - displays detailed tracing information from executing the
+  `dvc pull` command.

content/docs/command-reference/exp/diff.md

@@ -0,0 +1,77 @@
+# exp diff
+
+Show changes in [metrics](/doc/command-reference/metrics) and
+[parameters](/doc/command-reference/params) between `dvc experiments`.
+
+## Synopsis
+
+```usage
+usage: dvc experiments diff [-h] [-q | -v] [--all] [--show-json]
+                            [--show-md] [--old] [--no-path]
+                            [--precision <n>] [a_rev] [b_rev]
+
+positional arguments:
+  a_rev            Old experiment to compare (defaults to HEAD)
+  b_rev            New experiment to compare (defaults to the current
+                   workspace)
+```
+
+# Description
+
+Provides a quick way to compare `dvc params` and `dvc metrics` between two
+`dvc experiments` by printing a table of differences. By default, it includes
+the params/metrics file "Path", "Param" or "Metric" name, the new "Value", and
+the difference ("Change") for numeric values. Example:
+
+```dvc
+$ dvc exp diff
+Path         Metric    Value    Change
+scores.json  roc_auc   0.93345  0.018087
+scores.json  grade     B-       —
+
+Path         Param         Value    Change
+params.yaml  max_features  3000     1500
+```
+
+> This is similar to combining the reports from `dvc params diff` and
+> `dvc metrics diff` together, for the experiments in question.
+
+Without arguments, this command compares all the params and metrics defined in
+`dvc.yaml` files currently present in the <abbr>workspace</abbr> (e.g. the last
+`dvc exp run`) with the latest committed versions (required). Only
+params/metrics that changed are listed, by default (show everything with
+`--all`).
+
+`a_rev` and `b_rev` are optional experiments to compare. They accept experiment
+names or hashes (see `dvc exp run`). Providing a single one results in comparing
+it to the workspace.
+
+Another way to display experiments is the `dvc exp show` command, which lists
+all the current experiments (without comparisons).
+
+## Options
+
+- `--all` - list all parameters and metrics, including those without changes.
+
+- `--show-json` - prints the command's output in easily parsable JSON format,
+  instead of a human-readable table.
+
+- `--show-md` - prints the command's output in the Markdown table format
+  ([GFM](https://github.github.com/gfm/#tables-extension-)).
+
+- `--old` - include the "Old" value column in addition to the new "Value" (and
+  "Change"). Useful when the values are not numeric, for example
+
+- `--no-path` - hide the "Path" column that lists the param/metrics file
+  location. Useful when only one metrics or params file exists, for example
+
+- `--precision <n>` -
+  [round](https://docs.python.org/3/library/functions.html#round) decimal values
+  to `n` digits of precision (5 by default). Applies to metrics only.
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
+  problems arise, otherwise 1.
+
+- `-v`, `--verbose` - displays detailed tracing information.

content/docs/command-reference/exp/index.md

@@ -2,7 +2,7 @@
 
 ⚠️ This feature is only available in DVC 2.0 ⚠️
 
-A set of commands to generate and manage <abbr>experiment</abbr>:
+A set of commands to generate and manage <abbr>experiments</abbr>:
 [run](/doc/command-reference/exp/run), [show](/doc/command-reference/exp/show),
 [diff](/doc/command-reference/exp/diff),
 [apply](/doc/command-reference/exp/apply),
@@ -14,6 +14,8 @@ A set of commands to generate and manage <abbr>experiment</abbr>:
 
 > Aliased to `dvc exp`.
 
+> Requires that Git is being used to version the project.
+
 ## Synopsis
 
 ```usage
@@ -36,25 +38,14 @@ positional arguments:
 
 ## Description
 
-`dvc exp` subcommands provide specialized ways to create and manage data science
-experiments. The basic workflow goes like this:
-
-- Modify <abbr>dependencies</abbr> (e.g. input data or source code),
-  <abbr>parameters</abbr>, or commands (`cmd` field of `dvc.yaml`) of committed
-  stages.
-- Use `dvc exp run` (instead of `repro`) to execute the pipeline. This puts the
-  experiment's results in your <abbr>workspace</abbr>, and tracks it under the
-  hood.
-- Visualize experiment configurations and results with `dvc exp show`. Repeat.
-- Use [metrics](/doc/command-reference/metrics) in your pipeline to identify the
-  best experiment(s), and promote them to persistent experiments (regular
-  commits) with `dvc exp apply`.
+`dvc exp` subcommands provide specialized ways to create and manage data
+science/ machine learning experiments.
 
 📖 See [Experiment Management](/doc/user-guide/experiment-management) for more
 info.
 
-> Note that DVC assumes that <abbr>experiments</abbr> are deterministic (see
-> **Avoiding unexpected behavior** in `dvc run`).
+> ⚠️ Note that DVC assumes that experiments are deterministic (see **Avoiding
+> unexpected behavior** in `dvc stage add`).
 
 ## Options
 

content/docs/command-reference/exp/run.md

@@ -0,0 +1,144 @@
+# exp run
+
+Run an experiment: reproduce a variation of a committed
+[pipeline](/doc/command-reference/dag) in a hidden project branch.
+
+> Similar to `dvc repro` but for
+> [experimentation](/doc/user-guide/experiment-management).
+
+## Synopsis
+
+```usage
+usage: dvc exp run [-h] [-q | -v] [-f]
+                   [<repro_options> ...]
+                   [--params [<filename>:]<params_list>]
+                   [-n <name>] [--queue] [--run-all] [-j <number>]
+                   [targets [targets ...]]
+
+positional arguments:
+  targets               Stages to reproduce. 'dvc.yaml' by default.
+```
+
+## Description
+
+Provides a way to execute and track `dvc experiments` in your
+<abbr>project</abbr> without polluting it with unnecessary commits, branches,
+directories, etc.
+
+> `dvc exp run` is equivalent to `dvc repro` for <abbr>experiments</abbr>. It
+> has the same behavior when it comes to `targets` and stage execution (restores
+> the dependency graph, etc.). See the command [options](#options) for more the
+> differences.
+
+Before using this command, you'll probably want to make modifications such as
+data and code updates, or <abbr>hyperparameter</abbr> tuning. You can use the
+`--set-param` option to change `dvc param` values on-the fly.
+
+Each `dvc exp run` creates a variation based on the latest project version
+committed to Git, and tracks this experiment automatically with an automatic
+name like `exp-bfe64` (which can be customized with the `--name` option).
+
+<details>
+
+### How does DVC track experiments?
+
+Internally, `dvc exp` uses actual commits under custom
+[Git references](https://git-scm.com/book/en/v2/Git-Internals-Git-References)
+(found in `.git/refs/exps`). Each commit has the Git `HEAD` as parent and has
+it's own SHA-256 hash. These are not pushed to the Git remote by default (see
+`dvc exp push`).
+
+> References have a unique signature similar to the
+> [entries in the run-cache](/doc/user-guide/project-structure/internal-files#run-cache).
+
+</details>
+
+The results of the last experiment can be seen in the <abbr>workspace</abbr>. To
+display and compare your experiments, use `dvc exp show` or `dvc exp diff`. Use
+`dvc exp apply` to roll back the workspace to a previous experiment
+
+Successful experiments can be made
+[persistent](/doc/user-guide/experiment-management#persistent-experiments) by
+committing them to the Git repo. Unnecessary ones can be removed with
+`dvc exp gc`, or abandoned (and their data with `dvc gc`).
+
+## Checkpoints
+
+To track successive steps in a longer <abbr>experiment</abbr>, you can register
+checkpoints with DVC during your code or script runtime (similar to a logger).
+
+To do so, first mark stage `outs` with `checkpoint: true` in `dvc.yaml`. Then,
+in the corresponding code, either call the `dvc.api.make_checkpoint()` function
+(Python), or write a signal file (any programming language) following the same
+steps as `make_checkpoint()` — please refer to that reference for details.
+
+⚠️ Using the `checkpoint` field in `dvc.yaml` is only compatibly with
+`dvc exp run`, `dvc repro` will abort if any stage contains it.
+
+<details>
+
+### How are checkpoints captured by DVC?
+
+When DVC runs a checkpoint-enabled experiment, a custom Git branch (in
+`.git/refs/exps`) is started off the repo `HEAD`. A new commit is appended each
+time a checkpoint is registered by the code. These are not pushed to the Git
+remote by default (see `dvc exp push`).
+
+</details>
+
+## Queueing and parallel execution
+
+The `--queue` option lets you create an experiment as usual, except that nothing
+is actually run. Instead, the experiment is put in a wait-list for later
+execution. `dvc exp show` will mark queued experiments with an asterisk `*`.
+
+Use `dvc exp run --run-all` to process this queue. Adding `-j` (`--jobs`),
+queued experiments can be run in parallel for better performance. This creates a
+temporary workspace copy (in `.dvc/tmp/exp<n>` where _n_ is the job number) for
+each subprocess.
+
+⚠️ Parallel runs are experimental and may be unstable at this time. ⚠️ Make sure
+you're using a number of jobs that your environment can handle (no more than the
+CPU cores).
+
+> Note that each job runs the entire pipeline (or `targets`) serially — DVC
+> makes no attempt to distribute stage commands among jobs.
+
+## Options
+
+> In addition to the following, `dvc exp run` accepts all the options in
+> `dvc repro`, with the exception that `--no-commit` has no effect here.
+
+- `-S [<filename>:]<params_list>`, `--set-param [<filename>:]<params_list>` -
+  set the specified `dvc params` for this experiment. `filename` can be any
+  valid params file (`params.yaml` bu default). `params_list` accepts a
+  comma-separated list of key-value pairs in the form `key1=val1,key2=val2...`.
+  This will override the param values coming from the params file.
+
+- `-n <name>`, `--name <name>` - specify a name for this experiment. A default
+  name will generated by default, such as `exp-f80g4` (based on the experiment's
+  hash).
+
+- `-f`, `--force` - reproduce an experiment pipeline, regenerating its results,
+  even if no changes were found. For `checkpoint` experiments, `dvc exp run -f`
+  will force regeneration of an existing checkpoint experiment branch from
+  scratch.
+
+- `--queue` - place this experiment at the end of a line for future execution,
+  but do not actually run it yet. Use `dvc exp run --run-all` to process the
+  queue.
+
+- `--run-all` - run all queued experiments (see `--queue`). Use `-j` to execute
+  them [in parallel](#queueing-and-parallel-execution).
+
+- `-j <number>`, `--jobs <number>` - run this `number` of queued experiments in
+  parallel. Only applicable when used in conjunction with `--run-all`.
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if all
+  stages are up to date or if all stages are successfully executed, otherwise
+  exit with 1. The command defined in the stage is free to write output
+  regardless of this flag.
+
+- `-v`, `--verbose` - displays detailed tracing information.