Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get started tutorial for experiments (with readable diff) #2195

Merged
merged 46 commits into from
Feb 27, 2021
Merged
Show file tree
Hide file tree
Changes from 43 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
92e66cb
update existing get started sections to prep for experiments
dberenbaum Feb 15, 2021
a176145
add new experiments section
dberenbaum Feb 15, 2021
8d434e7
change --params to --set-param
dberenbaum Feb 16, 2021
a426953
update ml-pipelines to metrics, params, plots
dberenbaum Feb 16, 2021
0c9469b
update external refs to old experiments section
dberenbaum Feb 17, 2021
49de41b
fix metrics, parameters, and plots sidebar title
dberenbaum Feb 17, 2021
4f40f1e
introduce 2.0 warning for get started exps
dberenbaum Feb 18, 2021
e8b2482
shorten param names
dberenbaum Feb 18, 2021
0e05431
fix --set-params flags in exp queue
dberenbaum Feb 18, 2021
6eee8b0
reexpand featurize stage name
dberenbaum Feb 18, 2021
38b40c0
update get started plots modify command
dberenbaum Feb 19, 2021
c05c547
change -w to --workspace for more clarity
dberenbaum Feb 19, 2021
c991faf
update existing get started sections to prep for experiments
dberenbaum Feb 15, 2021
3d33c03
add new experiments section
dberenbaum Feb 15, 2021
91f7664
change --params to --set-param
dberenbaum Feb 16, 2021
165b734
update ml-pipelines to metrics, params, plots
dberenbaum Feb 16, 2021
1007256
update external refs to old experiments section
dberenbaum Feb 17, 2021
994b585
fix metrics, parameters, and plots sidebar title
dberenbaum Feb 17, 2021
90684a4
introduce 2.0 warning for get started exps
dberenbaum Feb 18, 2021
02272bc
shorten param names
dberenbaum Feb 18, 2021
a904722
fix --set-params flags in exp queue
dberenbaum Feb 18, 2021
5a14c30
reexpand featurize stage name
dberenbaum Feb 18, 2021
525f5dd
update get started plots modify command
dberenbaum Feb 19, 2021
a8b9a15
change -w to --workspace for more clarity
dberenbaum Feb 19, 2021
854701f
resolve conflicts in upstream
dberenbaum Feb 23, 2021
ecb8674
upstream changes pointing to exp mgmt guide
dberenbaum Feb 23, 2021
8fe1c67
use updated formatter
dberenbaum Feb 23, 2021
375ad9f
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
0dc3397
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
f39df81
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
b042bf3
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
3d7b0fd
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
b9a35e5
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
6e86b66
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
6cfc53d
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
9cfd82a
get started exp updates based on initial jorge feedback
dberenbaum Feb 25, 2021
1732fa0
Merge branch 'master' of github.com:iterative/dvc.org into experiment…
dberenbaum Feb 25, 2021
21647a2
Merge branch 'experiments_reset' of github.com:iterative/dvc.org into…
dberenbaum Feb 25, 2021
081f2c2
auto formatting
dberenbaum Feb 25, 2021
4b90f59
Update content/docs/start/experiments.md
dberenbaum Feb 25, 2021
052f46f
responding to more feedback from jorge
dberenbaum Feb 25, 2021
dfa9488
added note on ephemeral exps at top
dberenbaum Feb 25, 2021
9c40eac
remove data deps as being tracked by exps
dberenbaum Feb 26, 2021
54087eb
add'l cleanup of get started exps page
Feb 26, 2021
e04e38d
reverted blog link
Feb 26, 2021
e0800ed
Add metrics/plots show in get started (#2240)
Feb 27, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/blog/2020-08-27-august-20-community-gems.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Alternately, if your pipeline stage has already been created, you can manually
edit your `dvc.yaml` file to replace `params.yaml` with `params.json`.

For more about the `params.yaml` file,
[see our docs](https://dvc.org/doc/start/experiments#defining-parameters).
[see our docs](https://dvc.org/doc/start/metrics-parameters-plots#defining-parameters).
dberenbaum marked this conversation as resolved.
Show resolved Hide resolved

### [Q: Is there a guide for migrating from Git-LFS to DVC?](https://discordapp.com/channels/485586884165107732/485596304961962003/743559246599421974)

Expand Down
4 changes: 2 additions & 2 deletions content/docs/command-reference/metrics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,8 @@ to compare and pick the best performing experiment.
## Examples

> This example is based on our
> [Get Started](/doc/start/experiments#collecting-metrics), where you can find
> the actual source code.
> [Get Started](/doc/start/metrics-parameters-plots#collecting-metrics), where
> you can find the actual source code.
Comment on lines -106 to +107

This comment was marked as resolved.

Copy link
Contributor

@jorgeorpinel jorgeorpinel Feb 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 This makes me wonder if we should change the /doc/start/experiments URL completely, since it has totally new content. Links from 3rd party blogs will now end up in a very different page... And the search ranking may get disrupted (temporarily). Let me check the Search Console... ⌛

Copy link
Contributor

@jorgeorpinel jorgeorpinel Feb 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK so on the one hand, it has decent results and some traffic. Replacing the content like this may disturb that (prob. temporarily):

image

On the other, most of the searches are for experiment-related features 😆 so maybe it's better to replace this with what users are already searching and finding (could improve CTR/lower bounce rate in the end):

image

WDYT guys? BTW this is my only major concern so far (both to replace all existing links and whether to change the URL, which would imply setting up a redirect) cc @shcheklein

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I had skipped the blog content. I can update those to point to metrics-parameters-plots.

I lean towards wanting links to experiments to point to the new page to encourage using the updated experiments features. I did a quick Google search for "dvc experiments" and looked through the first two pages of results as a quick check. I only found one that actually linked directly to the old experiments page, and that one use the old link (see https://towardsdatascience.com/enable-ml-experiments-4ba8c3c8bdc2, which links to https://dvc.org/doc/get-started/compare-experiments). Maybe that old link should redirect to metrics-parameters-plots? But even then, it's probably better to redirect to the current experiments functionality.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a reference in https://dvc.org/blog/december-20-dvc-heartbeat to the old video. That was removed from the page since it is more focused on experiments. Thoughts on how to handle this reference, but also what to do with that video?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lean towards wanting links to experiments to point to the new page to encourage using the updated experiments features

Sure, depends on the context of each link. Up to you to leave some as-is.
Good to know there's not many 3rd party links so no worries there then.

/doc/get-started/compare-experiments). Maybe that old link should redirect to metrics-parameters-plots

Everything /doc/get-started* already redirects to /doc/start 👍


First, let's imagine we have a simple [stage](/doc/command-reference/run) that
produces a `eval.json` metrics file:
Expand Down
4 changes: 2 additions & 2 deletions content/docs/command-reference/metrics/show.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@ compares them with a previous version.
## Examples

> This example is based on the `evaluate` stage of our
> [Get Started](/doc/start/experiments#collecting-metrics), where you can find
> the actual source code.
> [Get Started](/doc/start/metrics-parameters-plots#collecting-metrics), where
> you can find the actual source code.

The basic use case shows the values in the current workspace:

Expand Down
6 changes: 5 additions & 1 deletion content/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,14 @@
}
},
{
"slug": "experiments",
"label": "Metrics, Parameters, and Plots",
"slug": "metrics-parameters-plots",
"tutorials": {
"katacoda": "https://katacoda.com/dvc/courses/get-started"
}
},
{
"slug": "experiments"
}
dberenbaum marked this conversation as resolved.
Show resolved Hide resolved
]
},
Expand Down
12 changes: 6 additions & 6 deletions content/docs/start/data-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,11 @@ The command options used above mean the following:
- `-n prepare` specifies a name for the stage. If you open the `dvc.yaml` file
you will see a section named `prepare`.

- `-p prepare.seed,prepare.split` is a special type of dependencies -
- `-p prepare.seed,prepare.split` defines special types of dependencies -
[parameters](/doc/command-reference/params). We'll get to them later in the
[Experiments](/doc/tutorials/get-started/experiments) section, but the idea is
that stage can depend on field values from a parameters file (`params.yaml` by
default):
[Metrics, parameters, and plots](/doc/start/metrics-parameters-plots) section,
but the idea is that the stage can depend on field values from a parameters
file (`params.yaml` by default):

```yaml
prepare:
Expand Down Expand Up @@ -207,7 +207,7 @@ with the same set of options:

```dvc
$ dvc run -n train \
-p train.seed,train.n_estimators \
-p train.seed,train.n_est,train.min_split \
-d src/train.py -d data/features \
-o model.pkl \
python src/train.py data/features model.pkl
Expand Down Expand Up @@ -240,7 +240,7 @@ parameters for the training stage:
$ vim params.yaml
```

Change `n_estimators` to `100` and run `dvc repro`, you should see:
Change `n_est` to `100` and run `dvc repro`, you should see:

```dvc
$ dvc repro
Expand Down
Loading