exp: expose implementation (custom Git refs)

[jorgeorpinel]

From iterative/dvc.org#2243 (comment)

Specifically, apply and branch are wrappers of narrow/complex git operations. Originally we didn't want users to worry about how experiments are implemented. That makes sense but is it the practical approach? If users are already familiar with Git (at least some, at least somewhat), why not expose the fact that exps are commits/branches that can be targeted by (certain) regular Git commands? Examples:

• Navigate between experiments: git checkout exp-1234 (detached HEAD)
• "Apply" one or more experiments: git merge --squash (--no-commit) (--theirs) exp-1234 (see Re-run previous experiments at current HEAD #5782)
• Branch off an exp: git branch exp-1234
• There's probably a git substitute for exp remove too

The main pros I see is that the Git alternatives give users more flexibility, and we don't need to maintain them
The con is that we may need to educate users on Git to some extent.

Cc @pmrowla @dberenbaum @shcheklein

[pmrowla]

The main limitation with forcing users to use CLI Git for these kinds of operations is that they cannot use experiment names with CLI Git. Additionally, the 1:1 substitutes for certain exp commands would be low-level plumbing commands which users are probably unfamiliar with (and generally should not be using unless they are very familiar with how git internals work).

As an example, exp remove exp-123 would need to be replaced by

git update-ref -d refs/exps/ab/c12345<...the rest of a SHA-256 hash...>/exp-123

which is probably not something we want to tell users to use (for several reasons).

---

Also, exp apply cannot be replaced by pure Git CLI commands, as it does additional DVC specific operations. exp apply is needed when users want to modify an intermediate checkpoint and then do a branching exp run from that modified point.

[jorgeorpinel]

Hmmmm OK.

Here's my thought process: from our many iterations on Experiments docs I've realized it can get pretty tricky to explain experiments without the context of how they're implemented, which is why I think maybe it would be easier to expose it anyway, whether we keep some of these utilities or not.

So maybe just consider removing exp branch (only one that can be easy subs with git, it seems) and explain exps differently in docs. Thoughts?

[pmrowla]

The issue with being unable to use exp names still applies with git branch, but yes, you can do git branch <new-branch-name> <exp-SHA> to get the same behavior

[jorgeorpinel]

On the issue of exp names vs. Git porcelain, one option is to show/list command SHAs by default instead (if we eventually decide to expose the underlying Git implementation).

[iesahin]

I do think the opposite, there are already too much Git-related messages, e.g., I'd simply commit the changes (subject to an option) instead of telling the user "You can now git add .gitignore dvc.lock etc.dvc". For the experiments, in particular, emphasizing to play with the related Git objects will result in more headaches for them, I think.

[dberenbaum]

Interesting discussion!

I do think the opposite, there are already too much Git-related messages, e.g., I'd simply commit the changes (subject to an option) instead of telling the user "You can now git add .gitignore dvc.lock etc.dvc".

This is a common suggestion since it feels like dvc requires an extra step that many users will always be doing by default. I think forcing users to git add gently forces them to understand what dvc is doing. If dvc committed all those files automatically, many users would have no idea that .gitignore gets modified or that dvc.lock or .dvc files even exist or how they got there.

@shcheklein has linked to this article before, which I find really useful in thinking about designing UI: https://hisham.hm/2020/12/18/user-power-not-power-users-htop-and-its-design-philosophy/. It's a bit long, but here's a quote that summarizes the point:

I wanted to make a tool that has the power but doesn't hide it from users, and instead invites users into becoming more powerful. A tool that reaches out its hand and guides them along the way, helping users to educate themselves about what's happening on the internals of their system, and in this way have more control of it.

Whether this is correct or not, there is some rationale for why the current UI exists, and a similar rationale could apply to gently exposing the experiments implementation.

[jorgeorpinel]

there are already too much Git-related messages

Idk if that's a problem. There's no way around the fact that DVC versioning and other features (experiments) need Git.

The problem with just asking users to "commit changes to Git" is that some people may not know what to do exactly. Failing to copy/paste git commit .gitignore data.dvc is much less likely.

A tool that reaches out its hand and guides them along the way

Great principle. I think that as far as DVC builds up on Git, we may need to educate about that as needed.

p.s. OK, moving this to discussions...

[iesahin]

Git is already more verbose than it should be: For day-to-day operations, there are too many moving parts and this is not power, this is bureacracy. It leads to more errors. If you forget to git add certain files, you either need a git commit --amend if you're lucky or git rebase if not. I do this at least once every day. It's true that rewriting history is power but if we need to use this power too frequently, I suspect if there is a problem with the underlying design. Git's CLI UI is notoriously bad and IMO we shouldn't aspire to be like Git on UI side.

I would like to have my tools disappear in the background for daily use. This is not possible for Git, it's always there telling me to add, branch, checkout, pull, push, merge, rebase and notify their differences. You need to be fully awake while using Git. (Why push doesn't push all branches but fetch fetches all of them? I still don't get after years of use and need to keep in mind always when I'm working with multiple branches.) When I'm struggling with a software problem, the last thing I'd want is to be distracted by the underlying tool but it happens a lot for Git. I don't think this is caused only by my limited mental capacity.

For our case, when DVC asks the user to git add this/.gitignore that.dvc, it feels powerful for the first few times but as a daily driver, (sometimes unavoidable) Git interaction becomes a burden. User deals with an ML problem, then deals with DVC, then deals with Git, and hopefully reaches some nirvana until the next iteration if they don't forget what they were working on.

I would have some set of daily commands like Git porcelain to hide the underlying Git mechanism completely and another set of commands for those who need power. We could measure the frequency of use that way, and I doubt more than 1% of the command-use will fall into plumbing commands for DVC.

[jorgeorpinel]

I agree the Git UI is bad (have thought so for years). But I'm not understanding how this relates to the discussion exactly.

I would have some set of daily commands like Git porcelain to hide the underlying Git mechanism completely

OK, do you have specific ideas for the exp commands? I feel like that's already what exp apply and exp branch are (please refer to OP).

and another set of commands for those who need power

Well, that would be git in our case. What I'm proposing is to stop hiding (in CLI messages and docs) the underlying Git implementation, so that users can access that power if needed.

[pmrowla]

@iesahin we already have a DVC config option to automatically stage changed files in git (i.e. do the suggested git add) if the user doesn't want to see the prompt all the time(core.autostage)

[jorgeorpinel]

I just realized we are already pretty open about the underlying Git implementation of experiments in the options of exp show. Specifically --all-branches/tag/commits and --sha. Why do so selectively, only in that command and its docs?

[pmrowla]

I don't really think we are actively hiding the underlying Git implementation for experiments, and it's not specific to exp show either?

The blog post about custom git refs is linked directly from the checkpoints user guide. --all-branches/tags/commits are also already used elsewhere in DVC. The examples for exp diff also show how to diff experiments to other git tags and branches. And exp push/pull/list also show how they are used directly with git remotes.

[pmrowla]

Also, exp show has already being changed to include the git SHAs for experiments by default in 2.1.0