stage add: insert stages mid-dvc.yaml (via --force)

[BradyJ27]

As of right now, if you want to insert a new stage between two existing stages in a DVC pipeline, you must either edit the .yaml manually or remake the pipeline completely. See Discord discussion https://discord.com/channels/485586884165107732/485596304961962003/854010209596473364
My proposed feature it something along the lines of either:

1. dvc stage add --location where location is the stage number you want it to be. E.g. if I have stages: x, y, z and I do dvc stage add -n t --location 2 the stages would now be x, t, y, z

OR

2. dvc stage insert where you could insert a stage into an already existing pipeline.

Any thoughts/improvements are much appreciated.

Edit:

After further discussion, this is possible. The steps are dvc stage add and redefine the following stage with dvc stage add --force. Currently the .yaml file does not get updated, and the new stage gets added to the end. The desired behavior is the .yaml file to be either in order based on dependencies and outputs, OR overwriten stages should be moved to the bottom of the yaml file.

[jorgeorpinel]

Hi @BradyJ27

As of right now, if you want to insert a new stage between two existing stages in a DVC pipeline, you must either edit the .yaml manually or remake the pipeline completely.

This is not exactly the case. You can insert stages with dvc stage add now in 2 steps:

1. stage add the new stage in between by using the correct dependencies and outputs
2. Re-define the stage with that should follow with stage add --force, by using the new stage's outputs as dependencies.

The order in which they are presented in dvc.yaml is not important for DVC, only the way in which they connect (the dependency graph or DAG). But for your own benefit you can easily edit dvc.yaml so it's more readable to you if you prefer.

Does that clarify the situation?

[BradyJ27]

Thank you for your response @jorgeorpinel ! This does clarify. I'm still thinking there might be some room for a feature here, maybe even something as small as rearranging the yaml file for readability. Any thoughts on making this into something or is this unnecessary? Thanks again!

[jorgeorpinel]

An idea would be to move overwritten stages (stage add -f) to the bottom in dvc.yaml or even to some logical intermediate place as needed, instead of keeping them in their current location, which seems to be the current behavior (I just tried it).

[BradyJ27]

Good idea! I'm thinking either moving forced stages to the bottom (like you said). Or moving stages based on dependencies and outputs. I will update the issue description.

[skshetry]

dvc.yaml does not have a sense of order or even the pipelines. A part of the pipeline could be very well in a different place within the dvc.yaml file or in dvc.yaml of different directories. These days we recommend users to edit dvc.yaml directly rather than using commands, so if you need order, editing is the correct way to do it.

[jorgeorpinel]

Def. agree. But I did notice that stage add -f preserves the existing stage order in dvc.yaml so there must be some special logic to do that already. What do you think about just removing it from where it it and writing it in the bottom of the file? That may be more intuitive.

[skshetry]

But I did notice that stage add -f preserves the existing stage order in dvc.yaml so there must be some special logic to do that already.

Not really. stages is a dictionary/hashmap, so we just overwrite it if it already exists and the order gets preserved automatically. If there's no stage with the name, it gets added to the bottom. So, we are just exploiting the fact that the dictionaries in Python preserve insertion order.

What do you think about just removing it from where it it and writing it in the bottom of the file? That may be more intuitive.

We used to this before. See #4003 for the motivation to preserving order. But open to discussion for the change.

[jorgeorpinel]

the order gets preserved automatically

I see so if this is dependent on the implementation and that's how structures are handled by default by Python then that's probably the best behavior to keep.

So is this request valid, or should we decide not to go this route? After all it's unclear where to put stages that will be considered ideal in all cases, and users can always reorder the YAML structure manually.