[Enhancement] Display cluster-level information in the ReactFlow workspace

[ohltyler]

The current view of ingest and search pipelines in the ReactFlow workspace is ambiguous on where/when these resources are created. A few ways this can be improved.

1. (first goal) Where the resources are created. We should display the cluster (local or remote) within the workspace in a visual way. Perhaps a larger parent node encapsulating both the ingest and search flows (see example below). At the least, have more helpful text indicating that these pipelines are being built within the specified cluster.

2. (stretch goal): When the resources are created. Users will iteratively build out these pipelines, and we should reflect what is created only after the users create them, not just based on the UI configuration. This could mean showing the pipelines as greyed-out/disabled, or not shown at all, until they are concretely provisioned.

Example:

[dblock]

[Catch All Triage - 1, 2, 3, 4]

[ohltyler]

Another potential option for Item 2 is adding labeling/status per pipeline (or per-component) indicating if it has been provisioned or not. Overall, the "Preview" is supposed to show the proposed configuration, and does not need to be one-to-one parity with what is provisioned.

[kamingleung]

@ohltyler What are the statuses and states we have for Ingest/Search pipeline? Do the two pipelines have different states?

[ohltyler]

For each pipeline individually, we currently track:

1. If it has been provisioned or not
2. If the form differs than what's been provisioned (unsaved "provision" changes - e.g., configure and provision processor A, configure processor B, but not yet provisioned processor B)

Further per-processor granularity is available for ingest, and for search, there is ongoing work for that - see opensearch-project/OpenSearch#14745. I'd consider per-processor state a P2+-type item based on the backend dependencies.

[kamingleung]

For First goal, we can leverage the existing multiple data source picker on the top menu bar. It displays which cluster this workflow and resources are belonging to:

[kamingleung]

For the Second goal, we can consider displaying statuses to the pipelines.

We should dive deep into the different possible states and status before implementing this.

• One area to display this is in the preview of a pipeline like this with OuiHealth:
  

• We can also consider displaying the statuses in the resources lists: we can add a Status column to the resource table:
  

[kamingleung]

To help exploring the states, I am mapping out the lifecycle of the pipeline as users are working through the workflows:

Proposing:
Pipeline status:

• Inactive: The pipeline has not been created.
• Provisioning: When the pipeline is building or updating. (Are we able to get this state? How long does it usually take to build?)
• Active: The pipeline is running.

Save status

• Pending changes: Perhaps we can show a secondary status for tracking changes that has not been provisioned yet.

[ohltyler]

Thanks Ming. Regarding the ‘Provisioning’ state, it will be quite immediate, in my opinion I don’t think we want to / should show. It would just flash for a moment and could cause user confusion. The rest of those states make sense to me.

[kamingleung]

@ohltyler Got it. We may want to consider having this "Provisioning" state in the future to cover edge cases that requires more time.