[da-vinci][common] DVC consumer for materialized view (batch only)

[xunyin8]

[da-vinci][common] DVC consumer for materialized view (batch only)

Limiting the change to support batch only materialized view to keep the PR small. Hybrid DVC consumer support and related features such as heartbeat will be added in a separate PR

1. New DaVinciClientFactory APIs for creating DVC for a given view.

2. Defined a new "storeName" rule for views to be used for metrics reporting and DVC client. See VeniceView.getViewStoreName for details.

3. Introduced NativeMetadataRepositoryViewAdapter and HelixReadOnlyStoreViewConfigRepositoryAdapter to provide read-only interface to access various store metadata for both regular Venice stores and store views with the VeniceView.getViewStoreName.

There is some issue with chunking support on the read path. When chunking is enabled the view topic keys are doubly wrapped by serializeNonChunkedKey. This is because during NR pass-through mode the view writer is essentially trying to chunk the chunk. The tactical fix now is to unwrap the key with chunked suffix bytes appended and pass it to the view writer to be wrapped again and sent to the correct partition. This only works with non-large messages. I.e. chunking is enabled but nothing is actually getting chunked. Large messages will require a proper fix.

How was this PR tested?

Unit tests and integration test

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change.

[gaojieliu]

Hi @xunyin8
I have a big question about how the adapting layer looks like and I read your PR and I think you are trying to plugin the adapting layer in a more spread fashion where sometimes we use the original store name and some other times, we use the combined name by concatenating store name and view name.
There are two points I can think of:

1. The code would become less maintainable as we need to make a decision at different places to use either the original store name or the concatenated name.
2. In theory, this solution is not 100% safe as some store name can be identical to the concatenated name, and if the customer subscribes both, it will encounter weird issues.

Here is one idea in my mind:

1. Make sure the concatenated name unique in Venice platform, and it means we need to guarantee the uniqueness among store names and concatenated view names.
2. In DaVinci world, we can treat the concatenated view name as the store name, and we can build a new adapting layer in the common repositories, such as StoreRepository, SchemaRepository and they need to handle both view and store request. For example, for the passed param, it can detect whether it is a view or a store and if it is a view, it will find out the corresponding store name and take the corresponding actions.
   With this way, it will be very safe to avoid conflict and we only need to change very few places to make it work everywhere and the idea is similar to HelixReadOnlySchemaRepositoryAdapter, which can handle both system stores and user stores in a transparent way.
3. Less error prone as the adapting logics are only in several important repositories, nowhere else (in theory).

If we build such capability, the change in DVRT or CDC will be fairly straightforward.

WDYT?

[xunyin8]

Member

Hi @xunyin8 I have a big question about how the adapting layer looks like and I read your PR and I think you are trying to plugin the adapting layer in a more spread fashion where sometimes we use the original store name and some other times, we use the combined name by concatenating store name and view name. There are two points I can think of:

1. The code would become less maintainable as we need to make a decision at different places to use either the original store name or the concatenated name.
2. In theory, this solution is not 100% safe as some store name can be identical to the concatenated name, and if the customer subscribes both, it will encounter weird issues.

Here is one idea in my mind:

1. Make sure the concatenated name unique in Venice platform, and it means we need to guarantee the uniqueness among store names and concatenated view names.
2. In DaVinci world, we can treat the concatenated view name as the store name, and we can build a new adapting layer in the common repositories, such as StoreRepository, SchemaRepository and they need to handle both view and store request. For example, for the passed param, it can detect whether it is a view or a store and if it is a view, it will find out the corresponding store name and take the corresponding actions.
   With this way, it will be very safe to avoid conflict and we only need to change very few places to make it work everywhere and the idea is similar to HelixReadOnlySchemaRepositoryAdapter, which can handle both system stores and user stores in a transparent way.
3. Less error prone as the adapting logics are only in several important repositories, nowhere else (in theory).

If we build such capability, the change in DVRT or CDC will be fairly straightforward.

WDYT?

This makes sense, I was hoping to be able to keep the adapting layer only in StoreBackend and ReadOnlyVersion. However, as you mentioned and probably noticed while reviewing the PR other components of DVC and ingestion path also requires differentiating a store view from a store hence the adapter keeps leaking. I think we can enforce (1) fairly easily during view/store creation and (2) could introduce more initial change but let me give it a try and it may be able to solve some of the known issue (e.g. chunking not working) cleanly since the suggested solution is more bullet proof.

[gaojieliu]

Left some more comments about the adapting layer?

BTW, can you remind me of the place where we can guarantee the uniqueness of view names?
One way we can do is to make sure the store name doesn't contain the special separator being used to concatenate the store name and view name.

[xunyin8]

Left some more comments about the adapting layer?

BTW, can you remind me of the place where we can guarantee the uniqueness of view names? One way we can do is to make sure the store name doesn't contain the special separator being used to concatenate the store name and view name.

Currently we don't check uniqueness of view names across different stores and I think that's fine. The way we make sure view store names can be processed is with added check to ensure store names and view names do not contain view separator:
See VeniceHelixAdmin#checkStoreNameConflict

    if (VeniceView.isViewStore(storeName)) {
      throw new VeniceException(
          "Store name: " + storeName + " clashes with the internal usage, please remove the prefix: "
              + VeniceView.VIEW_STORE_PREFIX);
    }

and validateAndDecorateStoreViewConfig

      if (viewName.contains(VIEW_NAME_SEPARATOR)) {
        throw new VeniceException(
            String.format("View name cannot contain view name separator: %s", VIEW_NAME_SEPARATOR));
      }

[gaojieliu]

Code change looks good overall and left some more comments.

[gaojieliu]

Thanks!