[Feature] Add cache for table metadata in TabletServer

[luoyuxia]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, we have no cache for table metadata in TabletServer, every time get metadata request, it'll query from zk and return to client.. Normally, it's serveral milliseconds for getting metadata from zk, but it'll time cost for it when there're many buckets in the table since we'll have to query leader for each bucket..
In my env, for 512 buckets, it cost about 4s...

This problem become worse when start a new job to write job with already running some jobs to write Fluss.. When the new job starts, it'll get metadata which costs serveral seconds and then blocking request queue to degrade the write throughput for the jobs which cause a high latency in a period of time..

Solution

Add a cache for TabletServer to avoid get from zk every time..

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[Vipamp]

@luoyuxia hi，I'd like to resolve it，i will extract interface MetadataContext for manager metadata and ZK will be an implement，this will prepare for removing ZK in near future

[luoyuxia]

@Vipamp Thanks for attentiion... We already started this work...