misc: Add partitioned writer count metrics to hive data sink #12410
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
|
@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
tanjialiang
force-pushed
the
p_writer_metric
branch
2 times, most recently
from
88edacd to
70aced7
Compare
|
@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
bikramSingh91
approved these changes
Feb 20, 2025
tanjialiang
force-pushed
the
p_writer_metric
branch
from
70aced7 to
a048901
Compare
|
@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
tanjialiang
force-pushed
the
p_writer_metric
branch
from
a048901 to
a7be2f1
Compare
tanjialiang
added a commit
to tanjialiang/velox-1
that referenced
this pull request
Feb 20, 2025
…kincubator#12410) Summary: large number of partitioned writers may cause issues like large number of open files and memory consumption. Adding this runtime metrics to disclose this critical stats for more efficient debugging related issues. Reviewed By: bikramSingh91 Differential Revision: D69936854 Pulled By: tanjialiang
|
This pull request was exported from Phabricator. Differential Revision: D69936854 |
xiaoxmeng
reviewed
Feb 21, 2025
| @@ -806,6 +806,8 @@ uint32_t HiveDataSink::appendWriter(const HiveWriterId& id) { | |||
| .stats = ioStats_.back().get(), | |||
| }), | |||
| options); | |||
| addThreadLocalRuntimeStat( | |||
There was a problem hiding this comment.
Shall we report once for each table write operator so we can see the distribution better?
tanjialiang
force-pushed
the
p_writer_metric
branch
from
a7be2f1 to
4520862
Compare
|
@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
xiaoxmeng
approved these changes
Feb 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
large number of partitioned writers may cause issues like large number of open files and memory consumption. Adding this runtime metrics to disclose this critical stats for more efficient debugging related issues.