enhance: record memory size (uncompressed) item for index #38770

chyezh · 2024-12-26T03:57:15Z

Current milvus use a serialized index size(compressed) for estimate resource for loading.
Add a new field MemSize (before compressing) for index to estimate resource.

sre-ci-robot · 2024-12-26T03:57:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign yanliang567 after the PR has been reviewed.
You can assign the PR to them by writing /assign @yanliang567 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-12-26T04:48:36Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T04:50:52Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-26T05:47:17Z

rerun cpp-unit-test

mergify · 2024-12-26T06:10:10Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T08:33:46Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-26T09:07:20Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T12:32:42Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-26T12:40:59Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-26T13:30:49Z

rerun cpp-unit-test

mergify · 2024-12-26T14:19:25Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

codecov · 2024-12-26T15:28:38Z

Codecov Report

Attention: Patch coverage is 69.09091% with 51 lines in your changes missing coverage. Please review.

Project coverage is 81.14%. Comparing base (11bfc93) to head (1831a35).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/index/CreateIndexResult.cpp	68.75%	10 Missing ⚠️
internal/util/indexcgowrapper/index.go	37.50%	9 Missing and 1 partial ⚠️
internal/core/src/index/TextMatchIndex.cpp	0.00%	8 Missing ⚠️
internal/core/src/common/protobuf_utils.cpp	50.00%	6 Missing ⚠️
internal/core/src/indexbuilder/index_c.cpp	0.00%	6 Missing ⚠️
internal/core/src/index/StringIndexMarisa.cpp	0.00%	5 Missing ⚠️
internal/core/src/index/ScalarIndexSort.cpp	0.00%	4 Missing ⚠️
internal/core/src/common/protobuf_utils.h	83.33%	1 Missing ⚠️
internal/core/src/storage/DiskFileManagerImpl.cpp	50.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #38770   +/-   ##
=======================================
  Coverage   81.14%   81.14%           
=======================================
  Files        1388     1392    +4     
  Lines      196471   196552   +81     
=======================================
+ Hits       159418   159498   +80     
- Misses      31465    31466    +1     
  Partials     5588     5588

Components	Coverage Δ
Client	`79.53% <ø> (ø)`
Core	`69.51% <59.80%> (-0.03%)`	⬇️
Go	`83.11% <84.12%> (+0.01%)`	⬆️

Files with missing lines	Coverage Δ
internal/core/src/index/BitmapIndex.cpp	`81.09% <100.00%> (+0.08%)`	⬆️
internal/core/src/index/BitmapIndex.h	`44.44% <ø> (ø)`
internal/core/src/index/CreateIndexResult.h	`100.00% <100.00%> (ø)`
internal/core/src/index/HybridScalarIndex.cpp	`79.12% <100.00%> (-0.33%)`	⬇️
internal/core/src/index/HybridScalarIndex.h	`39.02% <ø> (ø)`
internal/core/src/index/Index.h	`100.00% <ø> (ø)`
internal/core/src/index/InvertedIndexTantivy.cpp	`92.41% <100.00%> (-0.26%)`	⬇️
internal/core/src/index/InvertedIndexTantivy.h	`53.84% <ø> (ø)`
internal/core/src/index/ScalarIndexSort.h	`38.46% <ø> (ø)`
internal/core/src/index/StringIndexMarisa.h	`71.42% <ø> (ø)`
... and 29 more

... and 23 files with indirect coverage changes

chyezh · 2024-12-27T06:39:30Z

rerun cpp-unit-test

mergify · 2024-12-27T07:27:22Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-27T07:30:26Z

rerun cpp-unit-test

chyezh · 2024-12-27T07:30:45Z

rerun go-sdk

mergify · 2024-12-27T08:13:53Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-27T08:24:07Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2024-12-27T16:13:07Z

rerun go-sdk

chyezh · 2024-12-27T16:13:12Z

rerun cpp-unit-test

mergify · 2024-12-27T17:01:06Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-28T04:57:27Z

rerun cpp-unit-test

mergify · 2024-12-28T05:37:14Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-29T12:24:26Z

rerun cpp-unit-test

mergify · 2024-12-29T12:47:03Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-30T07:05:30Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-30T08:02:07Z

rerun cpp-unit-test

mergify · 2024-12-30T08:49:01Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-30T11:08:20Z

rerun cpp-unit-test

mergify · 2024-12-30T11:50:31Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-30T13:37:36Z

rerun cpp-unit-test

mergify · 2024-12-30T14:23:43Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

chyezh · 2024-12-31T01:51:21Z

rerun cpp-unit-test

mergify · 2024-12-31T02:36:53Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-31T10:44:13Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-31T11:03:53Z

@chyezh cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-31T12:11:19Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-01-02T03:08:49Z

rerun go-sdk

mergify · 2025-01-02T03:58:24Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-01-02T06:11:11Z

rerun go-sdk

mergify · 2025-01-02T06:48:39Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

chyezh · 2025-01-02T07:05:24Z

rerun go-sdk

mergify · 2025-01-02T08:00:19Z

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

- Current milvus use a serialized index size(compressed) for estimate resource for loading. - Add a new field MemSize (before compressing) for index to estimate resource. Signed-off-by: chyezh <[email protected]>

Signed-off-by: chyezh <[email protected]>

mergify · 2025-01-05T04:18:33Z

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

chyezh · 2025-01-05T05:31:15Z

/run-cpu-e2e

sre-ci-robot added area/internal-api size/XXL Denotes a PR that changes 1000+ lines. labels Dec 26, 2024

sre-ci-robot requested review from congqixia and czs007 December 26, 2024 03:57

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Dec 26, 2024

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from 4cb8035 to 1ebba3e Compare December 26, 2024 11:44

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch 2 times, most recently from 07910b5 to 5a89096 Compare December 30, 2024 06:21

chyezh mentioned this pull request Dec 30, 2024

enhance: record memory size (uncompressed) item for index #38844

Open

chyezh added 3 commits January 5, 2025 11:03

enhance: record memory size (uncompressed) item for index

b2febda

- Current milvus use a serialized index size(compressed) for estimate resource for loading. - Add a new field MemSize (before compressing) for index to estimate resource. Signed-off-by: chyezh <[email protected]>

fix: unittest

27d5080

Signed-off-by: chyezh <[email protected]>

fix: unittest

1831a35

Signed-off-by: chyezh <[email protected]>

chyezh force-pushed the enhance_serialized_index_file_multiply_factor branch from dbf4b36 to 1831a35 Compare January 5, 2025 03:03

mergify bot added the ci-passed label Jan 5, 2025

enhance: record memory size (uncompressed) item for index #38770

Are you sure you want to change the base?

enhance: record memory size (uncompressed) item for index #38770

Conversation

chyezh commented Dec 26, 2024

sre-ci-robot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

chyezh commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

chyezh commented Dec 26, 2024

mergify bot commented Dec 26, 2024

codecov bot commented Dec 26, 2024 • edited Loading

Codecov Report

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 27, 2024

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 27, 2024

chyezh commented Dec 27, 2024

mergify bot commented Dec 27, 2024

chyezh commented Dec 28, 2024

mergify bot commented Dec 28, 2024

chyezh commented Dec 29, 2024

mergify bot commented Dec 29, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 30, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 30, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 30, 2024

mergify bot commented Dec 30, 2024

chyezh commented Dec 31, 2024

mergify bot commented Dec 31, 2024

mergify bot commented Dec 31, 2024

mergify bot commented Dec 31, 2024

mergify bot commented Dec 31, 2024

chyezh commented Jan 2, 2025

mergify bot commented Jan 2, 2025

chyezh commented Jan 2, 2025

mergify bot commented Jan 2, 2025

chyezh commented Jan 2, 2025

mergify bot commented Jan 2, 2025

mergify bot commented Jan 5, 2025

chyezh commented Jan 5, 2025

codecov bot commented Dec 26, 2024 •

edited

Loading