milvus2.4.6，单机standalone_embed.sh启动报这个错,容器 unhealthy #38934

jhyever · 2025-01-02T06:58:49Z

jhyever
Jan 2, 2025

@yhmo 麻烦帮忙看看，感谢

[2025/01/02 06:38:33.793 +00:00] [INFO] [datacoord/channel_manager_v2.go:646] ["Notify channel operation"] [channel=by-dev-rootcoord-dml_11_454254081415433293v1] [assignment=11] [operation=ToWatch]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.793 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=datanode-11] [addr=]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.794 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.856 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=querynode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:33.900 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=querynode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.900 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=querynode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.901 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=querynode-11] [addr=]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.996 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.996 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.996 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.995 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.996 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.996 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.996 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:33.996 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=datanode-11] [addr=]
[2025/01/02 06:38:34.263 +00:00] [INFO] [observers/target_observer.go:463] ["Update readable segment version"] [collectionID=454254081416011250] [channelName=by-dev-rootcoord-dml_8_454254081416011250v0] [nodeID=12] [oldVersion=0] [newVersion=1735799904272369285]
[2025/01/02 06:38:34.264 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=d75b033a629436da1b12c05dfba75fcd] [collectionID=454254081415433293]
[2025/01/02 06:38:34.265 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=9b62e619ee2ad836d42aab2f9b9529be] [collectionID=454254081416011250]
[2025/01/02 06:38:34.265 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=9023b7ddaa92e3967df65395434065ca] [collectionID=454254081415433293]
[2025/01/02 06:38:34.265 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=f6cd4502c07d794a68b1f81d7f59b0a7] [collectionID=454254081415433258]
[2025/01/02 06:38:34.265 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=47e117af72f8573209c9566fffdd0d86] [collectionID=454254081416011210]
[2025/01/02 06:38:34.265 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=cf9c78d9a04fd295b3be47718814c8f4] [collectionID=454254081416011210]
[2025/01/02 06:38:34.266 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=5575c216fa99f27c862953300b349d02] [collectionID=454254081416011250] [channel=by-dev-rootcoord-dml_8_454254081416011250v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904272369285]
[2025/01/02 06:38:34.266 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=7cd5891220adecbfdab7df929da4958c] [collectionID=454254081416011210] [channel=by-dev-rootcoord-dml_6_454254081416011210v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904269166656]
[2025/01/02 06:38:34.266 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904272369285] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/01/02 06:38:34.266 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=eba5d12965a7dadb0f1b080df00e763d] [collectionID=454254081416011250]
[2025/01/02 06:38:34.266 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=dddf22408aeded9c5b7ea0474b502953] [collectionID=454254081415433293] [channel=by-dev-rootcoord-dml_11_454254081415433293v1] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904266107822]
[2025/01/02 06:38:34.266 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904269166656] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/01/02 06:38:34.266 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=90d52a01fa520226fc8fff81930ab6de] [collectionID=454254081415433258] [channel=by-dev-rootcoord-dml_5_454254081415433258v1] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904263232165]
[2025/01/02 06:38:34.266 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904266107822] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/01/02 06:38:34.266 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904263232165] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/01/02 06:38:34.267 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=cb636ac2524d4cca4abdbb81b5622e62] [collectionID=454254081416011250]
[2025/01/02 06:38:34.267 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=c243a55936cf2b6d541a48d5e1857890] [collectionID=454254081415433293]
[2025/01/02 06:38:34.267 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=680c8013278c187abbc3a42b296679f4] [collectionID=454254081416011210]
[2025/01/02 06:38:34.267 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=0f780a255a2fc0fb6ca5a35cd5ddc4e4] [collectionID=454254081415433258]
[2025/01/02 06:38:34.267 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=a52c605eba6a8005d0bae4ac3bfec5a2] [collectionID=454254081415433258]
[2025/01/02 06:38:34.268 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=ccc1ed01e2e01797399bb88e45ed3567] [collectionID=454254081416011250] [channel=by-dev-rootcoord-dml_9_454254081416011250v1] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904272369285]
[2025/01/02 06:38:34.268 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=e08c19eb4d89a25c3072ad9ecd31a688] [collectionID=454254081415433293] [channel=by-dev-rootcoord-dml_10_454254081415433293v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904266107822]
[2025/01/02 06:38:34.268 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904272369285] [growingSegmentNum=1] [sealedSegmentNum=1]
[2025/01/02 06:38:34.268 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=71735f41efd9f82c50feed80be0c2ebe] [collectionID=454254081415433258] [channel=by-dev-rootcoord-dml_4_454254081415433258v0] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904263232165]
[2025/01/02 06:38:34.268 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904266107822] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/01/02 06:38:34.268 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=de56268065912df9e1fc7f1671b50175] [collectionID=454254081416011210] [channel=by-dev-rootcoord-dml_7_454254081416011210v1] [currentNodeID=12] [Action=UpdateVersion] [TargetVersion=1735799904269166656]
[2025/01/02 06:38:34.268 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904263232165] [growingSegmentNum=0] [sealedSegmentNum=1]
[2025/01/02 06:38:34.268 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1735799904269166656] [growingSegmentNum=0] [sealedSegmentNum=0]
[2025/01/02 06:38:34.268 +00:00] [INFO] [observers/target_observer.go:493] ["observer trigger update current target"] [collectionID=454254081416011250]
[2025/01/02 06:38:34.268 +00:00] [INFO] [observers/target_observer.go:493] ["observer trigger update current target"] [collectionID=454254081415433293]
[2025/01/02 06:38:34.268 +00:00] [INFO] [observers/target_observer.go:493] ["observer trigger update current target"] [collectionID=454254081416011210]
[2025/01/02 06:38:34.268 +00:00] [INFO] [observers/target_observer.go:493] ["observer trigger update current target"] [collectionID=454254081415433258]
[2025/01/02 06:38:34.268 +00:00] [INFO] [datacoord/services.go:820] ["get recovery info request received"] [traceID=aeeec6ab7a84c19d121decc55f393fbd] [collectionID=454254081416011250] [partitionIDs="[]"]
[2025/01/02 06:38:34.268 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081416011250] [channel=by-dev-rootcoord-dml_8_454254081416011250v0] [numOfSegments=2] ["indexed segment"=1]
[2025/01/02 06:38:34.268 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_8_454254081416011250v0] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=aeeec6ab7a84c19d121decc55f393fbd] [collectionID=454254081416011250] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_8_454254081416011250v0] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081416011250] [channel=by-dev-rootcoord-dml_9_454254081416011250v1] [numOfSegments=2] ["indexed segment"=1]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:820] ["get recovery info request received"] [traceID=e2f42108f5fe93ea6a498d93ae43761e] [collectionID=454254081415433293] [partitionIDs="[]"]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_9_454254081416011250v1] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=aeeec6ab7a84c19d121decc55f393fbd] [collectionID=454254081416011250] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_9_454254081416011250v1] ["# of unflushed segments"=1] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:820] ["get recovery info request received"] [traceID=120525cb58e2273f705838aa552249fc] [collectionID=454254081416011210] [partitionIDs="[]"]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:820] ["get recovery info request received"] [traceID=b164db5d829fd15a165cbdc47e014f36] [collectionID=454254081415433258] [partitionIDs="[]"]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081416011210] [channel=by-dev-rootcoord-dml_6_454254081416011210v0] [numOfSegments=0] ["indexed segment"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081415433258] [channel=by-dev-rootcoord-dml_5_454254081415433258v1] [numOfSegments=1] ["indexed segment"=1]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_454254081416011210v0] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_5_454254081415433258v1] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=120525cb58e2273f705838aa552249fc] [collectionID=454254081416011210] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_6_454254081416011210v0] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=b164db5d829fd15a165cbdc47e014f36] [collectionID=454254081415433258] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_5_454254081415433258v1] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081415433258] [channel=by-dev-rootcoord-dml_4_454254081415433258v0] [numOfSegments=1] ["indexed segment"=1]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081415433293] [channel=by-dev-rootcoord-dml_10_454254081415433293v0] [numOfSegments=1] ["indexed segment"=1]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_10_454254081415433293v0] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=e2f42108f5fe93ea6a498d93ae43761e] [collectionID=454254081415433293] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_10_454254081415433293v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081416011210] [channel=by-dev-rootcoord-dml_7_454254081416011210v1] [numOfSegments=0] ["indexed segment"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=454254081415433293] [channel=by-dev-rootcoord-dml_11_454254081415433293v1] [numOfSegments=1] ["indexed segment"=1]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_11_454254081415433293v1] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=e2f42108f5fe93ea6a498d93ae43761e] [collectionID=454254081415433293] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_11_454254081415433293v1] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_7_454254081416011210v1] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_4_454254081415433258v0] [posTs=455029510535643137] [posTime=2025/01/02 06:37:09.619 +00:00]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=b164db5d829fd15a165cbdc47e014f36] [collectionID=454254081415433258] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_4_454254081415433258v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.269 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=120525cb58e2273f705838aa552249fc] [collectionID=454254081416011210] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_7_454254081416011210v1] ["# of unflushed segments"=0] ["# of flushed segments"=0] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [traceID=429787f771bece43482f408fb866afac] [clientRole=querynode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [traceID=16c86c4959bed090d091cf557c5251c2] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.398 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.397 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.398 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.398 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=datanode-11] [addr=]
[2025/01/02 06:38:34.398 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.398 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.398 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=proxy-11] [addr=]
[2025/01/02 06:38:34.657 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=querynode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:34.657 +00:00] [WARN] [grpcclient/client.go:560] ["ClientBase Call grpc call get error"] [role=querynode-11] [address=172.17.0.2:21123] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querynode/client/client.go:104 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetComponentStates\n/workspace/source/internal/querycoordv2/session/cluster.go:252 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates.func1\n/workspace/source/internal/querycoordv2/session/cluster.go:271 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n/workspace/source/internal/querycoordv2/session/cluster.go:251 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates\n/workspace/source/internal/querycoordv2/services.go:894 github.com/milvus-io/milvus/internal/querycoordv2.(*Server).checkNodeHealth.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:559\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querynode/client/client.go:90\n | github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/querynode/client/client.go:104\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates.func1\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:252\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:271\n | github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates\n | \t/workspace/source/internal/querycoordv2/session/cluster.go:251\n | github.com/milvus-io/milvus/internal/querycoordv2.(*Server).checkNodeHealth.func1\n | \t/workspace/source/internal/querycoordv2/services.go:894\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/querynode/client/client.go:104 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).GetComponentStates\n | /workspace/source/internal/querycoordv2/session/cluster.go:252 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates.func1\n | /workspace/source/internal/querycoordv2/session/cluster.go:271 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send\n | /workspace/source/internal/querycoordv2/session/cluster.go:251 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).GetComponentStates\n | /workspace/source/internal/querycoordv2/services.go:894 github.com/milvus-io/milvus/internal/querycoordv2.(*Server).checkNodeHealth.func1\n | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2025/01/02 06:38:34.657 +00:00] [WARN] [querycoordv2/server.go:740] ["unhealthy node exist, node up will be delayed"] [delayedNodeUpEvents=0] [unhealthyNodeNum=0] [unhealthyReason="[]"]
[2025/01/02 06:38:34.703 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=querynode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.703 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=querynode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:34.703 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=querynode-11] [addr=]
[2025/01/02 06:38:34.954 +00:00] [WARN] [rootcoord/timeticksync.go:168] ["session idle for long time"] ["idle list"="[11]"] ["idle time"=14000]
[2025/01/02 06:38:34.983 +00:00] [WARN] [rootcoord/timeticksync.go:168] ["session idle for long time"] ["idle list"="[11]"] ["idle time"=14000]
[2025/01/02 06:38:35.182 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.182 +00:00] [WARN] [grpcclient/client.go:560] ["ClientBase Call grpc call get error"] [role=datanode-11] [address=172.17.0.2:21124] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datanode/client/client.go:248 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).QueryImport\n/workspace/source/internal/datacoord/session_manager.go:429 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).QueryImport\n/workspace/source/internal/datacoord/cluster.go:171 github.com/milvus-io/milvus/internal/datacoord.(*ClusterImpl).QueryImport\n/workspace/source/internal/datacoord/import_scheduler.go:156 github.com/milvus-io/milvus/internal/datacoord.(*importScheduler).peekSlots.func2\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:559\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datanode/client/client.go:93\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).QueryImport\n | \t/workspace/source/internal/distributed/datanode/client/client.go:248\n | github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).QueryImport\n | \t/workspace/source/internal/datacoord/session_manager.go:429\n | github.com/milvus-io/milvus/internal/datacoord.(*ClusterImpl).QueryImport\n | \t/workspace/source/internal/datacoord/cluster.go:171\n | github.com/milvus-io/milvus/internal/datacoord.(*importScheduler).peekSlots.func2\n | \t/workspace/source/internal/datacoord/import_scheduler.go:156\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/datanode/client/client.go:248 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).QueryImport\n | /workspace/source/internal/datacoord/session_manager.go:429 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).QueryImport\n | /workspace/source/internal/datacoord/cluster.go:171 github.com/milvus-io/milvus/internal/datacoord.(*ClusterImpl).QueryImport\n | /workspace/source/internal/datacoord/import_scheduler.go:156 github.com/milvus-io/milvus/internal/datacoord.(*importScheduler).peekSlots.func2\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2025/01/02 06:38:35.182 +00:00] [WARN] [datacoord/import_scheduler.go:158] ["query import failed"] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datanode/client/client.go:248 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).QueryImport\n/workspace/source/internal/datacoord/session_manager.go:429 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).QueryImport\n/workspace/source/internal/datacoord/cluster.go:171 github.com/milvus-io/milvus/internal/datacoord.(*ClusterImpl).QueryImport\n/workspace/source/internal/datacoord/import_scheduler.go:156 github.com/milvus-io/milvus/internal/datacoord.(*importScheduler).peekSlots.func2\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:559\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datanode/client/client.go:93\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).QueryImport\n | \t/workspace/source/internal/distributed/datanode/client/client.go:248\n | github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).QueryImport\n | \t/workspace/source/internal/datacoord/session_manager.go:429\n | github.com/milvus-io/milvus/internal/datacoord.(*ClusterImpl).QueryImport\n | \t/workspace/source/internal/datacoord/cluster.go:171\n | github.com/milvus-io/milvus/internal/datacoord.(*importScheduler).peekSlots.func2\n | \t/workspace/source/internal/datacoord/import_scheduler.go:156\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/datanode/client/client.go:248 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).QueryImport\n | /workspace/source/internal/datacoord/session_manager.go:429 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).QueryImport\n | /workspace/source/internal/datacoord/cluster.go:171 github.com/milvus-io/milvus/internal/datacoord.(*ClusterImpl).QueryImport\n | /workspace/source/internal/datacoord/import_scheduler.go:156 github.com/milvus-io/milvus/internal/datacoord.(*importScheduler).peekSlots.func2\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2025/01/02 06:38:35.183 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:35.183 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:35.183 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=datanode-11] [addr=]
[2025/01/02 06:38:35.184 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:35.199 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.200 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.199 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.200 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.200 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.200 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.200 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.200 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.385 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:35.386 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:35.386 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=datanode-11] [addr=]
[2025/01/02 06:38:35.788 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:35.789 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:35.789 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=datanode-11] [addr=]
[2025/01/02 06:38:35.997 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [traceID=429787f771bece43482f408fb866afac] [clientRole=querynode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.998 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [traceID=429787f771bece43482f408fb866afac] [retried=4] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.998 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [traceID=16c86c4959bed090d091cf557c5251c2] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:35.998 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [traceID=16c86c4959bed090d091cf557c5251c2] [retried=4] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:36.000 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:36.000 +00:00] [WARN] [grpcclient/client.go:492] ["start to reset connection because of specific reasons"] [client_role=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:36.000 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=proxy-11] [addr=]
[2025/01/02 06:38:36.001 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=4] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 06:38:36.175 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2025/01/02 06:38:36.175 +00:00] [WARN] [grpcclient/client.go:560] ["ClientBase Call grpc call get error"] [role=datanode-11] [address=172.17.0.2:21124] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n/workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:559\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datanode/client/client.go:93\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | \t/workspace/source/internal/distributed/datanode/client/client.go:188\n | github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | \t/workspace/source/internal/datacoord/session_manager.go:262\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | /workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2025/01/02 06:38:36.175 +00:00] [INFO] [datacoord/session_manager.go:270] ["Get State failed"] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n/workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:559\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datanode/client/client.go:93\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | \t/workspace/source/internal/distributed/datanode/client/client.go:188\n | github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | \t/workspace/source/internal/datacoord/session_manager.go:262\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | /workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2025/01/02 06:38:36.175 +00:00] [WARN] [datacoord/compaction.go:534] ["failed to get compaction plans from all nodes"] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n/workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:559\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datanode/client/client.go:93\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | \t/workspace/source/internal/distributed/datanode/client/client.go:188\n | github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | \t/workspace/source/internal/datacoord/session_manager.go:262\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | /workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2025/01/02 06:38:36.175 +00:00] [WARN] [datacoord/compaction.go:150] ["fail to update compaction"] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n/workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:559\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:573\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datanode/client/client.go:93\n | github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | \t/workspace/source/internal/distributed/datanode/client/client.go:188\n | github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | \t/workspace/source/internal/datacoord/session_manager.go:262\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:559 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:573 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/datanode/client/client.go:93 github.com/milvus-io/milvus/internal/distributed/datanode/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/datanode/client/client.go:188 github.com/milvus-io/milvus/internal/distributed/datanode/client.(*Client).GetCompactionState\n | /workspace/source/internal/datacoord/session_manager.go:262 github.com/milvus-io/milvus/internal/datacoord.(*SessionManagerImpl).GetCompactionPlansResults.func1\n | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2025/01/02 06:38:36.177 +00:00] [WARN] [grpcclient/client.go:408] ["call received grpc error"] [clientRole=datanode-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]

yhmo · 2025-01-02T08:52:54Z

yhmo
Jan 2, 2025
Collaborator

日志不完整，这些报错应该不是根源

1 reply

jhyever Jan 2, 2025
Author

这个是最前面的报错了，后面都一样了，但是可以正常连接，后面就都是info日志了，但是milvus容器是显示unhealthy，不知道为什么这样
[2025/01/02 08:37:41.146 +00:00] [INFO] [roles/roles.go:256] [setupPrometheusHTTPServer]
[2025/01/02 08:37:41.146 +00:00] [INFO] [http/server.go:152] ["management listen"] [addr=:9091]
[2025/01/02 08:37:41.146 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
[2025/01/02 08:37:41.146 +00:00] [DEBUG] [rootcoord/service.go:184] ["init params done.."]
[2025/01/02 08:37:41.146 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.146 +00:00] [DEBUG] [rootcoord/service.go:204] ["etcd connect done ..."]
[2025/01/02 08:37:41.146 +00:00] [DEBUG] [rootcoord/service.go:264] ["start grpc "] [port=53100]
[2025/01/02 08:37:41.146 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.146 +00:00] [DEBUG] [datacoord/service.go:150] ["network port"] [port=13333]
[2025/01/02 08:37:41.146 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."]
[2025/01/02 08:37:41.146 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.146 +00:00] [DEBUG] [querycoord/service.go:218] [network] [port=19531]
[2025/01/02 08:37:41.147 +00:00] [DEBUG] [querynode/service.go:104] [QueryNode] [port=21123]
[2025/01/02 08:37:41.147 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.147 +00:00] [DEBUG] [querynode/service.go:124] ["QueryNode connect to etcd successfully"]
[2025/01/02 08:37:41.147 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.147 +00:00] [INFO] [datanode/service.go:256] ["DataNode address"] [address=172.17.0.2:21124]
[2025/01/02 08:37:41.147 +00:00] [INFO] [datanode/service.go:257] ["DataNode serverID"] [serverID=0]
[2025/01/02 08:37:41.147 +00:00] [DEBUG] [indexnode/indexnode.go:115] ["New IndexNode ..."]
[2025/01/02 08:37:41.147 +00:00] [DEBUG] [indexnode/service.go:87] [IndexNode] ["network address"=172.17.0.2:21121] ["network port: "=21121]
[2025/01/02 08:37:41.149 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"]
[2025/01/02 08:37:41.149 +00:00] [INFO] [runtime/asm_amd64.s:1650] ["Start check query node health loop"]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:225] ["RateLimiter register for rateType"] [rateType=DDLCollection] [rateLimit=+inf] [burst=1.7976931348623157e+308]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:225] ["RateLimiter register for rateType"] [rateType=DDLFlush] [rateLimit=+inf] [burst=1.7976931348623157e+308]
[2025/01/02 08:37:41.149 +00:00] [INFO] [hookutil/hook.go:46] ["empty so path, skip to load plugin"]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/service.go:122] ["create a new Proxy instance"] [state=2]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/service.go:423] ["init Proxy server"]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/service.go:454] ["Proxy init service's parameter table done"]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/service.go:456] ["Proxy init http server's parameter table done"]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/service.go:463] ["init Proxy's parameter table done"] [internalAddress=172.17.0.2:19529] [externalAddress=172.17.0.2:19530]
[2025/01/02 08:37:41.149 +00:00] [INFO] [accesslog/global.go:145] ["Init access logger success"]
[2025/01/02 08:37:41.149 +00:00] [DEBUG] [proxy/service.go:470] ["init Proxy's tracer done"] ["service name"="Proxy ip: 172.17.0.2, port: 19530"]
[2025/01/02 08:37:41.149 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.149 +00:00] [INFO] [proxy/service.go:370] ["Proxy internal server listen on tcp"] [port=19529]
[2025/01/02 08:37:41.149 +00:00] [INFO] [proxy/service.go:377] ["Proxy internal server already listen on tcp"] [port=19529]
[2025/01/02 08:37:41.150 +00:00] [INFO] [proxy/service.go:409] ["create Proxy internal grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"]
[2025/01/02 08:37:41.150 +00:00] [INFO] [proxy/service.go:502] ["Proxy server listen on tcp"] [port=19530]
[2025/01/02 08:37:41.150 +00:00] [INFO] [proxy/service.go:505] ["Proxy server already listen on tcp"] [port=19530]
[2025/01/02 08:37:41.150 +00:00] [DEBUG] [proxy/service.go:266] ["Get proxy rate limiter done"] [port=19530]
[2025/01/02 08:37:41.150 +00:00] [DEBUG] [proxy/service.go:345] ["create Proxy grpc server"] ["enforcement policy"="{"MinTime":5000000000,"PermitWithoutStream":true}"] ["server parameters"="{"MaxConnectionIdle":0,"MaxConnectionAge":0,"MaxConnectionAgeGrace":0,"Time":60000000000,"Timeout":10000000000}"]
[2025/01/02 08:37:41.150 +00:00] [INFO] [proxy/service.go:586] ["register Proxy http server"]
[2025/01/02 08:37:41.150 +00:00] [DEBUG] [proxy/service.go:593] ["create RootCoord client for Proxy"]
[2025/01/02 08:37:41.150 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.150 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
{"level":"info","ts":"2025-01-02T08:37:41.151Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 12; dropping index reading msg"}
{"level":"info","ts":"2025-01-02T08:37:41.196Z","caller":"etcdserver/v3_server.go:832","msg":"first commit in current term: resending ReadIndex request"}
{"level":"info","ts":"2025-01-02T08:37:41.196Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 12; dropping index reading msg"}
[2025/01/02 08:37:41.246 +00:00] [DEBUG] [rootcoord/service.go:221] ["grpc init done ..."]
[2025/01/02 08:37:41.246 +00:00] [DEBUG] [rootcoord/service.go:224] ["RootCoord start to create DataCoord client"]
[2025/01/02 08:37:41.247 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.247 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2025/01/02 08:37:41.247 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.247 +00:00] [DEBUG] [server/global_rmq.go:39] ["initializing global rmq"] [path=/var/lib/milvus/rdb_data]
[2025/01/02 08:37:41.248 +00:00] [INFO] [datanode/service.go:266] ["initializing RootCoord client for DataNode"]
[2025/01/02 08:37:41.248 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=true] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2025/01/02 08:37:41.248 +00:00] [DEBUG] [querynode/service.go:134] [QueryNode] [State=Initializing]
[2025/01/02 08:37:41.248 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.248 +00:00] [INFO] [indexnode/indexnode.go:207] ["IndexNode init"] [state=Initializing]
[2025/01/02 08:37:41.248 +00:00] [INFO] [querynodev2/server.go:286] ["QueryNode session info"] [metaPath=by-dev/meta]
[2025/01/02 08:37:41.248 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.248 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.255 +00:00] [DEBUG] [server/rocksmq_impl.go:177] ["Start rocksmq"] ["max proc"=32] [parallism=2] ["lru cache"=4043128012]
{"level":"info","ts":"2025-01-02T08:37:41.276Z","caller":"etcdserver/v3_server.go:832","msg":"first commit in current term: resending ReadIndex request"}
{"level":"info","ts":"2025-01-02T08:37:41.276Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d no leader at term 12; dropping index reading msg"}
[2025/01/02 08:37:41.383 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2025/01/02 08:31:06.496 +00:00] [save=2025/01/02 08:37:44.383 +00:00] [next=2025/01/02 08:37:41.383 +00:00]
[2025/01/02 08:37:41.383 +00:00] [INFO] [msgstream/mq_factory.go:18] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2025/01/02 08:37:41.384 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.383 +00:00] [DEBUG] [server/rocksmq_retention.go:80] ["Rocksmq retention goroutine start!"]
{"level":"info","ts":"2025-01-02T08:37:41.425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d is starting a new election at term 12"}
{"level":"info","ts":"2025-01-02T08:37:41.425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became pre-candidate at term 12"}
{"level":"info","ts":"2025-01-02T08:37:41.425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 12"}
{"level":"info","ts":"2025-01-02T08:37:41.425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became candidate at term 13"}
{"level":"info","ts":"2025-01-02T08:37:41.425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 13"}
{"level":"info","ts":"2025-01-02T08:37:41.425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8e9e05c52164694d became leader at term 13"}
{"level":"info","ts":"2025-01-02T08:37:41.425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 13"}
{"level":"warn","ts":"2025-01-02T08:37:41.426Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"275.065484ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"","error":"etcdserver: leader changed"}
{"level":"info","ts":"2025-01-02T08:37:41.426Z","caller":"traceutil/trace.go:171","msg":"trace[1300551157] range","detail":"{range_begin:health; range_end:; }","duration":"275.648589ms","start":"2025-01-02T08:37:41.150Z","end":"2025-01-02T08:37:41.426Z","steps":["trace[1300551157] 'agreement among raft nodes before linearized reading' (duration: 275.060346ms)"],"step_count":1}
[2025/01/02 08:37:41.426 +00:00] [WARN] [sessionutil/session_util.go:266] ["retry func failed"] [retried=0] [error="etcdserver: leader changed"]
{"level":"info","ts":"2025-01-02T08:37:41.427Z","caller":"etcdserver/v3_server.go:832","msg":"first commit in current term: resending ReadIndex request"}
{"level":"info","ts":"2025-01-02T08:37:41.428Z","caller":"etcdserver/server.go:2054","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://0.0.0.0:2379]}","request-path":"/0/members/8e9e05c52164694d/attributes","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
{"level":"info","ts":"2025-01-02T08:37:41.428Z","caller":"embed/serve.go:100","msg":"ready to serve client requests"}
{"level":"warn","ts":"2025-01-02T08:37:41.428Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"179.888136ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
{"level":"info","ts":"2025-01-02T08:37:41.428Z","caller":"traceutil/trace.go:171","msg":"trace[1191263840] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:1348436; }","duration":"179.951903ms","start":"2025-01-02T08:37:41.248Z","end":"2025-01-02T08:37:41.428Z","steps":["trace[1191263840] 'agreement among raft nodes before linearized reading' (duration: 179.827321ms)"],"step_count":1}
{"level":"warn","ts":"2025-01-02T08:37:41.428Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"181.305349ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"warn","ts":"2025-01-02T08:37:41.428Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"179.98094ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"info","ts":"2025-01-02T08:37:41.428Z","caller":"traceutil/trace.go:171","msg":"trace[286672040] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:1348436; }","duration":"181.391169ms","start":"2025-01-02T08:37:41.247Z","end":"2025-01-02T08:37:41.428Z","steps":["trace[286672040] 'agreement among raft nodes before linearized reading' (duration: 181.26225ms)"],"step_count":1}
{"level":"info","ts":"2025-01-02T08:37:41.428Z","caller":"traceutil/trace.go:171","msg":"trace[1390830741] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:1348436; }","duration":"180.034677ms","start":"2025-01-02T08:37:41.248Z","end":"2025-01-02T08:37:41.428Z","steps":["trace[1390830741] 'agreement among raft nodes before linearized reading' (duration: 179.870176ms)"],"step_count":1}
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
{"level":"warn","ts":"2025-01-02T08:37:41.428Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"179.933676ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"info","ts":"2025-01-02T08:37:41.428Z","caller":"traceutil/trace.go:171","msg":"trace[701353041] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:1348436; }","duration":"180.002346ms","start":"2025-01-02T08:37:41.248Z","end":"2025-01-02T08:37:41.428Z","steps":["trace[701353041] 'agreement among raft nodes before linearized reading' (duration: 179.862819ms)"],"step_count":1}
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
{"level":"warn","ts":"2025-01-02T08:37:41.428Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"181.336009ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:"health" ","response":"range_response_count:0 size:6"}
{"level":"info","ts":"2025-01-02T08:37:41.428Z","caller":"traceutil/trace.go:171","msg":"trace[1714627676] range","detail":"{range_begin:health; range_end:; response_count:0; response_revision:1348436; }","duration":"181.575231ms","start":"2025-01-02T08:37:41.247Z","end":"2025-01-02T08:37:41.428Z","steps":["trace[1714627676] 'agreement among raft nodes before linearized reading' (duration: 181.251663ms)"],"step_count":1}
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [rootcoord/service.go:233] ["RootCoord start to create QueryCoord client"]
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.428 +00:00] [DEBUG] [querycoord/service.go:168] ["QueryCoord try to wait for RootCoord ready"]
{"level":"warn","ts":"2025-01-02T08:37:41.428Z","caller":"etcdserver/v3_server.go:817","msg":"ignored out-of-date read index response; local node read indexes queueing up and waiting to be in sync with leader","sent-request-id":7587883838820091140,"received-request-id":7587883838820091139}
[2025/01/02 08:37:41.430 +00:00] [DEBUG] [sessionutil/session_util.go:315] [getServerID] [reuse=true]
{"level":"info","ts":"2025-01-02T08:37:41.431Z","caller":"embed/serve.go:146","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"[::]:2379"}
[2025/01/02 08:37:41.431 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.431 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.431 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.431 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.431 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2025/01/02 08:37:41.431 +00:00] [INFO] [msgstream/mq_factory.go:18] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
[2025/01/02 08:37:41.431 +00:00] [WARN] [grpcclient/client.go:557] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.431 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.431 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.431 +00:00] [WARN] [grpcclient/client.go:557] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.431 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.431 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.432 +00:00] [DEBUG] [sessionutil/session_util.go:373] ["Session get serverID success"] [key=id] [ServerId=12]
[2025/01/02 08:37:41.432 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexnode] [address=172.17.0.2:21121] [id=12]
[2025/01/02 08:37:41.432 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.432 +00:00] [DEBUG] [sessionutil/session_util.go:315] [getServerID] [reuse=true]
[2025/01/02 08:37:41.432 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.432 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.432 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.432 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=indexcoord] [address=172.17.0.2:13333] [id=12]
[2025/01/02 08:37:41.432 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.432 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.432 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.432 +00:00] [DEBUG] [sessionutil/session_util.go:315] [getServerID] [reuse=true]
[2025/01/02 08:37:41.432 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=querynode] [address=172.17.0.2:21123] [id=12]
[2025/01/02 08:37:41.432 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.433 +00:00] [DEBUG] [sessionutil/session_util.go:315] [getServerID] [reuse=true]
[2025/01/02 08:37:41.433 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=rootcoord] [address=172.17.0.2:53100] [id=12]
[2025/01/02 08:37:41.433 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.433 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Initializing]
[2025/01/02 08:37:41.432 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.432 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.434 +00:00] [DEBUG] [sessionutil/session_util.go:315] [getServerID] [reuse=true]
[2025/01/02 08:37:41.434 +00:00] [INFO] [sessionutil/session_util.go:289] ["start server"] [name=datacoord] [address=172.17.0.2:13333] [id=12]
[2025/01/02 08:37:41.434 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.435 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2025/01/02 08:31:06.868 +00:00] [save=2025/01/02 08:37:44.434 +00:00] [next=2025/01/02 08:37:41.434 +00:00]
[2025/01/02 08:37:41.435 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2025/01/02 08:37:41.435 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.435 +00:00] [INFO] [datacoord/server.go:343] ["init rootcoord client done"]
[2025/01/02 08:37:41.435 +00:00] [INFO] [datacoord/server.go:352] ["init chunk manager factory done"]
[2025/01/02 08:37:41.435 +00:00] [INFO] [datacoord/server.go:655] ["data coordinator connecting to metadata store"] [metaType=etcd]
[2025/01/02 08:37:41.435 +00:00] [INFO] [datacoord/server.go:668] ["data coordinator successfully connected to metadata store"] [metaType=etcd]
[2025/01/02 08:37:41.435 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="indexnode-12\n"] [filePath=/tmp/milvus/server_id_8]
[2025/01/02 08:37:41.435 +00:00] [INFO] [indexnode/indexnode.go:214] ["IndexNode init session successful"] [serverID=12]
[2025/01/02 08:37:41.435 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="querynode-12\n"] [filePath=/tmp/milvus/server_id_8]
[2025/01/02 08:37:41.435 +00:00] [INFO] [querynodev2/server.go:160] ["QueryNode init session"] [nodeID=12] ["node address"=172.17.0.2:21123]
[2025/01/02 08:37:41.435 +00:00] [INFO] [dependency/factory.go:86] ["try to init mq"] [standalone=true] [mqType=rocksmq]
[2025/01/02 08:37:41.435 +00:00] [INFO] [msgstream/mq_factory.go:18] ["init rocksmq msgstream success"] [path=/var/lib/milvus/rdb_data]
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250102 08:37:41.435773 47 knowhere_config.cc:107] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384
I20250102 08:37:41.435982 47 knowhere_config.cc:118] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0
I20250102 08:37:41.436012 47 knowhere_config.cc:54] [KNOWHERE][ShowVersion][milvus] Knowhere Version: v2.3.5-hotfix
[2025/01/02 08:37:41.436 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2025/01/02 08:36:51.938 +00:00] [save=2025/01/02 08:37:44.435 +00:00] [next=2025/01/02 08:37:41.435 +00:00]
[2025/01/02 08:37:41.436 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2025/01/02 08:37:41.436 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."]
I20250102 08:37:41.436264 47 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
[2025/01/02 08:37:41.436 +00:00] [DEBUG] [rootcoord/suffix_snapshot.go:558] ["suffix snapshot GC goroutine start!"]
I20250102 08:37:41.436300 47 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX512
[2025/01/02 08:37:41.436 +00:00] [INFO] [rootcoord/meta_table.go:150] ["recover databases"] ["num of dbs"=1]
I20250102 08:37:41.436832 47 Common.cpp:35] [SERVER][SetIndexSliceSize][milvus] set config index slice size (byte): 16777216
I20250102 08:37:41.436954 47 Common.cpp:41] [SERVER][SetHighPriorityThreadCoreCoefficient][milvus] set high priority thread pool core coefficient: 10
I20250102 08:37:41.436978 47 Common.cpp:48] [SERVER][SetMiddlePriorityThreadCoreCoefficient][milvus] set middle priority thread pool core coefficient: 5
I20250102 08:37:41.437002 47 Common.cpp:55] [SERVER][SetLowPriorityThreadCoreCoefficient][milvus] set low priority thread pool core coefficient: 1
[2025/01/02 08:37:41.437 +00:00] [INFO] [datacoord/index_meta.go:95] ["indexMeta reloadFromKV done"] [duration=1.717924ms]
I20250102 08:37:41.437161 47 thread_pool.h:152] [KNOWHERE][InitGlobalBuildThreadPool][milvus] Init global build thread pool with size 24
[2025/01/02 08:37:41.437 +00:00] [INFO] [indexnode/indexnode.go:219] ["init index node done"] [nodeID=12] [Address=172.17.0.2:21121]
[2025/01/02 08:37:41.437 +00:00] [DEBUG] [indexnode/service.go:74] ["IndexNode init done ..."]
[2025/01/02 08:37:41.437 +00:00] [INFO] [indexnode/indexnode.go:230] [IndexNode] [State=Healthy]
[2025/01/02 08:37:41.437 +00:00] [INFO] [indexnode/indexnode.go:233] ["IndexNode start finished"] []
[2025/01/02 08:37:41.437 +00:00] [DEBUG] [sessionutil/session_util.go:436] ["service begin to register to etcd"] [serverName=indexnode] [ServerID=12]
[2025/01/02 08:37:41.437 +00:00] [DEBUG] [indexnode/task_scheduler.go:247] ["IndexNode TaskScheduler start build loop ..."]
[2025/01/02 08:37:41.439 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=by-dev/meta/session/indexnode-12] [value="{"ServerID":12,"ServerName":"indexnode","Address":"172.17.0.2:21121","TriggerKill":true,"Version":"2.4.6","IndexEngineVersion":{},"LeaseID":7587883838820091169,"HostName":"b3954f411bd7","EnableDisk":true}"]
[2025/01/02 08:37:41.439 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=indexnode] [serverID=12]
[2025/01/02 08:37:41.439 +00:00] [DEBUG] [indexnode/service.go:209] ["IndexNode Register etcd success"]
[2025/01/02 08:37:41.439 +00:00] [DEBUG] [indexnode/service.go:78] ["IndexNode start done ..."]
[2025/01/02 08:37:41.439 +00:00] [DEBUG] [components/index_node.go:57] ["IndexNode successfully started"]
[2025/01/02 08:37:41.439 +00:00] [INFO] [datacoord/meta.go:163] ["DataCoord meta reloadFromKV done"] [duration=2.349204ms]
[2025/01/02 08:37:41.439 +00:00] [DEBUG] [sessionutil/session_util.go:646] ["Session version out of range"] [version=2.4.6] [serverID=11]
[2025/01/02 08:37:41.439 +00:00] [INFO] [datacoord/server.go:1238] ["all old data node down, enable auto balance!"]
[2025/01/02 08:37:41.440 +00:00] [INFO] [rootcoord/meta_table.go:190] ["collections recovered from db"] [db_name=default] [collection_num=4] [partition_num=8]
[2025/01/02 08:37:41.440 +00:00] [INFO] [rootcoord/meta_table.go:208] ["RootCoord meta table reload done"] [duration=3.963807ms]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_10_454254081415433293v0]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_11_454254081415433293v1]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_4_454254081415433258v0]
[2025/01/02 08:37:41.440 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_5_454254081415433258v1]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_6_454254081416011210v0]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_7_454254081416011210v1]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_8_454254081416011250v0]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:67] ["channel store reload channel"] [nodeID=11] [channel=by-dev-rootcoord-dml_9_454254081416011250v1]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/channel_store_v2.go:71] ["channel store reload done"] [duration=581.442µs]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/server.go:366] ["init datanode cluster done"]
[2025/01/02 08:37:41.440 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_0]
[2025/01/02 08:37:41.440 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.440 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_1]
[2025/01/02 08:37:41.440 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.440 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_2]
[2025/01/02 08:37:41.440 +00:00] [DEBUG] [sessionutil/session_util.go:650] ["SessionUtil GetSessions "] [prefix=datanode] [key=datanode-11] [address=172.17.0.2:21124]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/server.go:572] ["DataCoord success to get DataNode sessions"] [sessions="{"datanode-11":{"ServerID":11,"ServerName":"datanode","Address":"172.17.0.2:21124","TriggerKill":true,"Version":"2.4.6","IndexEngineVersion":{},"LeaseID":7587883838718159224,"HostName":"709ad33df4e4"}}"]
[2025/01/02 08:37:41.440 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.440 +00:00] [INFO] [datacoord/server.go:593] ["DataCoord Cluster Manager start up"]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_3]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_4]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_5]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_6]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_7]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_8]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_9]
[2025/01/02 08:37:41.441 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_10]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_11]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_12]
[2025/01/02 08:37:41.441 +00:00] [WARN] [grpcclient/client.go:557] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_13]
[2025/01/02 08:37:41.441 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_14]
[2025/01/02 08:37:41.441 +00:00] [INFO] [msgstream/common_mq_factory.go:31] ["Msg Stream state"] [can_produce=true]
[2025/01/02 08:37:41.441 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_15]
[2025/01/02 08:37:41.441 +00:00] [INFO] [datacoord/channel_manager_v2.go:162] ["starting channel balance loop"]
[2025/01/02 08:37:41.441 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.441 +00:00] [INFO] [datacoord/channel_manager_v2.go:170] ["cluster start up"] [allNodes="[11]"] [legacyNodes="[]"] [oldNodes="[11]"] [newOnlines="[]"] [offLines="[]"]
[2025/01/02 08:37:41.441 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.441 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [rootcoord/dml_channels.go:215] ["init dml channels"] [prefix=by-dev-rootcoord-dml] [num=16]
[2025/01/02 08:37:41.441 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [rootcoord/timeticksync.go:126] ["recover physical channels"] [collectionID=454254081416011210] ["physical channels"="[by-dev-rootcoord-dml_6,by-dev-rootcoord-dml_7]"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [datacoord/server.go:598] ["DataCoord Cluster Manager start up successfully"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [rootcoord/timeticksync.go:126] ["recover physical channels"] [collectionID=454254081416011250] ["physical channels"="[by-dev-rootcoord-dml_8,by-dev-rootcoord-dml_9]"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [rootcoord/timeticksync.go:126] ["recover physical channels"] [collectionID=454254081415433258] ["physical channels"="[by-dev-rootcoord-dml_4,by-dev-rootcoord-dml_5]"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [rootcoord/timeticksync.go:126] ["recover physical channels"] [collectionID=454254081415433293] ["physical channels"="[by-dev-rootcoord-dml_10,by-dev-rootcoord-dml_11]"]
[2025/01/02 08:37:41.441 +00:00] [INFO] [rootcoord/root_coord.go:451] ["create TimeTick sync done"]
[2025/01/02 08:37:41.441 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-11] [address=172.17.0.2:21121]
[2025/01/02 08:37:41.441 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=indexnode] [key=indexnode-12] [address=172.17.0.2:21121]
[2025/01/02 08:37:41.441 +00:00] [DEBUG] [datacoord/indexnode_manager.go:84] ["add IndexNode"] [nodeID=11] ["node address"=172.17.0.2:21121]
[2025/01/02 08:37:41.441 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.442 +00:00] [INFO] [rootcoord/root_coord.go:467] ["init proxy manager done"]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [datacoord/indexnode_manager.go:57] ["set IndexNode client"] [nodeID=11]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [datacoord/indexnode_manager.go:62] ["IndexNode IndexNodeManager setClient success"] [nodeID=11] ["IndexNode num"=1]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [datacoord/indexnode_manager.go:84] ["add IndexNode"] [nodeID=12] ["node address"=172.17.0.2:21121]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [rootcoord/root_coord.go:472] ["RootCoord init QuotaCenter done"]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [datacoord/indexnode_manager.go:57] ["set IndexNode client"] [nodeID=12]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [datacoord/indexnode_manager.go:62] ["IndexNode IndexNodeManager setClient success"] [nodeID=12] ["IndexNode num"=2]
[2025/01/02 08:37:41.442 +00:00] [INFO] [rootcoord/root_coord.go:477] ["init credentials done"]
[2025/01/02 08:37:41.442 +00:00] [INFO] [rootcoord/meta_table.go:1273] ["role already exists"] [role=admin]
[2025/01/02 08:37:41.442 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=querynode] [key=querynode-11] [address=172.17.0.2:21123]
[2025/01/02 08:37:41.442 +00:00] [INFO] [datacoord/index_engine_version_manager.go:65] ["addOrUpdate version"] [nodeId=11] [minimal=0] [current=4]
[2025/01/02 08:37:41.442 +00:00] [INFO] [rootcoord/meta_table.go:1273] ["role already exists"] [role=public]
[2025/01/02 08:37:41.442 +00:00] [INFO] [datacoord/server.go:373] ["init service discovery done"]
[2025/01/02 08:37:41.442 +00:00] [INFO] [datacoord/server.go:378] ["init compaction scheduler done"]
[2025/01/02 08:37:41.443 +00:00] [INFO] [rootcoord/root_coord.go:483] ["init rootcoord done"] [nodeID=12] [Address=172.17.0.2:53100]
[2025/01/02 08:37:41.443 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.443 +00:00] [DEBUG] [rootcoord/service.go:169] ["RootCoord init done ..."]
[2025/01/02 08:37:41.443 +00:00] [INFO] [rootcoord/service.go:312] ["RootCoord Core start ..."]
[2025/01/02 08:37:41.443 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.443 +00:00] [DEBUG] [sessionutil/session_util.go:436] ["service begin to register to etcd"] [serverName=rootcoord] [ServerID=12]
[2025/01/02 08:37:41.443 +00:00] [WARN] [grpcclient/client.go:557] ["fail to get grpc client"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.443 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.444 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2025/01/02 08:37:41.444 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.444 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.444 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.445 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=by-dev/meta/session/rootcoord] [value="{"ServerID":12,"ServerName":"rootcoord","Address":"172.17.0.2:53100","Exclusive":true,"TriggerKill":true,"Version":"2.4.6","IndexEngineVersion":{},"LeaseID":7587883838820091210,"HostName":"b3954f411bd7"}"]
[2025/01/02 08:37:41.445 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=rootcoord] [serverID=12]
[2025/01/02 08:37:41.445 +00:00] [INFO] [rootcoord/root_coord.go:273] ["RootCoord Register Finished"]
[2025/01/02 08:37:41.445 +00:00] [INFO] [proxyutil/proxy_watcher.go:96] ["succeed to init sessions on etcd"] [sessions="[{"ServerID":11,"ServerName":"proxy","Address":"172.17.0.2:19529","TriggerKill":true,"Version":"2.4.6","IndexEngineVersion":{},"LeaseID":7587883838718159293,"HostName":"709ad33df4e4"}]"] [revision=1348441]
[2025/01/02 08:37:41.445 +00:00] [INFO] [rootcoord/timeticksync.go:254] ["Init proxy sessions for timeticksync"] [serverID=11]
[2025/01/02 08:37:41.445 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.445 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.445 +00:00] [INFO] [proxyutil/proxy_client_manager.go:156] ["succeed to create proxy client"] [address=172.17.0.2:19529] [serverID=11]
[2025/01/02 08:37:41.446 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Healthy]
[2025/01/02 08:37:41.446 +00:00] [INFO] [sessionutil/session_util.go:1234] ["save server info into file"] [content="rootcoord-12\n"] [filePath=/tmp/milvus/server_id_8]
[2025/01/02 08:37:41.446 +00:00] [INFO] [rootcoord/quota_center.go:281] ["Start QuotaCenter"] [collectInterval=3s]
[2025/01/02 08:37:41.446 +00:00] [INFO] [proxyutil/proxy_watcher.go:119] ["start to watch etcd"]
[2025/01/02 08:37:41.446 +00:00] [INFO] [rootcoord/root_coord.go:730] ["rootcoord startup successfully"]
[2025/01/02 08:37:41.446 +00:00] [DEBUG] [rootcoord/service.go:174] ["RootCoord start done ..."]
[2025/01/02 08:37:41.446 +00:00] [INFO] [components/root_coord.go:58] ["RootCoord successfully started"]
[2025/01/02 08:37:41.448 +00:00] [WARN] [grpcclient/client.go:490] ["call received grpc error"] [clientRole=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 08:37:41.448 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=proxy-11] [key=proxy-11] [address=172.17.0.2:19529]
[2025/01/02 08:37:41.448 +00:00] [WARN] [retry/retry.go:104] ["start to reset connection because of specific reasons"] [client_role=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 08:37:41.448 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=proxy-11] [addr=]
[2025/01/02 08:37:41.449 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 08:37:41.474 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:27] ["query node use concurrent safe scheduler"] [max_concurrency=32]
[2025/01/02 08:37:41.475 +00:00] [INFO] [querynodev2/server.go:326] ["queryNode init scheduler"] [policy=fifo]
[2025/01/02 08:37:41.475 +00:00] [INFO] [segments/segment_loader.go:548] ["SegmentLoader created"] [ioPoolSize=256]
I20250102 08:37:41.475402 67 thread_pool.h:172] [KNOWHERE][InitGlobalSearchThreadPool][milvus] Init global search thread pool with size 32
I20250102 08:37:41.475520 67 knowhere_config.cc:64] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO
I20250102 08:37:41.475536 67 knowhere_config.cc:89] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX512
[2025/01/02 08:37:41.475 +00:00] [INFO] [querynodev2/server.go:227] ["set up knowhere build pool size"] [pool_size=16]
I20250102 08:37:41.475963 131 thread_pool.h:53] [KNOWHERE][operator()][knowhere_build0] Successfully set priority of knowhere thread.
I20250102 08:37:41.478183 67 thread_pool.h:188] [KNOWHERE][SetGlobalBuildThreadPoolSize][milvus] Global build thread pool size has already been set to 16
I20250102 08:37:41.478258 67 Common.cpp:62] [SERVER][SetDefaultExecEvalExprBatchSize][milvus] set default expr eval batch size: 8192
I20250102 08:37:41.478494 67 ChunkCache.h:38] [SERVER][ChunkCache][milvus] Init ChunkCache with prefix: /var/lib/milvus/data/mmap/chunk_cache, read_ahead_policy: willneed
[2025/01/02 08:37:41.478 +00:00] [INFO] [querynodev2/server.go:260] ["InitChunkCache done"] [dir=/var/lib/milvus/data/mmap/chunk_cache] [policy=willneed]
I20250102 08:37:41.478586 67 Tracer.cpp:66] [SERVER][initTelemetry][milvus] Empty Trace
[2025/01/02 08:37:41.482 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=60646920192]
[2025/01/02 08:37:41.483 +00:00] [INFO] [querynodev2/server.go:385] ["query node init successfully"] [queryNodeID=12] [Address=172.17.0.2:21123]
[2025/01/02 08:37:41.483 +00:00] [DEBUG] [querynode/service.go:234] ["QueryNode init done ..."]
[2025/01/02 08:37:41.483 +00:00] [INFO] [querynodev2/server.go:405] ["query node start successfully"] [queryNodeID=12] [Address=172.17.0.2:21123] [mmapEnabled=false]
[2025/01/02 08:37:41.483 +00:00] [INFO] [tasks/concurrent_safe_scheduler.go:213] ["start execute loop"]
[2025/01/02 08:37:41.483 +00:00] [DEBUG] [sessionutil/session_util.go:436] ["service begin to register to etcd"] [serverName=querynode] [ServerID=12]
[2025/01/02 08:37:41.489 +00:00] [INFO] [sessionutil/session_util.go:466] ["put session key into etcd"] [key=by-dev/meta/session/querynode-12] [value="{"ServerID":12,"ServerName":"querynode","Address":"172.17.0.2:21123","TriggerKill":true,"Version":"2.4.6","IndexEngineVersion":{"CurrentIndexVersion":4},"LeaseID":7587883838820091218,"HostName":"b3954f411bd7"}"]
[2025/01/02 08:37:41.489 +00:00] [INFO] [sessionutil/session_util.go:476] ["Service registered successfully"] [ServerName=querynode] [serverID=12]
[2025/01/02 08:37:41.489 +00:00] [DEBUG] [sessionutil/session_util.go:779] ["watch services"] ["add kv"="key:"by-dev/meta/session/querynode-12" create_revision:1348442 mod_revision:1348442 version:1 value:"{\"ServerID\":12,\"ServerName\":\"querynode\",\"Address\":\"172.17.0.2:21123\",\"TriggerKill\":true,\"Version\":\"2.4.6\",\"IndexEngineVersion\":{\"CurrentIndexVersion\":4},\"LeaseID\":7587883838820091218,\"HostName\":\"b3954f411bd7\"}" lease:7587883838820091218 "]
[2025/01/02 08:37:41.489 +00:00] [DEBUG] [sessionutil/session_util.go:807] [WatchService] ["event type"=SessionAddEvent]
[2025/01/02 08:37:41.489 +00:00] [DEBUG] [querynode/service.go:239] ["QueryNode start done ..."]
[2025/01/02 08:37:41.489 +00:00] [DEBUG] [components/query_node.go:59] ["QueryNode successfully started"]
[2025/01/02 08:37:41.538 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=rootcoord] [key=by-dev/meta/session/rootcoord]
[2025/01/02 08:37:41.538 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=indexnode] [key=by-dev/meta/session/indexnode-12]
[2025/01/02 08:37:41.538 +00:00] [INFO] [sessionutil/session_util.go:914] ["register session success"] [role=querynode] [key=by-dev/meta/session/querynode-12]
[2025/01/02 08:37:41.627 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.627 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.627 +00:00] [DEBUG] [proxy/service.go:599] ["create RootCoord client for Proxy done"]
[2025/01/02 08:37:41.627 +00:00] [DEBUG] [proxy/service.go:602] ["Proxy wait for RootCoord to be healthy"]
[2025/01/02 08:37:41.627 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.17.0.2:53100]
[2025/01/02 08:37:41.627 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=172.17.0.2:53100] [serverID=12]
[2025/01/02 08:37:41.628 +00:00] [DEBUG] [rootcoord/root_coord.go:787] ["RootCoord current state"] [StateCode=Healthy]
[2025/01/02 08:37:41.629 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [proxy/service.go:607] ["Proxy wait for RootCoord to be healthy done"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [proxy/service.go:609] ["set RootCoord client for Proxy"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [proxy/service.go:611] ["set RootCoord client for Proxy done"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [proxy/service.go:615] ["create DataCoord client for Proxy"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [proxy/service.go:621] ["create DataCoord client for Proxy done"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [proxy/service.go:624] ["Proxy wait for DataCoord to be healthy"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2025/01/02 08:37:41.629 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.629 +00:00] [WARN] [grpcclient/client.go:557] ["fail to get grpc client"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.629 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.629 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2025/01/02 08:37:41.629 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.629 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.629 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.634 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.634 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.634 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.17.0.2:53100]
[2025/01/02 08:37:41.634 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=172.17.0.2:53100] [serverID=12]
[2025/01/02 08:37:41.634 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.17.0.2:53100]
[2025/01/02 08:37:41.634 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=172.17.0.2:53100] [serverID=12]
[2025/01/02 08:37:41.642 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.642 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=rootcoord] [key=rootcoord] [address=172.17.0.2:53100]
[2025/01/02 08:37:41.642 +00:00] [DEBUG] [client/client.go:93] ["RootCoordClient GetSessions success"] [address=172.17.0.2:53100] [serverID=12]
[2025/01/02 08:37:41.645 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"]
[2025/01/02 08:37:41.646 +00:00] [WARN] [rootcoord/root_coord.go:195] ["zero ts was met, this should be only occurred in starting state"] [minBgDdlTs=455031406368129025] [minNormalDdlTs=0]
[2025/01/02 08:37:41.650 +00:00] [WARN] [grpcclient/client.go:490] ["call received grpc error"] [clientRole=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 08:37:41.650 +00:00] [DEBUG] [sessionutil/session_util.go:620] ["SessionUtil GetSessions"] [prefix=proxy-11] [key=proxy-11] [address=172.17.0.2:19529]
[2025/01/02 08:37:41.650 +00:00] [WARN] [retry/retry.go:104] ["start to reset connection because of specific reasons"] [client_role=proxy-11] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=11][actualNodeID=12]"]
[2025/01/02 08:37:41.651 +00:00] [INFO] [grpcclient/client.go:239] ["previous client closed"] [role=proxy-11] [addr=]
[2025/01/02 08:37:41.830 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.830 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2025/01/02 08:37:41.830 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:41.830 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.036 +00:00] [DEBUG] [rootcoord/root_coord.go:787] ["RootCoord current state"] [StateCode=Healthy]
[2025/01/02 08:37:42.036 +00:00] [DEBUG] [rootcoord/root_coord.go:787] ["RootCoord current state"] [StateCode=Healthy]
[2025/01/02 08:37:42.036 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2025/01/02 08:37:42.036 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2025/01/02 08:37:42.036 +00:00] [INFO] [datanode/service.go:277] ["RootCoord client is ready for DataNode"]
[2025/01/02 08:37:42.036 +00:00] [DEBUG] [datanode/service.go:285] ["starting DataCoord client for DataNode"]
[2025/01/02 08:37:42.036 +00:00] [DEBUG] [querycoord/service.go:178] ["QueryCoord report RootCoord ready"]
[2025/01/02 08:37:42.036 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:42.036 +00:00] [DEBUG] [sessionutil/session_util.go:257] ["Session try to connect to etcd"]
[2025/01/02 08:37:42.037 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:42.037 +00:00] [DEBUG] [sessionutil/session_util.go:272] ["Session connect to etcd success"]
[2025/01/02 08:37:42.037 +00:00] [DEBUG] [querycoord/service.go:189] ["QueryCoord try to wait for DataCoord ready"]
[2025/01/02 08:37:42.037 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:557] ["fail to get grpc client"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:557] ["fail to get grpc client"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [retry/retry.go:104] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [DEBUG] [client/client.go:90] ["DataCoordClient, not existed in msess "] [key=datacoord] ["len of msess"=0]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:213] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:479] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.037 +00:00] [WARN] [grpcclient/client.go:470] ["retry func failed"] [retried=0] [error="empty grpc client: find no available datacoord, check datacoord state"]
[2025/01/02 08:37:42.047 +00:00] [INFO] [rootcoord/service.go:93] ["received request to list databases"] [msgID=0]
[2025/01/02 08:37:42.048 +00:00] [INFO] [rootcoord/service.go:93] ["done to list databases"] [msgID=0] ["num of databases"=1]
[2025/01/02 08:37:42.048 +00:00] [WARN] [datacoord/segment_manager.go:245] ["failed to get ts from rootCoord for globalLastExpire"] []
[2025/01/02 08:37:42.048 +00:00] [INFO] [datacoord/server.go:384] ["init segment manager done"]

yhmo · 2025-01-02T09:23:36Z

yhmo
Jan 2, 2025
Collaborator

多半是因为存在数据量很大的表，实例在重启的时候重新加载表，或者有大量数据还在message queue里还没被data node persist，querynode和datanode都要重message queue里拉取大量数据，在这过程中是不能查询服务的。

8点37分的这段日志显示大部分节点已经启动成功，除了datacoord可能还在等datanode。只要最终日志显示"Proxy successfully started"即为可服务状态。

2 replies

jhyever Jan 2, 2025
Author

这个是单机的呀，不是集群呢

yhmo Jan 2, 2025
Collaborator

就算是单机，一个进程，内部也是区分各种节点，运行机制和集群并无太多区别

xiaofan-luan · 2025-01-02T10:20:06Z

xiaofan-luan
Jan 2, 2025
Maintainer

我们会针对多表的场景做一个优化，预计能够支持几千个表和十万级别的partition。
在此之前，表数目超过1000会有严重的性能，目前恢复速度可能会比较长

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

milvus2.4.6，单机standalone_embed.sh启动报这个错,容器 unhealthy #38934

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

milvus2.4.6，单机standalone_embed.sh启动报这个错,容器 unhealthy #38934

jhyever Jan 2, 2025

Replies: 3 comments · 3 replies

yhmo Jan 2, 2025 Collaborator

jhyever Jan 2, 2025 Author

yhmo Jan 2, 2025 Collaborator

jhyever Jan 2, 2025 Author

yhmo Jan 2, 2025 Collaborator

xiaofan-luan Jan 2, 2025 Maintainer

jhyever
Jan 2, 2025

Replies: 3 comments 3 replies

yhmo
Jan 2, 2025
Collaborator

jhyever Jan 2, 2025
Author

yhmo
Jan 2, 2025
Collaborator

jhyever Jan 2, 2025
Author

yhmo Jan 2, 2025
Collaborator

xiaofan-luan
Jan 2, 2025
Maintainer