Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Leader task stuck and retry again and again #38201

Open
1 task done
weiliu1031 opened this issue Dec 4, 2024 · 2 comments
Open
1 task done

[Bug]: Leader task stuck and retry again and again #38201

weiliu1031 opened this issue Dec 4, 2024 · 2 comments
Assignees
@weiliu1031
Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@weiliu1031
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.4 and master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

leader task require to update delegator's distribution, and only success after the distribution change has been applyed to delegator. but the delegator will reject the distribution change if it's version is older than current version in delegator. which cause the leader task stuck and retry forever.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@weiliu1031 weiliu1031 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 4, 2024
@weiliu1031
Copy link
Contributor Author

/assign

@weiliu1031 weiliu1031 mentioned this issue Dec 4, 2024
Merged
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 4, 2024
@yanliang567 yanliang567 removed their assignment Dec 4, 2024
sre-ci-robot pushed a commit that referenced this issue Dec 10, 2024
@weiliu1031
fix: Leader task stuck and retry again and again (#38202)
856e2aa 
issue: #38201
leader task require to update delegator's distribution, and only success
after the distribution change has been applyed to delegator. but the
delegator will reject the distribution change if it's version is older
than current version in delegator. which cause the leader task stuck and
retry forever.

this PR remove the leader task finish check.

Signed-off-by: Wei Liu <[email protected]>
weiliu1031 added a commit to weiliu1031/milvus that referenced this issue Dec 10, 2024
@weiliu1031
fix: Leader task stuck and retry again and again (milvus-io#38202)
2257cbe 
issue: milvus-io#38201
leader task require to update delegator's distribution, and only success
after the distribution change has been applyed to delegator. but the
delegator will reject the distribution change if it's version is older
than current version in delegator. which cause the leader task stuck and
retry forever.

this PR remove the leader task finish check.

Signed-off-by: Wei Liu <[email protected]>
@weiliu1031 weiliu1031 mentioned this issue Dec 10, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Dec 11, 2024
@weiliu1031
fix: Leader task stuck and retry again and again (#38202) (#38349)
f4696a1 
issue: #38201
pr: #38202
leader task require to update delegator's distribution, and only success
after the distribution change has been applyed to delegator. but the
delegator will reject the distribution change if it's version is older
than current version in delegator. which cause the leader task stuck and
retry forever.

this PR remove the leader task finish check.

Signed-off-by: Wei Liu <[email protected]>
@stale Stale
Copy link

stale bot commented Jan 4, 2025

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Jan 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@weiliu1031 weiliu1031

Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yanliang567 @weiliu1031