You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a one-way sync from the control plane to the ingesters, but the opposite path does not exist. In the case of Airmal, they resized their disks, which scratched the WALs. When the ingesters were restarted, they had no way to tell the CP that the shards they hosted no longer existed. The decommissioning logic should help prevent this scenario in the case of a graceful shutdown, but if, for some reason, it is not triggered, we can only fix the cluster state manually (clean up shards in metastore and restart CP).
The text was updated successfully, but these errors were encountered:
We have a one-way sync from the control plane to the ingesters, but the opposite path does not exist. In the case of Airmal, they resized their disks, which scratched the WALs. When the ingesters were restarted, they had no way to tell the CP that the shards they hosted no longer existed. The decommissioning logic should help prevent this scenario in the case of a graceful shutdown, but if, for some reason, it is not triggered, we can only fix the cluster state manually (clean up shards in metastore and restart CP).
The text was updated successfully, but these errors were encountered: