Optimize counter polling interval by making it more accurate #1457
+666
−52
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR requires swss to be updated correspondingly. The swss PR will be opened soon. |
|
Depends on sonic-net/sonic-swss-common#950 |
stephenxs
force-pushed
the
counter-optimization-all-in-one
branch
from
c14fd22 to
6b362f6
Compare
stephenxs
force-pushed
the
counter-optimization-all-in-one
branch
from
6b362f6 to
b82e233
Compare
stephenxs
force-pushed
the
counter-optimization-all-in-one
branch
from
b82e233 to
5442b37
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
stephenxs
force-pushed
the
counter-optimization-all-in-one
branch
from
5442b37 to
6c1086a
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
stephenxs
force-pushed
the
counter-optimization-all-in-one
branch
from
9624365 to
677bfc2
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azpw run |
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Hi @kcudnik |
kcudnik
approved these changes
Feb 7, 2025
|
Hi @kcudnik |
|
hi @kperumalbfn , do you mind to help get this PR merged into 202411? |
mssonicbld
added a commit
to mssonicbld/sonic-sairedis
that referenced
this pull request
Feb 10, 2025
Define bulk chunk size and bulk chunk size per counter ID. This is to resolve the VS test failure in sonic-net#1457, which is caused by loop dependency. In PR sonic-net#1457, new fields `bulk_chunk_size` and `bulk_chunk_size_per_prefix` have been introduced to `sai_redis_flex_counter_group_parameter_t` whose instances are initialized by orchagent. However, the orchagent is still compiled with the old sairedis header, which prevents both new fields from being uninitialized which in turn fails vs test. We have to split this PR into two: 1. sonic-net#1519 which updates the header sairedis.h only. the motivation is to compile swss(orchagent) with both new fields initiated. 2. sonic-net#1457 contains all the rest of code The order to merge: 1. sonic-net#1519 2. sonic-net/sonic-swss#3391 3. sonic-net#1457
|
@stephenxs could you pls check the conflict in 202411? |
mssonicbld
added a commit
that referenced
this pull request
Feb 10, 2025
Define bulk chunk size and bulk chunk size per counter ID. This is to resolve the VS test failure in #1457, which is caused by loop dependency. In PR #1457, new fields `bulk_chunk_size` and `bulk_chunk_size_per_prefix` have been introduced to `sai_redis_flex_counter_group_parameter_t` whose instances are initialized by orchagent. However, the orchagent is still compiled with the old sairedis header, which prevents both new fields from being uninitialized which in turn fails vs test. We have to split this PR into two: 1. #1519 which updates the header sairedis.h only. the motivation is to compile swss(orchagent) with both new fields initiated. 2. #1457 contains all the rest of code The order to merge: 1. #1519 2. sonic-net/sonic-swss#3391 3. #1457
|
Hi @kperumalbfn
However, there is also a conflict in cherry-picking 1. #1234 to 202411. |
|
hi @kperumalbfn , do you mind to help with the previous PRs? |
|
@stephenxs We are not planning to merge WRED stats to 202411 branch as it is a full feature and requires multiple PRs to be merged. Could you double commit this PR to 202411 branch? #1484 is cherrypicked to 202411 branch. |
Thank you for letting me know it. |
@kperumalbfn what about 202012-msft? thanks |
|
@stephenxs 202412-msft branch will have all the WRED stats related PRs. So we no need additional PR. |
|
@r12f Could you merge the WRED PRs to 202412 branch. We have few PRs in swss, sairedis and have tagged them. |
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Feb 19, 2025
…et#1457) What I did Optimize the counter-polling performance in terms of polling interval accuracy Enable bulk counter-polling to run at a smaller chunk size There is one counter-polling thread for each counter group. All such threads can compete for the critical sections at the vendor SAI level, which means a counter-polling thread can wait for a critical section if another thread has been in it, which introduces latency for the waiting counter group. An example is the competition between the PFC watchdog and the port counter groups. The port counter group contains many counters and is polled in a bulk mode which takes a relatively longer time. The PFC watchdog counter group contains only a few counters but is polled at a short interval. Sometimes, PFC watchdog counters need to wait before polling, which makes the polling interval inaccurate and prevents the PFC storm from being detected in time. To resolve this issue, we can reduce the chunk size of the port counter group. The port counter group polls the counters of all ports in a single bulk operation by default. By using a smaller chunk size, it polls the counters in several bulk operations with each polling counter of a subset (whose size <= chunk size) of all ports. By doing so, the port counter group stays in the critical section for a shorter time and the PFC watchdog is more likely to be scheduled to poll counters and detect the PFC storm in time. Collect the time stamp immediately after vendor SAI API returns. Currently, many counter groups require a Lua plugin to execute based on polling interval, to calculate rates, detect certain events, etc. Eg. For PFC watchdog counter group to PFC storm. In this case, the polling interval is calculated based on the difference of time stamps between the current and last poll to avoid deviation due to scheduling latency. However, the timestamp is collected in the Lua plugin which is several steps after the SAI API returns and is executed in a different context (redis-server). Both introduce even larger deviations. To overcome this, we collect the timestamp immediately after the SAI API returns.
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Feb 19, 2025
…et#1457) What I did Optimize the counter-polling performance in terms of polling interval accuracy Enable bulk counter-polling to run at a smaller chunk size There is one counter-polling thread for each counter group. All such threads can compete for the critical sections at the vendor SAI level, which means a counter-polling thread can wait for a critical section if another thread has been in it, which introduces latency for the waiting counter group. An example is the competition between the PFC watchdog and the port counter groups. The port counter group contains many counters and is polled in a bulk mode which takes a relatively longer time. The PFC watchdog counter group contains only a few counters but is polled at a short interval. Sometimes, PFC watchdog counters need to wait before polling, which makes the polling interval inaccurate and prevents the PFC storm from being detected in time. To resolve this issue, we can reduce the chunk size of the port counter group. The port counter group polls the counters of all ports in a single bulk operation by default. By using a smaller chunk size, it polls the counters in several bulk operations with each polling counter of a subset (whose size <= chunk size) of all ports. By doing so, the port counter group stays in the critical section for a shorter time and the PFC watchdog is more likely to be scheduled to poll counters and detect the PFC storm in time. Collect the time stamp immediately after vendor SAI API returns. Currently, many counter groups require a Lua plugin to execute based on polling interval, to calculate rates, detect certain events, etc. Eg. For PFC watchdog counter group to PFC storm. In this case, the polling interval is calculated based on the difference of time stamps between the current and last poll to avoid deviation due to scheduling latency. However, the timestamp is collected in the Lua plugin which is several steps after the SAI API returns and is executed in a different context (redis-server). Both introduce even larger deviations. To overcome this, we collect the timestamp immediately after the SAI API returns.
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Feb 20, 2025
…et#1457) What I did Optimize the counter-polling performance in terms of polling interval accuracy Enable bulk counter-polling to run at a smaller chunk size There is one counter-polling thread for each counter group. All such threads can compete for the critical sections at the vendor SAI level, which means a counter-polling thread can wait for a critical section if another thread has been in it, which introduces latency for the waiting counter group. An example is the competition between the PFC watchdog and the port counter groups. The port counter group contains many counters and is polled in a bulk mode which takes a relatively longer time. The PFC watchdog counter group contains only a few counters but is polled at a short interval. Sometimes, PFC watchdog counters need to wait before polling, which makes the polling interval inaccurate and prevents the PFC storm from being detected in time. To resolve this issue, we can reduce the chunk size of the port counter group. The port counter group polls the counters of all ports in a single bulk operation by default. By using a smaller chunk size, it polls the counters in several bulk operations with each polling counter of a subset (whose size <= chunk size) of all ports. By doing so, the port counter group stays in the critical section for a shorter time and the PFC watchdog is more likely to be scheduled to poll counters and detect the PFC storm in time. Collect the time stamp immediately after vendor SAI API returns. Currently, many counter groups require a Lua plugin to execute based on polling interval, to calculate rates, detect certain events, etc. Eg. For PFC watchdog counter group to PFC storm. In this case, the polling interval is calculated based on the difference of time stamps between the current and last poll to avoid deviation due to scheduling latency. However, the timestamp is collected in the Lua plugin which is several steps after the SAI API returns and is executed in a different context (redis-server). Both introduce even larger deviations. To overcome this, we collect the timestamp immediately after the SAI API returns.
r12f
added
Request for 202411 Branch
and removed
Request for 202411 Branch
Cherry Pick Conflict_202411
labels
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I did
Optimize the counter-polling performance in terms of polling interval accuracy
Enable bulk counter-polling to run at a smaller chunk size
There is one counter-polling thread for each counter group. All such threads can compete for the critical sections at the vendor SAI level, which means a counter-polling thread can wait for a critical section if another thread has been in it, which introduces latency for the waiting counter group.
An example is the competition between the PFC watchdog and the port counter groups.
The port counter group contains many counters and is polled in a bulk mode which takes a relatively longer time. The PFC watchdog counter group contains only a few counters but is polled at a short interval. Sometimes, PFC watchdog counters need to wait before polling, which makes the polling interval inaccurate and prevents the PFC storm from being detected in time.
To resolve this issue, we can reduce the chunk size of the port counter group. The port counter group polls the counters of all ports in a single bulk operation by default. By using a smaller chunk size, it polls the counters in several bulk operations with each polling counter of a subset (whose size <=
chunk size) of all ports.By doing so, the port counter group stays in the critical section for a shorter time and the PFC watchdog is more likely to be scheduled to poll counters and detect the PFC storm in time.
Collect the time stamp immediately after vendor SAI API returns.
Currently, many counter groups require a Lua plugin to execute based on polling interval, to calculate rates, detect certain events, etc.
Eg. For PFC watchdog counter group to PFC storm. In this case, the polling interval is calculated based on the difference of time stamps between the
currentandlastpoll to avoid deviation due to scheduling latency. However, the timestamp is collected in the Lua plugin which is several steps after the SAI API returns and is executed in a different context (redis-server). Both introduce even larger deviations. To overcome this, we collect the timestamp immediately after the SAI API returns.Depends on
Why I did it
How I verified it
Run regression test and observe counter-polling performance.
A comparison test shows very good results if we put any/or all of the above optimizations.
Details if related
For 2, each counter group contains more than one counter context based on the type of objects. counter context is mapped from (group, object type). But the counters fetched from different counter groups will be pushed into the same entry for the same objects.
eg. PFC_WD group contains counters of ports and queues. PORT group contains counters of ports. QUEUE_STAT group contains counters of queues.
Both PFC_WD and PORT groups will push counter data into an item representing a port. but each counter has its own polling interval, which means counter IDs polled from different counter groups can be polled with different time stamps.
We use the name of a counter group to identify the time stamp of the counter group.
Eg. In port counter entry, PORT_timestamp represents last time when the port counter group polls the counters. PFC_WD_timestamp represents the last time when the PFC watchdog counter group polls the counters