[action] [PR:21771] [chassis-packet] Change the Orchagent redis pop batch size to 128 to handle link notification faster. #21803
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…handle link notification faster. What I did: Change the Orchagent redis pop batch size to 128 to handle link notification faster. Why I did: As part of Ixia BGP convergence test https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snappi_tests/multidut/bgp/test_bgp_outbound_uplink_multi_po_flap.py we found that because of SAI programming slowness which is around 1500 Routes/sec. [Takes about approx 40sec+/- to program 60K routes across multiple iteration] Because of above slowness even if we have Link Notification available Orchagent will not process it immediately as current OA will process 1024 Entries (Route entries in our case) before it can pick Link Notification for processing. Now 1K entries can take about 2 sec+/- and if link notification are little spread out [not back 2 back] we can have batch of 1K entries which accumulate SAI delay of 2 sec. To optimize Link processing and give more chance to OA to pick Link Notification we reduce OA processing to 128 entries and this helped to reduce convergence time to about overall 2 sec. Changing OA processing from 1K to 128 entries does not have any impact of Route Programming as SAI slowness seems be tied with sequential processing at 1500 Routes per/sec. However this is helpful in processing Link Notification quicker. For our test it reduce Convergence time from about 12-15sec to 2 sec. How I verify: Ran above ixia test in multiple iteration with both 1024 and 128 pop batch size and compared the performance.
|
Original PR: #21771 |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I did:
Change the Orchagent redis pop batch size to 128 to handle link notification faster.
Why I did:
As part of Ixia BGP convergence test https://github.com/sonic-net/sonic-mgmt/blob/master/tests/snappi_tests/multidut/bgp/test_bgp_outbound_uplink_multi_po_flap.py we found that because of SAI programming slowness which is around 1500 Routes/sec. [Takes about approx 40sec+/- to program 60K routes across multiple iteration]
Because of above slowness even if we have Link Notification available Orchagent will not process it immediately as current OA will process 1024 Entries (Route entries in our case) before it can pick Link Notification for processing. Now 1K entries can take about 2 sec+/- and if link notification are little spread out [not back 2 back] we can have batch of 1K entries which accumulate SAI delay of 2 sec.
To optimize Link processing and give more chance to OA to pick Link Notification we reduce OA processing to 128 entries and this helped to reduce convergence time to about overall 2 sec.
Changing OA processing from 1K to 128 entries does not have any impact of Route Programming as SAI slowness seems be tied with sequential processing at 1500 Routes per/sec. However this is helpful in processing Link Notification quicker. For our test it reduce Convergence time from about 12-15sec to 2 sec.
How I verify:
Ran above ixia test in multiple iteration with both 1024 and 128 pop batch size and compared the performance.