Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: as7326_56x (leaf) has excess ports/lanes in config -> crashes syncd #33

Open
wdoekes opened this issue Nov 14, 2024 · 2 comments
Open

runtime: as7326_56x (leaf) has excess ports/lanes in config -> crashes syncd #33

wdoekes opened this issue Nov 14, 2024 · 2 comments
Labels
has-fix Has a fix needs-upstream Has possible fixes but needs tickets/PRs upstream

Comments

@wdoekes
Copy link
Member

wdoekes commented Nov 14, 2024

Description

SONiC Software Version: SONiC-OS-4.4.0-Enterprise_Base
Product: Enterprise SONiC Distribution by Broadcom - Enterprise Base package
Distribution: Debian 11.10
Kernel: 5.10.0-21-amd64
Config DB Version: version_4_3_1
Build commit: f1b9727192
Build date: Tue Aug 13 23:52:24 UTC 2024
Built by: sonicbld@bld-lvn-csg-10

Platform: x86_64-accton_as7326_56x-r0
HwSKU: Accton-AS7326-56X
ASIC: broadcom
ASIC Count: 1

Nov 14 15:44:34.951220+00:00 2024 leaf1.dostno.systems NOTICE syncd#syncd: [none] SAI_API_PORT:brcm_sai_remove_port:9962 REMOVE_PORT port_id:0x100000042, bcm_port:66, phy:129
Nov 14 15:44:34.951305+00:00 2024 leaf1.dostno.systems ERR syncd#syncd: [none] SAI_API_PORT:brcm_sai_remove_port:10028 port subsidiary ports get failed with error Invalid port (0xffffffee).
Nov 14 15:44:34.951421+00:00 2024 leaf1.dostno.systems ERR syncd#syncd: :- processQuadEvent: failed to execute api: remove, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000029, status: SAI_STATUS_INVALID_PORT_NUMBER

Due do problems here:
/usr/share/sonic/device/x86_64-accton_as7326_56x-r0/Accton-AS7326-56X/platform.json

    },
    "Ethernet80": {
        "index": "80",
        "default_brkout_mode": "1x10G",
        "lanes": "129",
        "alias_at_lanes": "Eth1/57",
        "breakout_modes": "1x10G"
    },
    "Ethernet81": {
        "index": "81",
        "default_brkout_mode": "1x10G",
        "lanes": "128",
        "alias_at_lanes": "Eth1/58",
        "breakout_modes": "1x10G"
    }

^- drop these two

/usr/share/sonic/device/x86_64-accton_as7326_56x-r0/Accton-AS7326-56X/port_config.ini

...
Ethernet68      113,114,115,116       hundredGigE54     68         100000  100000,40000
Ethernet72      121,122,123,124       hundredGigE55     72         100000  100000,40000
Ethernet76      125,126,127,128       hundredGigE56     76         100000  100000,40000
Ethernet80      129                   mgmtTenGigE57     80         10000   10000,1000
Ethernet81      128                   mgmtTenGigE58     81         10000   10000,1000

^- drop these two

/usr/share/sonic/device/x86_64-accton_as7326_56x-r0/Accton-AS7326-56X/td3-as7326-48x25G+8x100G.config.bcm

...
# Not able to breakout, Since overlap with possible mgmt ports
dport_map_port_127=77
portmap_127=125:100
phy_chain_rx_lane_map_physical{125.0}=0x1203
phy_chain_tx_lane_map_physical{125.0}=0x2031
phy_chain_rx_polarity_flip_physical{125.0}=0x0
phy_chain_rx_polarity_flip_physical{126.0}=0x0
phy_chain_rx_polarity_flip_physical{127.0}=0x1
phy_chain_rx_polarity_flip_physical{128.0}=0x1
phy_chain_tx_polarity_flip_physical{125.0}=0x1
phy_chain_tx_polarity_flip_physical{126.0}=0x1
phy_chain_tx_polarity_flip_physical{127.0}=0x0
phy_chain_tx_polarity_flip_physical{128.0}=0x0

dport_map_port_130=82
dport_map_port_66=81
portmap_130=128:10:m
portmap_66=129:10:m

^-- we need to drop these 4

Which build are we running (if any)

broadcom-4.4.0

Upstream issues/PRs

Result

Nov 14 16:01:46.131997+00:00 2024 leaf1.dostno.systems NOTICE syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_initialize:458 BRCM SAI ver: [11.4.0.0], OCP SAI ver: [1.14.0], SDK ver: [sdk-6.5.30-SP6]

Finally stops crashing:

root@leaf1:~# show interfaces status
  Interface            Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin    Type    Asym PFC
-----------  ---------------  -------  -----  -----  -------  ------  ------  -------  ------  ----------
  Ethernet0                3      25G   9100    N/A   Eth1/1  routed    down       up     N/A         N/A
  Ethernet1                2      25G   9100    N/A   Eth1/2  routed    down       up     N/A         N/A
  Ethernet2                4      25G   9100    N/A   Eth1/3  routed    down       up     N/A         N/A
  Ethernet3                8      25G   9100    N/A   Eth1/4  routed    down       up     N/A         N/A
  Ethernet4                7      25G   9100    N/A   Eth1/5  routed    down       up     N/A         N/A
  Ethernet5                1      25G   9100    N/A   Eth1/6  routed    down       up     N/A         N/A
  Ethernet6                5      25G   9100    N/A   Eth1/7  routed    down       up     N/A         N/A
  Ethernet7               16      25G   9100    N/A   Eth1/8  routed    down       up     N/A         N/A
  Ethernet8                6      25G   9100    N/A   Eth1/9  routed    down       up     N/A         N/A
  Ethernet9               14      25G   9100    N/A  Eth1/10  routed    down       up     N/A         N/A
 Ethernet10               13      25G   9100    N/A  Eth1/11  routed    down       up     N/A         N/A
 Ethernet11               15      25G   9100    N/A  Eth1/12  routed    down       up     N/A         N/A
 Ethernet12               23      25G   9100    N/A  Eth1/13  routed    down       up     N/A         N/A
 Ethernet13               22      25G   9100    N/A  Eth1/14  routed    down       up     N/A         N/A
 Ethernet14               24      25G   9100    N/A  Eth1/15  routed    down       up     N/A         N/A
 Ethernet15               32      25G   9100    N/A  Eth1/16  routed    down       up     N/A         N/A
 Ethernet16               31      25G   9100    N/A  Eth1/17  routed    down       up     N/A         N/A
 Ethernet17               21      25G   9100    N/A  Eth1/18  routed    down       up     N/A         N/A
 Ethernet18               29      25G   9100    N/A  Eth1/19  routed    down       up     N/A         N/A
 Ethernet19               36      25G   9100    N/A  Eth1/20  routed    down       up     N/A         N/A
 Ethernet20               30      25G   9100    N/A  Eth1/21  routed    down       up     N/A         N/A
 Ethernet21               34      25G   9100    N/A  Eth1/22  routed    down       up     N/A         N/A
 Ethernet22               33      25G   9100    N/A  Eth1/23  routed    down       up     N/A         N/A
 Ethernet23               35      25G   9100    N/A  Eth1/24  routed    down       up     N/A         N/A
 Ethernet24               43      25G   9100    N/A  Eth1/25  routed    down       up     N/A         N/A
 Ethernet25               42      25G   9100    N/A  Eth1/26  routed    down       up     N/A         N/A
 Ethernet26               44      25G   9100    N/A  Eth1/27  routed    down       up     N/A         N/A
 Ethernet27               52      25G   9100    N/A  Eth1/28  routed    down       up     N/A         N/A
 Ethernet28               51      25G   9100    N/A  Eth1/29  routed    down       up     N/A         N/A
 Ethernet29               41      25G   9100    N/A  Eth1/30  routed    down       up     N/A         N/A
 Ethernet30               49      25G   9100    N/A  Eth1/31  routed    down       up     N/A         N/A
 Ethernet31               60      25G   9100    N/A  Eth1/32  routed    down       up     N/A         N/A
 Ethernet32               50      25G   9100    N/A  Eth1/33  routed    down       up     N/A         N/A
 Ethernet33               58      25G   9100    N/A  Eth1/34  routed    down       up     N/A         N/A
 Ethernet34               57      25G   9100    N/A  Eth1/35  routed    down       up     N/A         N/A
 Ethernet35               59      25G   9100    N/A  Eth1/36  routed    down       up     N/A         N/A
 Ethernet36               62      25G   9100    N/A  Eth1/37  routed    down       up     N/A         N/A
 Ethernet37               63      25G   9100    N/A  Eth1/38  routed    down       up     N/A         N/A
 Ethernet38               64      25G   9100    N/A  Eth1/39  routed    down       up     N/A         N/A
 Ethernet39               65      25G   9100    N/A  Eth1/40  routed    down       up     N/A         N/A
 Ethernet40               66      25G   9100    N/A  Eth1/41  routed    down       up     N/A         N/A
 Ethernet41               61      25G   9100    N/A  Eth1/42  routed    down       up     N/A         N/A
 Ethernet42               68      25G   9100    N/A  Eth1/43  routed    down       up     N/A         N/A
 Ethernet43               69      25G   9100    N/A  Eth1/44  routed    down       up     N/A         N/A
 Ethernet44               67      25G   9100    N/A  Eth1/45  routed    down       up     N/A         N/A
 Ethernet45               71      25G   9100    N/A  Eth1/46  routed    down       up     N/A         N/A
 Ethernet46               72      25G   9100    N/A  Eth1/47  routed    down       up     N/A         N/A
 Ethernet47               70      25G   9100    N/A  Eth1/48  routed    down       up     N/A         N/A
 Ethernet48      77,78,79,80     100G   9100    N/A  Eth1/49  routed    down       up     N/A         N/A
 Ethernet52      85,86,87,88     100G   9100    N/A  Eth1/50  routed    down       up     N/A         N/A
 Ethernet56      93,94,95,96     100G   9100    N/A  Eth1/51  routed    down       up     N/A         N/A
 Ethernet60     97,98,99,100     100G   9100    N/A  Eth1/52  routed    down       up     N/A         N/A
 Ethernet64  105,106,107,108     100G   9100    N/A  Eth1/53  routed    down       up  QSFP28         N/A
 Ethernet68  113,114,115,116     100G   9100    N/A  Eth1/54  routed    down       up     N/A         N/A
 Ethernet72  121,122,123,124     100G   9100    N/A  Eth1/55  routed    down       up  QSFP28         N/A
 Ethernet76  125,126,127,128     100G   9100    N/A  Eth1/56  routed    down       up  QSFP28         N/A
@wdoekes
Copy link
Member Author

wdoekes commented Nov 15, 2024

In addition, we can start our own image:

root@leaf1:~# show version

SONiC Software Version: SONiC.osso202405.0-49b1c0f39
SONiC OS Version: 12
Distribution: Debian 12.8
Kernel: 6.1.0-22-2-amd64
Build commit: 49b1c0f39
Build date: Wed Nov 13 23:13:12 UTC 2024
Built by: [email protected]

Platform: x86_64-accton_as7326_56x-r0
HwSKU: Accton-AS7326-56X
ASIC: broadcom
ASIC Count: 1

.. which crashes, because the AS7326 does not play nice with BCRM 11.2.13.

But if we replace libsaibcm in the syncd container with libsaibcm_11.4.0.0+repack_amd64.deb from the broadcom-4-4-0 image, we stop the crashing:

--- bcrm-11.2.13.txt	2024-11-15 11:01:35.260985338 +0000
+++ bcrm-11.4.0.txt	2024-11-15 11:01:36.220985312 +0000
@@ -1,4 +1,4 @@
-2024 Nov 14 16:40:06.241963
+2024 Nov 15 10:08:43.098719
 leaf1 DEBUG syncd#syncd: :> syncd_main: enter
 leaf1 NOTICE syncd#syncd: :- initialize: initializeing metadata log function
 leaf1 WARNING syncd#syncd: :- parseCommandLine: param -s is depreacated, use -z
@@ -10,7 +10,6 @@
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VLAN
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_VIRTUAL_ROUTER
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
-leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTE
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEXT_HOP_GROUP
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ROUTER_INTERFACE
@@ -23,7 +22,6 @@
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_LAG
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
-leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_POLICER
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_WRED
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QOS_MAP
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_QUEUE
@@ -33,7 +31,6 @@
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HASH
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_UDF
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
-leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TUNNEL
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_L2MC
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_IPMC
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_RPF_GROUP
@@ -55,6 +52,7 @@
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MY_MAC
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_IPSEC
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_GENERIC_PROGRAMMABLE
+leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_GENERIC_PROGRAMMABLE
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ARS
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ARS_PROFILE
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TWAMP
@@ -79,7 +77,7 @@
 leaf1 NOTICE syncd#syncd: :- initialize: SAI API vendor version: 11400
 leaf1 NOTICE syncd#syncd: :- initialize: SAI API min version: 10900
 leaf1 NOTICE syncd#syncd: :- initialize: SAI API headers version: 11400
-leaf1 NOTICE syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_initialize:458 BRCM SAI ver: [11.2.13.1], OCP SAI ver: [1.14.0], SDK ver: [sdk-6.5.30-SP4]
+leaf1 NOTICE syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_initialize:458 BRCM SAI ver: [11.4.0.0], OCP SAI ver: [1.14.0], SDK ver: [sdk-6.5.30-SP6]
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_BMTOR: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_INBOUND_ROUTING: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_METER: SAI_STATUS_NOT_IMPLEMENTED (-15)
@@ -97,25 +95,218 @@
 leaf1 NOTICE syncd#syncd: :- Syncd: syncd started
 leaf1 NOTICE syncd#syncd: :- onSyncdStart: performing hard reinit since COLD start was performed
 leaf1 NOTICE syncd#syncd: :- readAsicState: loaded 0 switches
-leaf1 NOTICE syncd#syncd: :- readAsicState: read asic state took 0.000095 sec
+leaf1 NOTICE syncd#syncd: :- readAsicState: read asic state took 0.000152 sec
 leaf1 NOTICE syncd#syncd: :- onSyncdStart: hard reinit succeeded
-leaf1 NOTICE syncd#syncd: :- onSyncdStart: on syncd start took 0.000183 sec
+leaf1 NOTICE syncd#syncd: :- onSyncdStart: on syncd start took 0.000268 sec
 leaf1 NOTICE syncd#syncd: :- run: syncd listening for events
 leaf1 NOTICE syncd#syncd: :- run: starting main loop
 leaf1 NOTICE syncd#syncd: :- syncd_ipc_task_main: IPC service is online
 leaf1 NOTICE syncd#syncd: :- processNotifySyncd: very first run is TRUE, op = INIT_VIEW
 leaf1 NOTICE syncd#syncd: :- clearTempView: clearing current TEMP VIEW
-leaf1 NOTICE syncd#syncd: :- clearTempView: clear temp view took 0.000042 sec
+leaf1 NOTICE syncd#syncd: :- clearTempView: clear temp view took 0.000049 sec
 leaf1 NOTICE syncd#syncd: :- processOidCreate: creating switch number 1
-leaf1 NOTICE syncd#syncd: :- threadFunction: time span 959 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
-leaf1 NOTICE syncd#syncd: :- threadFunction: time span 1959 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
-leaf1 ERR syncd#syncd: [none] SAI_API_SWITCH:platform_process_command:430 Platform command "init bcm" failed, rc = -1.
-leaf1 CRIT syncd#syncd: [none] SAI_API_SWITCH:platformInit:1938 initialization command "init bcm" failed, rc = -1 (Internal error).
-leaf1 CRIT syncd#syncd: [none] SAI_API_SWITCH:sai_driver_init:753 Error initializing driver, rc = -1.
-leaf1 CRIT syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1577 initializing SDK failed with error Operation failed (0xfffffff5).
-leaf1 ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_FAILURE
-leaf1 ERR syncd#syncd: :- processQuadEvent: attr: SAI_SWITCH_ATTR_INIT_SWITCH: true
-leaf1 ERR syncd#syncd: :- processQuadEvent: attr: SAI_SWITCH_ATTR_FDB_EVENT_NOTIFY: 0x55a1ca5c9390
-leaf1 ERR syncd#syncd: :- processQuadEvent: attr: SAI_SWITCH_ATTR_PORT_STATE_CHANGE_NOTIFY: 0x55a1ca5c93a0
-leaf1 ERR syncd#syncd: :- processQuadEvent: attr: SAI_SWITCH_ATTR_SWITCH_SHUTDOWN_REQUEST_NOTIFY: 0x55a1ca5c93d0
-leaf1 ERR syncd#syncd: :- processQuadEvent: attr: SAI_SWITCH_ATTR_SRC_MAC_ADDRESS: F8-xx-xx-xx-xx-xx
+leaf1 NOTICE syncd#syncd: :- threadFunction: time span 960 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
+leaf1 NOTICE syncd#syncd: :- threadFunction: time span 1961 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
+leaf1 NOTICE syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1591 SAI Timing: SDK init time 2 seconds
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_BUFFER:_brcm_sai_mmu_retain_mmu_settings_on_speed_change:4763 Do not reset MMU settings on port speed change
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1791 _brcm_sai_no_efp_grp is 0.
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_TAM:_brcm_sai_eapp_config_init:893 There is no Embedded apps configuration.
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_dev_features_init:1312 Switch mac tnl term enable = 0
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_dev_features_init:1322 Svi tcam opt setting = 0
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_TAM:_brcm_sai_eapp_is_config_validated:1304 EAPP: Feature 3 is not valid
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1888 Fast convergence setting = 0
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1892 use route max setting = 0

@wdoekes
Copy link
Member Author

wdoekes commented Nov 15, 2024

For reference, here's the diff between 10.1.42 and 11.4.0:

--- bcrm-10.1.42.txt	2024-11-15 11:01:37.760985374 +0000
+++ bcrm-11.4.0.txt	2024-11-15 11:01:36.220985412 +0000
@@ -1,8 +1,8 @@
-2024 Nov  6 09:15:51.128912
+2024 Nov 15 10:08:43.098719
 leaf1 DEBUG syncd#syncd: :> syncd_main: enter
 leaf1 NOTICE syncd#syncd: :- initialize: initializeing metadata log function
 leaf1 WARNING syncd#syncd: :- parseCommandLine: param -s is depreacated, use -z
-leaf1 NOTICE syncd#syncd: :- Syncd: sairedis git revision 45ff42c3, SAI git revision: be52377
+leaf1 NOTICE syncd#syncd: :- Syncd: sairedis git revision 17e893c5, SAI git revision: e0e9787
 leaf1 NOTICE syncd#syncd: :- threadFunction: starting timer watchdog thread
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SWITCH
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_PORT
@@ -16,6 +16,7 @@
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_NEIGHBOR
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ACL
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
+leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_HOSTIF
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MIRROR
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_SAMPLEPACKET
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_STP
@@ -51,6 +52,7 @@
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_MY_MAC
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_IPSEC
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_GENERIC_PROGRAMMABLE
+leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_GENERIC_PROGRAMMABLE
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ARS
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_ARS_PROFILE
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_TWAMP
@@ -66,8 +68,6 @@
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DASH_VNET
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DASH_PA_VALIDATION
 leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DASH_VIP
-leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DASH_HA
-leaf1 NOTICE syncd#syncd: :- saiLoglevelNotify: Setting SAI loglevel SAI_LOG_LEVEL_NOTICE on SAI_API_DASH_TUNNEL
 leaf1 NOTICE syncd#syncd: :- Syncd: command line:  EnableDiagShell=YES EnableTempView=YES DisableExitSleep=NO EnableUnittests=NO EnableConsistencyCheck=NO EnableSyncMode=YES RedisCommunicationMode=redis_async EnableSaiBulkSuport=NO StartType=cold ProfileMapFile=/etc/sai.d/sai.profile GlobalContext=0 ContextConfig= BreakConfig=/tmp/break_before_make_objects WatchdogWarnTimeSpan=30000000
 leaf1 NOTICE syncd#syncd: :- loadFromFile: no context config specified, will load default context config
 leaf1 NOTICE syncd#syncd: :- insert: added switch: idx 0, hwinfo ''
@@ -74,15 +74,16 @@
 leaf1 WARNING syncd#syncd: :- Syncd: enable sync mode is deprecated, please use communication mode, FORCING redis sync mode
 leaf1 NOTICE syncd#syncd: :- RedisSelectableChannel: opened redis channel
 leaf1 NOTICE syncd#syncd: :- isVeryFirstRun: First Run: True
-leaf1 INFO syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_initialize:451 BRCM SAI ver: [10.1.42.0], OCP SAI ver: [1.13.2], SDK ver: [sdk-6.5.29]
+leaf1 NOTICE syncd#syncd: :- initialize: SAI API vendor version: 11400
-leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_ENI: SAI_STATUS_NOT_IMPLEMENTED (-15)
+leaf1 NOTICE syncd#syncd: :- initialize: SAI API min version: 10900
-leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_HA: SAI_STATUS_NOT_IMPLEMENTED (-15)
+leaf1 NOTICE syncd#syncd: :- initialize: SAI API headers version: 11400
+leaf1 NOTICE syncd#syncd: [none] SAI_API_UNSPECIFIED:sai_api_initialize:458 BRCM SAI ver: [11.4.0.0], OCP SAI ver: [1.14.0], SDK ver: [sdk-6.5.30-SP6]
+leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_BMTOR: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_INBOUND_ROUTING: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_METER: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_OUTBOUND_CA_TO_PA: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_OUTBOUND_ROUTING: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_PA_VALIDATION: SAI_STATUS_NOT_IMPLEMENTED (-15)
-leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_TUNNEL: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_VIP: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_DASH_VNET: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_GENERIC_PROGRAMMABLE: SAI_STATUS_NOT_IMPLEMENTED (-15)
@@ -89,59 +90,60 @@
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_IPSEC: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_POE: SAI_STATUS_NOT_IMPLEMENTED (-15)
 leaf1 NOTICE syncd#syncd: :- sai_metadata_apis_query: :- failed to query api SAI_API_TWAMP: SAI_STATUS_NOT_IMPLEMENTED (-15)
-leaf1 NOTICE syncd#syncd: :- apiInitialize: sai_api_query failed for 14 apis
+leaf1 NOTICE syncd#syncd: :- initialize: sai_api_query failed for 12 apis
-leaf1 NOTICE syncd#syncd: :- apiInitialize: SAI API vendor version: 11302
-leaf1 NOTICE syncd#syncd: :- apiInitialize: SAI API min version: 10900
-leaf1 NOTICE syncd#syncd: :- apiInitialize: SAI API headers version: 11400
 leaf1 NOTICE syncd#syncd: :- parseBreakConfig: break config parse success, contains 2 entries
 leaf1 NOTICE syncd#syncd: :- Syncd: syncd started
 leaf1 NOTICE syncd#syncd: :- onSyncdStart: performing hard reinit since COLD start was performed
 leaf1 NOTICE syncd#syncd: :- readAsicState: loaded 0 switches
-leaf1 NOTICE syncd#syncd: :- readAsicState: read asic state took 0.000167 sec
+leaf1 NOTICE syncd#syncd: :- readAsicState: read asic state took 0.000152 sec
 leaf1 NOTICE syncd#syncd: :- onSyncdStart: hard reinit succeeded
-leaf1 NOTICE syncd#syncd: :- onSyncdStart: on syncd start took 0.000325 sec
+leaf1 NOTICE syncd#syncd: :- onSyncdStart: on syncd start took 0.000268 sec
 leaf1 NOTICE syncd#syncd: :- run: syncd listening for events
 leaf1 NOTICE syncd#syncd: :- run: starting main loop
 leaf1 NOTICE syncd#syncd: :- syncd_ipc_task_main: IPC service is online
 leaf1 NOTICE syncd#syncd: :- processNotifySyncd: very first run is TRUE, op = INIT_VIEW
 leaf1 NOTICE syncd#syncd: :- clearTempView: clearing current TEMP VIEW
-leaf1 NOTICE syncd#syncd: :- clearTempView: clear temp view took 0.000051 sec
+leaf1 NOTICE syncd#syncd: :- clearTempView: clear temp view took 0.000049 sec
 leaf1 NOTICE syncd#syncd: :- processOidCreate: creating switch number 1
-leaf1 NOTICE syncd#syncd: :- threadFunction: time span 575 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
+leaf1 NOTICE syncd#syncd: :- threadFunction: time span 960 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
-leaf1 NOTICE syncd#syncd: :- threadFunction: time span 1576 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
+leaf1 NOTICE syncd#syncd: :- threadFunction: time span 1961 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
-leaf1 NOTICE syncd#syncd: :- threadFunction: time span 2576 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
+leaf1 NOTICE syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1591 SAI Timing: SDK init time 2 seconds
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1495 SAI Timing: SDK init time 3 seconds
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_BUFFER:_brcm_sai_mmu_retain_mmu_settings_on_speed_change:4763 Do not reset MMU settings on port speed change
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_BUFFER:_brcm_sai_mmu_retain_mmu_settings_on_speed_change:4536 Do not reset MMU settings on port speed change
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1791 _brcm_sai_no_efp_grp is 0.
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_TAM:_brcm_sai_eapp_config_init:892 There is no Embedded apps configuration.
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_TAM:_brcm_sai_eapp_config_init:893 There is no Embedded apps configuration.
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_dev_features_init:1178 Switch mac tnl term enable = 0
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_dev_features_init:1312 Switch mac tnl term enable = 0
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_dev_features_init:1188 Svi tcam opt setting = 0
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:_brcm_sai_dev_features_init:1322 Svi tcam opt setting = 0
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1763 Fast convergence setting = 0
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_TAM:_brcm_sai_eapp_is_config_validated:1304 EAPP: Feature 3 is not valid
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1767 Sonic use route max setting = 0
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1888 Fast convergence setting = 0
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_UNSPECIFIED:syncdb_main:3284 SyncDB: path = "/var/warmboot"
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_SWITCH:brcm_sai_create_switch:1892 use route max setting = 0
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_UNSPECIFIED:syncdb_main:3304 SyncDB: _syncdb_nv_path = "/var/warmboot/nvram"
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_UNSPECIFIED:syncdb_main:3285 SyncDB: path = "/var/warmboot"
-leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_UNSPECIFIED:syncdb_main:3312 SyncDB: _syncdb_db_file = "/var/warmboot/brcm_sai_syncdb.dat"
+leaf1 NOTICE|INFO syncd#syncd: [none] SAI_API_UNSPECIFIED:syncdb_main:3306 SyncDB: _syncdb_nv_path = "/var/warmboot/nvram"
...
root@leaf1:~# grep create:SAI_OBJECT_TYPE_SWITCH *.txt 

bcrm-10.1.42.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 575 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
bcrm-10.1.42.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 1576 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
bcrm-10.1.42.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 2576 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
bcrm-10.1.42.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 3576 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'

bcrm-11.2.13.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 959 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
bcrm-11.2.13.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 1959 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'

bcrm-11.4.0.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 960 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
bcrm-11.4.0.txt:leaf1 NOTICE syncd#syncd: :- threadFunction: time span 1961 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'
bcrm-11.4.0.txt:leaf1 NOTICE|INFO syncd#syncd: :- threadFunction: time span 2961 ms for 'create:SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000'

@wdoekes wdoekes changed the title runtime: bcm remove port runtime: as7326_56x (leaf) has excess ports/lanes in config -> crashes syncd Nov 15, 2024
@wdoekes wdoekes added needs-upstream Has possible fixes but needs tickets/PRs upstream has-fix Has a fix labels Nov 19, 2024
wdoekes pushed a commit that referenced this issue Nov 22, 2024
[#33] possible fix for leaf1 workage -- and others, FIXME
b8bd195
wdoekes pushed a commit that referenced this issue Nov 22, 2024
[#33] possible fix for leaf1 workage -- and others, FIXME
7eed3f1
wdoekes pushed a commit that referenced this issue Nov 22, 2024
[#33] possible fix for leaf1 workage -- and others, FIXME
fc2e1a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
has-fix Has a fix needs-upstream Has possible fixes but needs tickets/PRs upstream
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@wdoekes