kindling-collector运行占用cpu过高

[zcl1115]

如题，kindling-collector在部分机器上运行，cpu占用过高，影响到了业务，想问一下有什么优化解决方案吗，已经加了2C的cgroup限制了

[dxsup]

能否提供一下环境信息？

• 主机Pod数
• CPU核心数
• CPU使用率
• kindling-collector的CPU使用率

[dxsup]

这台机器与那些机器的区别就在于内核版本吗？是否还存在其他变量？麻烦再看一下内核日志，看有没有kindling相关的信息。

[zcl1115]

有类似这种内核日志，跟别的机器不太一样
Nov 27 10:35:55 bigo kernel: : [17424363.004500] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
Nov 27 10:35:55 bigo kernel: : [17424391.014680] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
Nov 27 10:35:55 bigo kernel: : [17424418.949634] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
Nov 27 10:35:55 bigo kernel: : [17424446.952238] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
Nov 27 10:35:55 bigo kernel: : [17424474.947042] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
Nov 27 10:35:55 bigo kernel: : [17424502.959421] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
Nov 27 14:23:22 bigo kernel: : [17438010.706391] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
Nov 27 14:23:22 bigo kernel: : [17438038.676255] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) sctp libcrc32c dccp_diag dccp unix_diag af_packet_diag netlink_diag nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag nvme nvme_core ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack 8021q garp mrp stp llc bonding binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp kvm_intel kvm irqbypass intel_rapl_perf input_leds joydev ioatdma mei_me dca lpc_ich mei shpchp nfit mac_hid acpi_power_meter sch_fq tcp_bbr autofs4 btrfs xor raid6_pq hid_generic usbhid hid crct10dif_pclmul crc32_pclmul

[zcl1115]

Nov 27 10:57:23 bigo kernel: : [9332592.513335] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) dccp_diag dccp unix_diag af_packet_diag netlink_diag ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp 8021q garp mrp stp llc kvm_intel kvm irqbypass intel_rapl_perf joydev input_leds lpc_ich mei_me mei ioatdma shpchp dca nfit acpi_power_meter mac_hid sch_fq tcp_bbr bonding autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic aesni_intel aes_x86_64
Nov 27 10:57:23 bigo kernel: : [9332592.513402] [] record_event_consumer.part.9+0x817/0xda0 [kindling_falcolib_probe]
Nov 27 10:57:23 bigo kernel: : [9332592.513413] [] record_event_all_consumers+0x63/0x90 [kindling_falcolib_probe]
Nov 27 10:57:23 bigo kernel: : [9332592.513416] [] syscall_exit_probe+0x10e/0x140 [kindling_falcolib_probe]
Nov 27 10:57:23 bigo kernel: : [9332620.485744] Modules linked in: nf_conntrack_netlink kindling_falcolib_probe(OE) dccp_diag dccp unix_diag af_packet_diag netlink_diag ipmi_si ipmi_devintf ipmi_watchdog ipmi_msghandler bridge nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack xt_multiport nf_log_ipv4 nf_log_common xt_LOG xt_set ip_set_hash_net ip_set nfnetlink iptable_filter ip_tables x_tables tcp_diag udp_diag inet_diag binfmt_misc intel_rapl skx_edac edac_core x86_pkg_temp_thermal coretemp 8021q garp mrp stp llc kvm_intel kvm irqbypass intel_rapl_perf joydev input_leds lpc_ich mei_me mei ioatdma shpchp dca nfit acpi_power_meter mac_hid sch_fq tcp_bbr bonding autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul ghash_clmulni_intel hid_generic aesni_intel aes_x86_64
Nov 27 10:57:23 bigo kernel: : [9332620.485820] [] record_event_consumer.part.9+0x817/0xda0 [kindling_falcolib_probe]
Nov 27 10:57:23 bigo kernel: : [9332620.485834] [] record_event_all_consumers+0x63/0x90 [kindling_falcolib_probe]
Nov 27 10:57:23 bigo kernel: : [9332620.485837] [] syscall_exit_probe+0x10e/0x140 [kindling_falcolib_probe]

[dxsup]

目前看起来是内核模块导致的异常，brpc的那个测试进程的源码能否开源出来？

[zcl1115]

那个测试进程不归属我们监控团队这边负责，是业务团队的，我们开源不了这个，而且感觉这个是跟机器本身有关的，不关乎业务进程的问题，我明天尝试在这台机器上再重新生成一个模块看看是否有异常

[dxsup]

brpc的那个测试进程的源码能否开源出来？我们测试看看。