Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[runtime] Bug Fix: Remove task id leak in scheduler #1339

Merged
merged 1 commit into from
Jul 24, 2024

Conversation

iyzhang
Copy link
Contributor

@iyzhang iyzhang commented Jul 23, 2024

This PR fixes a leak in our scheduler that was causing the id map to grow continuously.

@iyzhang iyzhang requested a review from ppenna July 23, 2024 15:41
@iyzhang iyzhang self-assigned this Jul 23, 2024
Copy link
Contributor

@anandbonde anandbonde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@iyzhang iyzhang force-pushed the bugfix-scheduler-task-id branch from 24079a8 to 789f62b Compare July 23, 2024 15:53
Copy link

libos = catpowder
commit id = 789f62b

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 76.74 908.96 408.08
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 76.42 6955.00 3134.92
bgc::inetstack::poll_recv 61.64 1239.71 556.44
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 31.79 1510.94 653.16
demikernel::new 27.97 94836405.03 42628551.28
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 26.00 1221.25 556.01
bgc::inetstack::tcp::established::background;tcp::established::background::sender 18.83 815.94 369.26
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 15.58 360.94 158.00
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 8.21 371.70 168.38
bgc::inetstack::tcp::established::background 6.76 4308.15 1920.23
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 4.98 429.30 190.54
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4.33 58.47 26.30
ioc::network::libos::push 3.54 6088.06 2832.95
ioc::network::libos::pushto 3.01 5745.69 2492.13
ioc::network::libos::pop 1.03 959.22 436.11
ioc::network::libos::connect 0.33 26051.00 11816.09
demikernel::push 0.29 629.82 291.13
demikernel::pushto 0.27 546.65 237.06
ioc::network::libos::close 0.24 10773.17 4832.34
demikernel::pop 0.15 492.45 221.22
demikernel::sgaalloc 0.10 126.70 58.18
bgc::inetstack::poll_recv;inetstack::poll 0.04 9077.73 4089.51
demikernel::sgafree 0.04 74.87 33.69
bgc::inetstack::tcp::passiveopen::background 0.01 12327.71 5653.42
demikernel::connect 0.01 5201.38 2402.91
demikernel::async_close 0.01 700.55 298.66
demikernel::bind 0.00 10355.06 4641.87
bgc::passive_listening::poll 0.00 4885.74 2232.76
demikernel::listen 0.00 4797.07 2177.62
bgc::inetstack::icmp::background 0.00 2882.90 1288.30
bgc::inetstack::arp::background 0.00 2689.58 1199.60
ioc::network::libos::accept 0.00 2472.07 1128.93
demikernel::socket 0.00 1593.32 714.22
demikernel::accept 0.00 1167.91 531.63
demikernel::close 0.00 nan nan

Copy link

libos = catnip
commit id = 789f62b

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 88.38 17859.13 7932.62
demikernel::new 61.30 1813535910.97 804259706.56
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 44.09 203.08 90.00
bgc::inetstack::poll_recv 35.17 481.04 213.16
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 27.61 844.94 370.12
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 26.21 1081.42 484.64
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.73 746.68 331.50
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 11.41 359.60 157.66
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 10.38 51.86 22.96
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 8.48 326.57 146.02
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 6.40 1295.40 576.71
bgc::inetstack::tcp::established::background 2.77 3286.06 1460.26
ioc::network::libos::pop 0.52 894.08 401.89
ioc::network::libos::push 0.47 1983.23 905.24
bgc::inetstack::poll_recv;inetstack::poll 0.28 40400.57 17949.92
demikernel::push 0.15 646.29 292.61
ioc::network::libos::pushto 0.06 1196.69 529.36
demikernel::pop 0.06 460.97 205.78
demikernel::sgaalloc 0.05 146.86 65.66
ioc::network::libos::close 0.02 7478.75 3335.06
demikernel::pushto 0.02 420.64 186.01
ioc::network::libos::connect 0.01 14476.12 6602.36
demikernel::sgafree 0.01 67.67 30.32
demikernel::bind 0.00 22383.94 9523.02
bgc::inetstack::tcp::passiveopen::background 0.00 13858.98 6284.21
bgc::passive_listening::poll 0.00 5635.98 2540.02
demikernel::listen 0.00 4701.93 2112.54
demikernel::connect 0.00 4308.40 2001.70
bgc::inetstack::arp::background 0.00 2695.08 1197.76
bgc::inetstack::icmp::background 0.00 2684.45 1189.01
ioc::network::libos::accept 0.00 2487.93 1120.63
demikernel::accept 0.00 1692.70 756.54
demikernel::socket 0.00 1448.93 642.59
demikernel::async_close 0.00 759.45 318.46
demikernel::close 0.00 nan nan

Copy link

libos = catnap
commit id = 789f62b

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
demikernel::socket;catnap::linux::transport::socket 97.35 54382.61 24439.80
ioc::network::libos::close;catnap::linux::transport::close 95.82 22313.65 10030.88
demikernel::listen;catnap::linux::transport::listen 92.75 14015.29 6254.77
demikernel::bind;catnap::linux::transport::bind 80.54 25944.32 11600.19
ioc::network::libos::accept 42.95 902581075.12 406492143.68
bgc::catnap::transport::epoll 33.73 2254.84 1041.80
ioc::network::libos::pop 25.57 1616560.58 691733.21
ioc::network::libos::connect 25.00 4101046.04 1936141.14
ioc::network::libos::pushto 23.74 8406.61 3436.56
ioc::network::libos::push 6.97 17743.30 8114.82
ioc::network::libos::pushto;catnap::linux::transport::push 6.57 542.95 223.95
ioc::network::libos::push;catnap::linux::transport::push 4.38 744.35 338.78
ioc::network::libos::pop;catnap::linux::transport::pop 3.25 497.15 223.50
ioc::network::libos::connect;catnap::linux::transport::connect 2.04 52771.25 24173.67
demikernel::pushto 1.28 421.36 173.52
demikernel::new 0.49 1621957.31 730755.53
demikernel::sgaalloc 0.24 278.50 126.17
demikernel::push 0.18 509.67 231.51
demikernel::pop 0.11 609.07 273.92
demikernel::socket 0.10 55879.16 25111.02
ioc::network::libos::close 0.07 23335.55 10488.03
demikernel::sgafree 0.04 77.79 35.19
demikernel::async_close 0.02 977.15 420.30
demikernel::connect 0.01 2259.90 1037.84
demikernel::bind 0.00 32539.38 14559.24
demikernel::listen 0.00 15072.76 6727.62
ioc::network::libos::accept;catnap::linux::transport::accept 0.00 3191.16 1432.30
demikernel::accept 0.00 1599.28 715.45
demikernel::close 0.00 nan nan

Copy link

libos = catpowder
commit id = 789f62b

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 77.01 7461.93 3406.55
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 76.44 923.11 419.04
bgc::inetstack::poll_recv 61.09 1277.87 579.69
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 32.60 1569.94 718.51
demikernel::new 28.60 93574171.07 42750291.95
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 24.48 1143.13 529.66
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.40 838.24 386.95
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 12.62 433.26 196.05
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 7.57 338.88 156.54
bgc::inetstack::tcp::established::background 6.85 4291.75 1973.72
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 4.67 432.17 197.37
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4.39 61.59 27.88
ioc::network::libos::push 3.40 6093.61 2801.78
ioc::network::libos::pushto 2.72 6027.63 2711.64
ioc::network::libos::pop 1.05 1032.61 475.97
ioc::network::libos::connect 0.27 25360.84 11401.35
demikernel::push 0.27 623.16 284.92
demikernel::pushto 0.24 599.93 269.68
ioc::network::libos::close 0.20 10228.37 4613.96
demikernel::pop 0.15 523.82 239.49
demikernel::sgaalloc 0.10 152.91 68.47
bgc::inetstack::poll_recv;inetstack::poll 0.04 9616.23 4389.27
demikernel::sgafree 0.04 78.84 35.89
bgc::inetstack::tcp::passiveopen::background 0.01 12947.92 5970.89
demikernel::connect 0.01 3501.05 1560.98
demikernel::async_close 0.01 838.13 388.09
demikernel::bind 0.00 8543.88 3963.18
demikernel::listen 0.00 6553.50 3076.49
bgc::passive_listening::poll 0.00 5312.15 2457.93
bgc::inetstack::icmp::background 0.00 2959.68 1346.01
ioc::network::libos::accept 0.00 2601.99 1203.88
bgc::inetstack::arp::background 0.00 2468.27 1133.61
demikernel::socket 0.00 1500.66 686.66
demikernel::accept 0.00 1464.83 669.21
demikernel::close 0.00 nan nan

Copy link

libos = catnip
commit id = 789f62b

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 88.59 17742.62 7896.30
demikernel::new 61.33 1819658862.30 809231881.70
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 44.66 202.87 90.22
bgc::inetstack::poll_recv 35.11 472.86 210.46
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 28.32 864.92 383.29
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.93 1080.28 491.80
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.54 742.69 334.95
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 10.15 49.90 22.28
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 8.41 326.48 148.37
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 6.30 1253.32 557.95
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 2.92 394.59 170.62
bgc::inetstack::tcp::established::background 2.79 3297.24 1484.61
ioc::network::libos::pop 0.52 902.79 409.40
ioc::network::libos::push 0.46 1956.80 886.86
ioc::network::libos::pushto 0.34 1202.13 527.91
bgc::inetstack::poll_recv;inetstack::poll 0.19 40029.70 17814.94
demikernel::push 0.14 654.72 293.56
demikernel::pushto 0.13 440.21 193.61
demikernel::pop 0.06 467.60 208.31
demikernel::sgaalloc 0.05 147.75 67.99
ioc::network::libos::close 0.02 10916.93 4803.83
ioc::network::libos::connect 0.01 15017.09 6661.40
demikernel::sgafree 0.01 68.76 30.79
demikernel::bind 0.00 23929.50 10397.19
bgc::inetstack::tcp::passiveopen::background 0.00 14985.94 6739.37
bgc::passive_listening::poll 0.00 5686.44 2551.36
demikernel::listen 0.00 4750.00 2122.58
demikernel::connect 0.00 3839.94 1679.78
bgc::inetstack::icmp::background 0.00 3565.60 1591.81
bgc::inetstack::arp::background 0.00 2715.12 1205.75
ioc::network::libos::accept 0.00 2554.07 1149.82
demikernel::accept 0.00 1730.39 773.95
demikernel::socket 0.00 1451.00 640.34
demikernel::async_close 0.00 579.26 247.47
demikernel::close 0.00 nan nan

Copy link

libos = catnap
commit id = 789f62b

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
demikernel::socket;catnap::linux::transport::socket 97.25 58483.69 26196.01
ioc::network::libos::close;catnap::linux::transport::close 95.77 22240.75 9973.06
demikernel::listen;catnap::linux::transport::listen 92.76 13211.63 5958.90
demikernel::bind;catnap::linux::transport::bind 81.02 26376.32 11788.72
ioc::network::libos::accept 43.00 888180637.52 404367963.15
bgc::catnap::transport::epoll 33.68 2188.54 991.93
ioc::network::libos::pop 25.59 3808234.76 1610886.99
ioc::network::libos::connect 25.34 5440243.22 2437393.15
ioc::network::libos::pushto 23.08 8983.05 3862.70
ioc::network::libos::push 6.81 17603.95 8003.80
ioc::network::libos::pushto;catnap::linux::transport::push 6.40 564.15 243.62
ioc::network::libos::push;catnap::linux::transport::push 4.28 711.44 322.30
ioc::network::libos::pop;catnap::linux::transport::pop 2.71 507.65 227.47
ioc::network::libos::connect;catnap::linux::transport::connect 1.94 55143.94 24836.88
demikernel::pushto 1.20 430.07 185.48
demikernel::new 0.55 1618455.60 725638.27
demikernel::sgaalloc 0.22 237.97 109.35
demikernel::push 0.17 499.13 225.27
demikernel::pop 0.11 546.10 244.12
demikernel::socket 0.08 60169.91 26953.64
ioc::network::libos::close 0.06 23286.77 10441.01
demikernel::sgafree 0.04 79.91 35.95
demikernel::bind 0.01 32557.05 14579.18
demikernel::connect 0.01 2158.04 971.60
demikernel::async_close 0.01 900.14 386.74
demikernel::listen 0.00 14239.21 6424.07
ioc::network::libos::accept;catnap::linux::transport::accept 0.00 3060.98 1385.54
demikernel::accept 0.00 1536.57 692.18
demikernel::close 0.00 nan nan

@iyzhang iyzhang force-pushed the bugfix-scheduler-task-id branch from 789f62b to d5fd49f Compare July 24, 2024 00:44
Copy link

libos = catpowder
commit id = d5fd49f

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 77.56 6903.00 3069.67
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 76.63 905.09 402.67
bgc::inetstack::poll_recv 61.60 1248.98 555.68
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 32.48 1485.10 646.93
demikernel::new 27.79 92234922.40 40897334.63
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.33 1143.00 506.91
bgc::inetstack::tcp::established::background;tcp::established::background::sender 18.50 787.91 347.77
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 7.98 342.04 151.85
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 7.59 322.06 154.72
ioc::network::libos::pushto 7.53 4665.48 2234.16
bgc::inetstack::tcp::established::background 6.70 4113.71 1810.04
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 4.74 401.43 178.14
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4.34 59.31 26.45
ioc::network::libos::push 3.58 6061.22 2734.74
ioc::network::libos::pop 1.05 923.90 411.68
demikernel::pushto 0.69 426.30 204.31
demikernel::push 0.31 642.84 286.36
ioc::network::libos::connect 0.29 21426.59 9515.24
ioc::network::libos::close 0.22 10209.38 4507.95
demikernel::pop 0.16 479.29 212.91
demikernel::sgaalloc 0.12 160.61 70.85
demikernel::sgafree 0.05 75.75 33.85
bgc::inetstack::poll_recv;inetstack::poll 0.02 8820.13 3920.73
demikernel::async_close 0.02 809.48 342.89
bgc::inetstack::tcp::passiveopen::background 0.01 12461.19 5630.73
demikernel::connect 0.01 4142.46 1831.95
demikernel::bind 0.00 10024.50 4479.55
demikernel::listen 0.00 5253.07 2358.58
bgc::passive_listening::poll 0.00 5056.29 2279.65
bgc::inetstack::icmp::background 0.00 2792.67 1241.52
ioc::network::libos::accept 0.00 2549.27 1145.51
bgc::inetstack::arp::background 0.00 2379.38 1058.04
demikernel::socket 0.00 2004.32 894.04
demikernel::accept 0.00 1446.87 646.11
demikernel::close 0.00 nan nan

Copy link

libos = catnip
commit id = d5fd49f

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 88.47 17766.88 7897.15
demikernel::new 61.23 1811650797.57 804381678.59
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 44.79 202.92 90.12
bgc::inetstack::poll_recv 35.21 471.50 209.65
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 27.92 841.19 371.11
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.97 1079.35 488.41
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.63 744.50 333.76
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 10.06 49.45 22.11
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 8.47 328.41 148.22
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 6.18 391.95 172.06
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 6.16 1228.02 545.09
bgc::inetstack::tcp::established::background 2.79 3278.84 1468.99
ioc::network::libos::pop 0.53 896.76 405.57
ioc::network::libos::push 0.46 1949.59 888.71
bgc::inetstack::poll_recv;inetstack::poll 0.22 40125.80 17831.89
ioc::network::libos::pushto 0.15 1199.28 525.28
demikernel::push 0.14 651.92 294.93
demikernel::pop 0.06 452.73 201.63
demikernel::pushto 0.06 421.65 184.87
demikernel::sgaalloc 0.06 144.67 65.10
ioc::network::libos::close 0.02 7630.45 3383.23
ioc::network::libos::connect 0.01 15037.86 6712.77
demikernel::sgafree 0.01 67.91 30.36
demikernel::bind 0.00 23432.69 10239.45
bgc::inetstack::tcp::passiveopen::background 0.00 15285.63 6924.35
bgc::passive_listening::poll 0.00 5750.20 2597.54
demikernel::listen 0.00 5194.36 2332.45
demikernel::connect 0.00 4062.79 1802.94
bgc::inetstack::icmp::background 0.00 2882.42 1277.83
bgc::inetstack::arp::background 0.00 2882.17 1280.44
ioc::network::libos::accept 0.00 2547.18 1150.81
demikernel::socket 0.00 1612.04 717.67
demikernel::accept 0.00 1344.91 605.81
demikernel::async_close 0.00 637.23 270.32
demikernel::close 0.00 nan nan

Copy link

libos = catnap
commit id = d5fd49f

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
demikernel::socket;catnap::linux::transport::socket 97.51 55925.57 25185.35
ioc::network::libos::close;catnap::linux::transport::close 95.69 22653.74 10199.68
demikernel::listen;catnap::linux::transport::listen 92.78 13575.24 6107.48
demikernel::bind;catnap::linux::transport::bind 81.31 25639.52 11529.51
ioc::network::libos::accept 42.71 884188404.77 399909411.46
bgc::catnap::transport::epoll 34.41 2476.31 1158.76
ioc::network::libos::pop 24.65 1814479.20 792085.50
ioc::network::libos::connect 24.44 4530790.99 2078634.65
ioc::network::libos::pushto 23.55 8659.75 3786.29
ioc::network::libos::push 7.64 17347.64 7940.50
ioc::network::libos::pushto;catnap::linux::transport::push 6.21 525.33 229.08
ioc::network::libos::push;catnap::linux::transport::push 4.35 709.90 323.79
ioc::network::libos::pop;catnap::linux::transport::pop 2.63 532.82 240.15
ioc::network::libos::connect;catnap::linux::transport::connect 2.12 57806.71 26180.73
demikernel::pushto 1.26 415.08 181.03
demikernel::new 0.45 1631743.00 736744.70
demikernel::sgaalloc 0.24 253.31 116.06
demikernel::push 0.18 519.84 236.21
demikernel::pop 0.13 541.76 243.79
demikernel::socket 0.08 57380.30 25838.14
ioc::network::libos::close 0.07 23734.67 10686.34
demikernel::sgafree 0.04 80.96 36.82
demikernel::async_close 0.02 1106.59 477.07
demikernel::connect 0.01 2155.24 972.17
demikernel::bind 0.00 31639.60 14225.49
demikernel::listen 0.00 14632.45 6583.02
ioc::network::libos::accept;catnap::linux::transport::accept 0.00 2998.68 1341.98
demikernel::accept 0.00 1591.23 714.84
demikernel::close 0.00 nan nan

@iyzhang iyzhang merged commit 8184596 into dev Jul 24, 2024
19 checks passed
@iyzhang iyzhang deleted the bugfix-scheduler-task-id branch July 24, 2024 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants