Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[shim] Missing Initialization on Many Calls #1343

Merged
merged 4 commits into from
Jul 25, 2024
Merged

Conversation

ppenna
Copy link
Collaborator

@ppenna ppenna commented Jul 24, 2024

Description

This PR fixes initialization issues of Demikernel in the POSIX SHIM layer.

Without this modification, initialization would be performed only when epoll_create() is the very first call performed by the application.

@ppenna ppenna added the bug Something Isn't Working label Jul 24, 2024
@ppenna ppenna self-assigned this Jul 24, 2024
Copy link

libos = catnip
commit id = b379e65

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 88.91 18204.45 8165.59
demikernel::new 61.36 1822074333.10 818072707.64
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 44.47 203.59 91.45
bgc::inetstack::poll_recv 35.13 473.56 213.09
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 28.57 857.67 382.95
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.86 1099.42 500.14
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.54 759.36 343.65
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 9.93 48.52 21.99
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 8.41 331.79 150.44
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 7.08 373.11 162.13
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 6.14 1252.27 561.03
bgc::inetstack::tcp::established::background 2.67 3329.99 1503.61
ioc::network::libos::pop 0.54 911.63 413.42
ioc::network::libos::push 0.50 2015.70 920.65
bgc::inetstack::poll_recv;inetstack::poll 0.25 40917.43 18351.89
demikernel::push 0.20 831.74 379.62
ioc::network::libos::pushto 0.10 1331.96 582.09
demikernel::pop 0.06 453.85 203.04
demikernel::sgaalloc 0.05 141.28 64.76
demikernel::pushto 0.04 447.23 194.38
ioc::network::libos::close 0.02 7828.82 3544.18
ioc::network::libos::connect 0.01 14859.97 6754.18
demikernel::sgafree 0.01 67.89 30.61
demikernel::bind 0.00 24062.94 10535.95
bgc::inetstack::tcp::passiveopen::background 0.00 14807.94 6673.86
bgc::passive_listening::poll 0.00 5719.30 2570.52
demikernel::listen 0.00 4645.36 2085.86
demikernel::connect 0.00 3460.82 1545.07
bgc::inetstack::icmp::background 0.00 2860.17 1282.70
bgc::inetstack::arp::background 0.00 2615.34 1170.70
ioc::network::libos::accept 0.00 2470.88 1108.65
demikernel::accept 0.00 1329.79 597.60
demikernel::socket 0.00 1207.47 539.63
demikernel::async_close 0.00 593.42 259.38
demikernel::close 0.00 nan nan

Copy link

libos = catnap
commit id = b379e65

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
demikernel::socket;catnap::linux::transport::socket 97.59 56438.60 25679.15
ioc::network::libos::close;catnap::linux::transport::close 95.84 22746.04 10363.11
demikernel::listen;catnap::linux::transport::listen 92.90 13196.58 5943.84
demikernel::bind;catnap::linux::transport::bind 81.39 25004.12 11259.20
ioc::network::libos::accept 43.25 928431867.45 417791577.22
bgc::catnap::transport::epoll 33.83 2149.56 1022.24
ioc::network::libos::pop 25.29 1727308.50 781865.26
ioc::network::libos::connect 24.87 5131178.28 2404968.12
ioc::network::libos::pushto 23.77 8554.88 3813.37
ioc::network::libos::push 6.90 17757.86 8216.04
ioc::network::libos::pushto;catnap::linux::transport::push 6.40 541.49 241.55
ioc::network::libos::push;catnap::linux::transport::push 4.30 721.31 332.18
ioc::network::libos::connect;catnap::linux::transport::connect 1.87 54127.37 25459.36
ioc::network::libos::pop;catnap::linux::transport::pop 1.73 535.16 242.39
demikernel::pushto 1.29 436.40 194.54
demikernel::new 0.48 1608677.85 733778.50
demikernel::push 0.22 594.82 274.25
demikernel::sgaalloc 0.22 227.46 107.63
demikernel::pop 0.16 751.48 340.04
demikernel::socket 0.10 57801.43 26298.90
ioc::network::libos::close 0.07 23808.30 10848.35
demikernel::sgafree 0.04 78.96 36.33
demikernel::connect 0.01 2170.01 1023.55
demikernel::async_close 0.01 997.94 435.07
demikernel::bind 0.00 30771.48 13853.23
demikernel::listen 0.00 14201.11 6396.00
ioc::network::libos::accept;catnap::linux::transport::accept 0.00 3132.58 1412.36
demikernel::accept 0.00 1733.61 780.03
demikernel::close 0.00 nan nan

Copy link

libos = catpowder
commit id = b379e65

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 76.95 7057.17 3232.88
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 76.56 939.93 432.32
bgc::inetstack::poll_recv 61.89 1300.48 599.15
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 32.21 1562.43 701.30
demikernel::new 27.21 87480384.57 40072515.33
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.57 1196.19 563.14
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.05 824.84 385.67
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 13.96 329.35 142.87
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 7.94 352.91 165.34
bgc::inetstack::tcp::established::background 7.13 4319.20 1989.71
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 5.22 463.50 212.39
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4.53 66.40 30.91
ioc::network::libos::pushto 3.70 5045.62 2184.81
ioc::network::libos::push 3.57 6282.13 2965.30
ioc::network::libos::pop 1.03 993.20 464.71
demikernel::push 0.45 836.52 399.34
demikernel::pushto 0.33 466.76 202.12
ioc::network::libos::connect 0.30 25284.36 11851.10
ioc::network::libos::close 0.22 10879.32 4969.00
demikernel::pop 0.17 512.16 233.83
demikernel::sgaalloc 0.16 249.77 122.36
demikernel::sgafree 0.05 99.08 45.79
bgc::inetstack::poll_recv;inetstack::poll 0.04 9094.73 4162.25
bgc::inetstack::tcp::passiveopen::background 0.01 12065.93 5527.84
demikernel::connect 0.01 4375.77 2080.91
demikernel::async_close 0.01 706.57 310.36
demikernel::bind 0.00 9860.88 4485.36
demikernel::listen 0.00 5187.43 2460.53
bgc::passive_listening::poll 0.00 4788.04 2218.35
bgc::inetstack::icmp::background 0.00 2759.32 1258.75
bgc::inetstack::arp::background 0.00 2514.86 1160.12
ioc::network::libos::accept 0.00 2450.19 1138.10
demikernel::socket 0.00 2053.04 929.63
demikernel::accept 0.00 1242.21 569.36
demikernel::close 0.00 nan nan

@ppenna ppenna requested review from anandbonde, iyzhang and stolet July 24, 2024 19:01
@ppenna ppenna marked this pull request as ready for review July 24, 2024 19:01
Copy link
Collaborator

@stolet stolet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the above change the shim will hang in a multi-threaded program like memcached, but it should run fine after being addressed.

shim/src/interpose.c Outdated Show resolved Hide resolved
Copy link

libos = catpowder
commit id = 57ef93d

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 79.36 8280.40 3650.82
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 76.56 898.55 396.46
bgc::inetstack::poll_recv 61.77 1227.51 541.56
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 32.92 1599.56 691.65
demikernel::new 27.72 95737013.57 42161495.90
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.50 1117.18 492.18
bgc::inetstack::tcp::established::background;tcp::established::background::sender 18.30 756.89 332.24
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 11.78 354.06 154.15
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 7.72 325.74 143.79
bgc::inetstack::tcp::established::background 6.79 4164.43 1820.00
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 4.92 472.30 208.46
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4.40 58.91 26.02
ioc::network::libos::pushto 3.82 4881.13 2188.82
ioc::network::libos::push 3.48 6121.27 2751.76
ioc::network::libos::pop 1.09 1005.10 442.38
demikernel::push 0.39 702.82 313.29
demikernel::pushto 0.35 445.95 199.98
ioc::network::libos::connect 0.31 25135.12 11307.04
ioc::network::libos::close 0.18 10542.67 4665.32
demikernel::pop 0.17 498.30 219.54
demikernel::sgaalloc 0.14 202.14 91.32
demikernel::sgafree 0.05 85.56 37.90
bgc::inetstack::poll_recv;inetstack::poll 0.03 10308.83 4545.23
bgc::inetstack::tcp::passiveopen::background 0.01 12113.53 5347.51
demikernel::connect 0.01 5310.03 2415.08
demikernel::async_close 0.01 719.11 305.81
demikernel::bind 0.00 9804.00 4278.11
bgc::passive_listening::poll 0.00 5149.81 2268.38
demikernel::listen 0.00 5060.07 2212.54
bgc::inetstack::icmp::background 0.00 3141.68 1385.55
ioc::network::libos::accept 0.00 3114.33 1380.43
bgc::inetstack::arp::background 0.00 2792.97 1224.89
demikernel::socket 0.00 1581.60 696.91
demikernel::accept 0.00 1573.62 689.77
demikernel::close 0.00 nan nan

Copy link

libos = catnip
commit id = 57ef93d

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 88.35 17723.60 7890.78
demikernel::new 61.19 1816533173.82 808165084.79
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 44.50 202.60 90.19
bgc::inetstack::poll_recv 35.25 471.97 210.59
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 28.57 868.41 384.00
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.67 1089.63 495.08
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.87 766.07 345.05
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 11.31 370.95 163.28
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 10.17 49.78 22.42
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 8.41 332.13 150.58
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 6.16 1231.83 547.03
bgc::inetstack::tcp::established::background 2.75 3333.40 1498.68
ioc::network::libos::pop 0.53 897.03 406.94
ioc::network::libos::push 0.47 1989.06 906.19
bgc::inetstack::poll_recv;inetstack::poll 0.24 40105.10 17854.12
demikernel::push 0.20 798.66 362.99
ioc::network::libos::pushto 0.06 1248.22 546.07
demikernel::pop 0.06 455.86 203.15
demikernel::sgaalloc 0.04 129.17 59.35
demikernel::pushto 0.03 490.57 214.54
ioc::network::libos::close 0.02 7886.36 3518.65
ioc::network::libos::connect 0.01 15861.09 7189.47
demikernel::sgafree 0.01 70.60 31.63
demikernel::bind 0.00 21883.44 9443.01
bgc::inetstack::tcp::passiveopen::background 0.00 15243.10 6786.70
demikernel::connect 0.00 6145.82 2777.79
bgc::passive_listening::poll 0.00 5751.10 2556.49
demikernel::listen 0.00 5255.14 2319.96
bgc::inetstack::icmp::background 0.00 2889.80 1285.65
bgc::inetstack::arp::background 0.00 2824.96 1253.05
ioc::network::libos::accept 0.00 2605.42 1158.85
demikernel::socket 0.00 1630.12 723.82
demikernel::accept 0.00 1514.00 672.79
demikernel::async_close 0.00 678.65 291.20
demikernel::close 0.00 nan nan

Copy link

libos = catnap
commit id = 57ef93d

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
demikernel::socket;catnap::linux::transport::socket 97.58 53201.97 24157.83
ioc::network::libos::close;catnap::linux::transport::close 95.96 22569.47 10218.03
demikernel::listen;catnap::linux::transport::listen 92.23 12819.29 5837.50
demikernel::bind;catnap::linux::transport::bind 80.52 24134.65 10979.03
ioc::network::libos::accept 42.76 885784207.96 404507378.53
bgc::catnap::transport::epoll 33.91 2416.42 1107.14
ioc::network::libos::connect 25.37 5748527.28 2593344.94
ioc::network::libos::pop 25.04 3972823.88 1767940.17
ioc::network::libos::pushto 23.19 9455.50 4115.60
ioc::network::libos::push 7.20 17648.22 8061.24
ioc::network::libos::pushto;catnap::linux::transport::push 6.38 579.17 251.09
ioc::network::libos::push;catnap::linux::transport::push 4.48 740.09 337.56
ioc::network::libos::pop;catnap::linux::transport::pop 2.07 550.23 248.19
ioc::network::libos::connect;catnap::linux::transport::connect 1.69 53700.64 24215.70
demikernel::pushto 1.22 439.27 190.87
demikernel::new 0.53 1600766.29 726111.78
demikernel::push 0.24 590.38 267.86
demikernel::sgaalloc 0.21 206.09 94.04
demikernel::pop 0.17 718.38 325.85
demikernel::socket 0.08 54490.42 24742.16
ioc::network::libos::close 0.06 23589.76 10679.51
demikernel::sgafree 0.04 76.79 34.68
demikernel::bind 0.01 30135.48 13710.36
demikernel::connect 0.01 2058.98 927.14
demikernel::async_close 0.01 959.84 427.41
demikernel::listen 0.00 13907.32 6331.81
ioc::network::libos::accept;catnap::linux::transport::accept 0.00 2998.28 1357.92
demikernel::accept 0.00 1638.61 744.19
demikernel::close 0.00 nan nan

@ppenna ppenna force-pushed the bugfix-shim-initialization branch from 57ef93d to fe5307c Compare July 25, 2024 23:01
@ppenna ppenna requested a review from stolet July 25, 2024 23:01
Copy link
Collaborator

@stolet stolet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

libos = catpowder
commit id = fe5307c

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 76.89 7251.40 3269.65
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 76.30 891.48 401.00
bgc::inetstack::poll_recv 62.29 1216.89 547.85
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 32.24 1474.05 649.45
demikernel::new 27.61 92720557.00 41417577.96
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.77 1188.12 542.84
bgc::inetstack::tcp::established::background;tcp::established::background::sender 18.51 788.77 357.21
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 14.34 358.97 158.34
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 7.88 343.51 156.83
bgc::inetstack::tcp::established::background 6.51 4169.36 1874.39
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 5.27 472.83 212.32
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 4.53 59.97 27.16
ioc::network::libos::push 3.45 6177.74 2842.89
ioc::network::libos::pushto 2.75 5032.23 2198.25
ioc::network::libos::pop 1.08 956.11 437.18
ioc::network::libos::connect 0.29 25189.71 11439.58
demikernel::push 0.29 682.87 312.59
demikernel::pushto 0.25 471.46 206.01
ioc::network::libos::close 0.22 10466.44 4688.47
demikernel::pop 0.15 496.30 222.28
demikernel::sgaalloc 0.10 133.94 61.41
demikernel::sgafree 0.05 87.54 39.79
bgc::inetstack::poll_recv;inetstack::poll 0.03 9387.83 4231.31
bgc::inetstack::tcp::passiveopen::background 0.01 12428.38 5621.44
demikernel::connect 0.01 4910.80 2231.52
demikernel::async_close 0.01 735.52 314.41
demikernel::bind 0.00 8924.31 3972.86
demikernel::listen 0.00 5214.86 2345.58
bgc::passive_listening::poll 0.00 4806.13 2174.65
bgc::inetstack::icmp::background 0.00 3115.20 1398.12
ioc::network::libos::accept 0.00 2528.79 1146.01
bgc::inetstack::arp::background 0.00 2431.40 1087.32
demikernel::socket 0.00 1845.78 829.76
demikernel::accept 0.00 1220.82 550.67
demikernel::close 0.00 nan nan

Copy link

libos = catnip
commit id = fe5307c

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::receive 88.79 18350.97 8349.22
demikernel::new 61.11 1815016774.05 826099341.20
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::receive 44.83 203.26 92.52
bgc::inetstack::poll_recv 35.47 470.48 214.36
bgc::inetstack::tcp::established::background;tcp::established::background::receiver 27.83 853.39 381.07
bgc::inetstack::tcp::established::background;tcp::established::background::retransmitter 25.93 1086.20 492.97
bgc::inetstack::tcp::established::background;tcp::established::background::sender 19.97 771.50 349.79
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for 9.93 48.57 22.21
bgc::inetstack::poll_recv;inetstack::poll_bg_work::for::for;udp::receive 9.75 377.95 165.43
bgc::inetstack::tcp::established::background;tcp::established::background::acknowledger 8.39 326.10 147.75
bgc::inetstack::poll_recv;inetstack::poll;inetstack::poll_bg_work::for::for 5.89 1211.62 552.06
bgc::inetstack::tcp::established::background 2.65 3323.87 1501.26
ioc::network::libos::pop 0.52 896.92 404.34
ioc::network::libos::push 0.47 1994.22 918.03
bgc::inetstack::poll_recv;inetstack::poll 0.28 41323.80 18806.01
demikernel::push 0.15 672.96 309.78
demikernel::pop 0.06 469.31 209.78
ioc::network::libos::pushto 0.05 1202.28 580.29
demikernel::sgaalloc 0.05 130.91 60.95
ioc::network::libos::close 0.02 7628.28 3498.98
demikernel::pushto 0.02 421.58 204.46
ioc::network::libos::connect 0.01 15246.39 7034.96
demikernel::sgafree 0.01 68.44 31.03
demikernel::bind 0.00 22977.00 10024.04
bgc::inetstack::tcp::passiveopen::background 0.00 15226.13 7043.73
bgc::passive_listening::poll 0.00 5670.61 2599.12
demikernel::listen 0.00 5157.00 2316.97
demikernel::connect 0.00 4186.05 1956.95
bgc::inetstack::icmp::background 0.00 2774.72 1257.09
bgc::inetstack::arp::background 0.00 2649.68 1192.38
ioc::network::libos::accept 0.00 2503.92 1144.38
demikernel::accept 0.00 1420.42 657.46
demikernel::socket 0.00 1376.79 657.02
demikernel::async_close 0.00 671.59 293.12
demikernel::close 0.00 nan nan

Copy link

libos = catnap
commit id = fe5307c

collapsed_stack percent_time cycles_per_call nanoseconds_per_call
demikernel::socket;catnap::linux::transport::socket 97.47 55850.60 25464.32
ioc::network::libos::close;catnap::linux::transport::close 95.68 23149.62 10536.86
demikernel::listen;catnap::linux::transport::listen 92.23 12990.42 5928.97
demikernel::bind;catnap::linux::transport::bind 81.36 25401.48 11554.97
ioc::network::libos::accept 42.96 891531711.59 404389339.48
bgc::catnap::transport::epoll 33.78 2356.17 1086.32
ioc::network::libos::pop 25.29 3838793.34 1681304.13
ioc::network::libos::connect 25.21 4120580.06 1883821.36
ioc::network::libos::pushto 23.04 9696.12 4221.39
ioc::network::libos::push 7.02 17528.88 8081.95
ioc::network::libos::pushto;catnap::linux::transport::push 6.39 589.66 256.64
ioc::network::libos::push;catnap::linux::transport::push 4.30 715.54 329.46
ioc::network::libos::pop;catnap::linux::transport::pop 2.53 542.12 246.28
ioc::network::libos::connect;catnap::linux::transport::connect 2.19 54103.94 24960.59
demikernel::pushto 1.25 501.76 218.38
demikernel::new 0.57 1626342.40 740646.69
demikernel::sgaalloc 0.23 271.52 124.86
demikernel::push 0.17 510.61 234.76
demikernel::pop 0.12 561.92 253.81
demikernel::socket 0.11 57281.03 26114.79
ioc::network::libos::close 0.08 24264.41 11044.26
demikernel::sgafree 0.04 78.66 35.69
demikernel::async_close 0.02 837.00 361.36
demikernel::bind 0.01 31325.88 14248.13
demikernel::connect 0.01 2254.10 1044.66
demikernel::listen 0.00 14091.00 6425.83
ioc::network::libos::accept;catnap::linux::transport::accept 0.00 2931.77 1339.91
demikernel::accept 0.00 1760.52 792.67
demikernel::close 0.00 nan nan

@ppenna ppenna merged commit efd03c5 into dev Jul 25, 2024
19 checks passed
@ppenna ppenna deleted the bugfix-shim-initialization branch July 25, 2024 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something Isn't Working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants