NAT functionality of TheRouter is based on the NPF standalone lib. So the configuration process of source NAT is very similar to the configuration of the NetBSD packet filter except that the router uses a fork of NPF lib. Our NPF fork has a lot of new code and a different core for handling the connection state database.
To describe and measure the perfomance of the source NAT function of TheRouter a testing lab was build. The lab consists of 3 servers and one L2 switch connecting all the servers into a L2 network. From a L3 point of view the lab consists of two hosts and a router connectiong them. Host S1 establishes tcp connections to host S3 and the router performs source NAT for those tcp connections. To establish a large number of connections and generate traffic the Stateful Traffic Generator WARP17 is used on both S1 and S2 hosts.
- Processor: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz
- NIC: Intel X520-DA2
- Ram: 8Gb DDR3
- Processor: Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz
- NIC: Intel X520-DA2
- Ram: 8Gb DDR3
- Processor: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
- NIC: Intel X520-DA2
- Ram: 32Gb 4x8
-
/etc/router.conf
startup { sysctl set mbuf 8192 port 0 mtu 1500 tpid 0x8100 state enabled port 1 mtu 1500 tpid 0x8100 state enabled rx_queue port 0 queue 0 lcore 1 rx_queue port 0 queue 1 lcore 2 rx_queue port 0 queue 2 lcore 3 rx_queue port 0 queue 3 lcore 4 rx_queue port 0 queue 4 lcore 5 rx_queue port 1 queue 0 lcore 5 rx_queue port 1 queue 1 lcore 4 rx_queue port 1 queue 2 lcore 3 rx_queue port 1 queue 3 lcore 2 rx_queue port 1 queue 4 lcore 1 sysctl set global_packet_counters 1 } runtime { vif add name p0 port 1 type untagged ip addr add 10.0.0.1/24 dev p0 vif add name p1 port 0 type untagged ip addr add 10.0.1.1/24 dev p1 ip route add 0.0.0.0/0 via 10.0.1.2 src 10.0.1.1 npf load "/etc/npf.conf.warp17" }
-
/etc/npf.conf.warp17
map p1 netmap 10.0.100.0/24 group default { pass stateful final on p1 all pass final on p0 all }
set tests mtu port 0 1500
add tests l3_intf port 0 ip 10.0.0.2 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.3 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.4 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.5 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.6 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.7 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.8 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.9 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.10 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.11 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.12 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.13 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.14 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.15 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.16 mask 255.255.255.0
add tests l3_intf port 0 ip 10.0.0.17 mask 255.255.255.0
add tests l3_gw port 0 gw 10.0.0.1
add tests client tcp port 0 test-case-id 0 src 10.0.0.2 10.0.0.17 sport 1001 60000 dest 10.0.1.2 10.0.1.2 dport 6001 6001
set tests timeouts port 0 test-case-id 0 init 4
set tests timeouts port 0 test-case-id 0 uptime infinite
set tests timeouts port 0 test-case-id 0 downtime 2
set tests rate port 0 test-case-id 0 open infinite
set tests rate port 0 test-case-id 0 send infinite
set tests criteria port 0 test-case-id 0 run-time 600
set tests client raw port 0 test-case-id 0 data-req-plen 64 data-resp-plen 64
start tests port 0
show tests ui
add tests l3_intf port 0 ip 10.0.1.2 mask 255.255.255.0
add tests l3_gw port 0 gw 10.0.1.1
add tests server tcp port 0 test-case-id 0 src 10.0.1.2 10.0.1.2 sport 6001 6001
set tests server raw port 0 test-case-id 0 data-req-plen 64 data-resp-plen 64
start tests port 0
show tests ui
-
Switch port 26 counters
Port Number : 26 ================================================================== Frame Size/Type Frame Counts Frames/sec --------------- ---------------------- ----------- 64 87024610 2815507 65-127 84094530 2815504 128-255 0 0 256-511 0 0 512-1023 0 0 1024-1518 0 0 Unicast RX 86007867 2809947 Multicast RX 0 0 Broadcast RX 0 0 Frame Type Total Total/sec --------------- ---------------------- ----------- RX Bytes 7970481638 261333199 RX Frames 86008496 2809978 TX Bytes 7858637280 261334242 TX Frames 85110670 2809971
-
Switch port 27 counters
Port Number : 27 ================================================================== Frame Size/Type Frame Counts Frames/sec --------------- ---------------------- ----------- 64 350547141 2813693 65-127 347636843 2813692 128-255 0 0 256-511 0 0 512-1023 0 0 1024-1518 0 0 Unicast RX 348611411 2809496 Multicast RX 0 0 Broadcast RX 19 0 Frame Type Total Total/sec --------------- ---------------------- ----------- RX Bytes 32365232450 261001311 RX Frames 348612008 2809484 TX Bytes 32481488302 261000601 TX Frames 349571993 2809480
-
Warp client
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |Port 0 |Port 0: link UP, speed 10Gbps, duplex full(manual), TX: 19.09% RX: 19.09% |TcId Type Criteria State Runtime Quick stats |============================================================================================================================ |================================================================================== | CL s/s Established Closed Data | 0 TCP CL Run Time: 600s RUNNING 157.20s Run Time: 157s | TCP/UDP 0 0 1407040 | | | |============================================================================================================================ | |TCP SM Stats | |============================================================================================================================ | |INIT : 0 LISTEN : 0 | |SYN_SENT : 0 SYN_RECV : 0 | |ESTAB : 944000 FIN_WAIT_1 : 0 | |FIN_WAIT_2 : 0 LAST_ACK : 0 | |CLOSING : 0 TIME_WAIT : 0 | |CLOSE_WAIT : 0 CLOSED : 0 | | | |============================================================================================================================ | |IP Stats | |============================================================================================================================ | |Rx Pkts : 431074214 Invalid Cksum : 0 | |Rx Bytes : 30975894896 Small Mbuf : 0 | |Rx ICMP : 0 Small Hdr : 0 | |Rx TCP : 431074214 Invalid Len : 0 |-----------------------------------------------------------------------------------|Rx UDP : 0 Rx Frags : 0 |Port 0, Test Case 0 |Rx Other : 0 | |Invalid Ver : 0 Res Bit : 0 |Statistics: | | Estab/s Closed/s Data Send/s |============================================================================================================================ | 0 0 1407040 |Link Stats | Requests Replies |============================================================================================================================ | 214939234 213995246 | Link Rate SW | | Rx Pkts 431073821 2810955 431074234 | | Rx Bytes 38309881648 250175093 38309918532 | | Tx Pkts 432033873 2810950 432034182 | | Tx Bytes 38422304376 250174444 37123258996 | | Rx Err 0 0 N/A | | Tx Err 0 0 0 | |Rx No Mbuf 0 0 N/A | |
-
Warp server
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |Port 0 |Port 0: link UP, speed 10Gbps, duplex full(manual), TX: 19.08 |Port 1: link DOWN, speed 0Mbps, duplex N/A(manual), TX: 0.00% |TcId Type Criteria State Runtime Quick stats |============================================================= |============================================================= |================================================================================== | CL s/s Established Closed Data | CL s/s Established Closed Data | 0 TCP SRV Srv Up: 1 PASSED 180.94s Srv Up: 1 | TCP/UDP 0 0 1403773 | TCP/UDP 0 0 0 |Port 1 | | |TcId Type Criteria State Runtime Quick stats |============================================================= |============================================================= |================================================================================== |TCP SM Stats |TCP SM Stats | |============================================================= |============================================================= | |INIT : 0 LISTEN : |INIT : 0 LISTEN : | |SYN_SENT : 0 SYN_RECV : |SYN_SENT : 0 SYN_RECV : | |ESTAB : 944000 FIN_WAIT_1 : |ESTAB : 0 FIN_WAIT_1 : | |FIN_WAIT_2 : 0 LAST_ACK : |FIN_WAIT_2 : 0 LAST_ACK : | |CLOSING : 0 TIME_WAIT : |CLOSING : 0 TIME_WAIT : | |CLOSE_WAIT : 0 CLOSED : |CLOSE_WAIT : 0 CLOSED : | | 0 | 0 | |============================================================= |============================================================= | |IP Stats |IP Stats | |============================================================= |============================================================= | |Rx Pkts : 507430261 Invalid Cksum : 0 |Rx Pkts : 0 Invalid Cksum : 0 | |Rx Bytes : 36503736456 Small Mbuf : 0 |Rx Bytes : 0 Small Mbuf : 0 | |Rx ICMP : 0 Small Hdr : 0 |Rx ICMP : 0 Small Hdr : 0 | |Rx TCP : 507430261 Invalid Len : 0 |Rx TCP : 0 Invalid Len : 0 |-----------------------------------------------------------------------------------|Rx UDP : 0 Rx Frags : 0 |Rx UDP : 0 Rx Frags : 0 |Port 0, Test Case 0 |Rx Other : 0 |Rx Other : 0 | |Invalid Ver : 0 Res Bit : 0 |Invalid Ver : 0 Res Bit : 0 |Statistics: | | | Estab/s Closed/s Data Send/s |============================================================= |============================================================= | 0 0 1403773 |Link Stats |Link Stats | Requests Replies |============================================================= |============================================================= | 253088595 252144654 | Link Rate SW | Link Rate SW | | Rx Pkts 507429788 2810313 507430262 | Rx Pkts 0 0 0 | | Rx Bytes 45132938602 250117941 45132979922 | Rx Bytes 0 0 0 | | Tx Pkts 506532071 2810313 506532396 | Tx Pkts 0 0 0 | | Tx Bytes 45024694482 250117942 43499265892 | Tx Bytes 0 0 0 | | Rx Err 0 0 N/A | Rx Err 0 0 N/A | | Tx Err 0 0 0 | Tx Err 0 0 0 | |Rx No Mbuf 0 0 N/A |Rx No Mbuf 0 0 N/A | | |
-
Port 26
DGS-1510-28X/ME:admin#show packet ports 26 Command: show packet ports 26 Port Number : 26 ================================================================== Frame Size/Type Frame Counts Frames/sec --------------- ---------------------- ----------- 64 1677113329 0 65-127 1675070297 0 128-255 0 0 256-511 0 0 512-1023 0 0 1024-1518 0 0 Unicast RX 1676097213 0 Multicast RX 0 0 Broadcast RX 5 0 Frame Type Total Total/sec --------------- ---------------------- ----------- RX Bytes 155810310186 0 RX Frames 1676097218 0 TX Bytes 155883519104 0 TX Frames 1676086408 0
-
Port 27
DGS-1510-28X/ME:admin#show packet ports 27 Command: show packet ports 27 Port Number : 27 ================================================================== Frame Size/Type Frame Counts Frames/sec --------------- ---------------------- ----------- 64 1677132153 0 65-127 1674906127 0 128-255 0 0 256-511 0 0 512-1023 0 0 1024-1518 0 0 Unicast RX 1675841274 0 Multicast RX 0 0 Broadcast RX 138 0 Frame Type Total Total/sec --------------- ---------------------- ----------- RX Bytes 155856494618 0 RX Frames 1675841412 0 TX Bytes 155818510668 0 TX Frames 1676196868 0
-
Warp client
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |Port 0 |Port 0: link UP, speed 10Gbps, duplex full(manual), TX: 0.00% RX: 0.00%6% |TcId Type Criteria State Runtime Quick stats |============================================================================================================================ |================================================================================== | CL s/s Established Closed Data | 0 TCP CL Run Time: 600s PASSED 600.36s Run Time: 600s | TCP/UDP 0 0 0 | | | |============================================================================================================================ | |TCP SM Stats | |============================================================================================================================ | |INIT : 0 LISTEN : 0 | |SYN_SENT : 0 SYN_RECV : 0 | |ESTAB : 0 FIN_WAIT_1 : 0 | |FIN_WAIT_2 : 0 LAST_ACK : 0 | |CLOSING : 0 TIME_WAIT : 0 | |CLOSE_WAIT : 0 CLOSED : 0 | | | |============================================================================================================================ | |IP Stats | |============================================================================================================================ | |Rx Pkts : 1675196132 Invalid Cksum : 0 | |Rx Bytes : 120597080096 Small Mbuf : 0 | |Rx ICMP : 0 Small Hdr : 0 | |Rx TCP : 1675196132 Invalid Len : 0 |-----------------------------------------------------------------------------------|Rx UDP : 0 Rx Frags : 0 |Port 0, Test Case 0 |Rx Other : 0 | |Invalid Ver : 0 Res Bit : 0 |Statistics: | | Estab/s Closed/s Data Send/s |============================================================================================================================ | 0 0 0 |Link Stats | Requests Replies |============================================================================================================================ | 836880606 835936606 | Link Rate SW | | Rx Pkts 1675196270 0 1675196271 | | Rx Bytes 149077020252 0 149077020312 | | Tx Pkts 1676196843 0 1676196868 | | Tx Bytes 149113721696 0 144078117044 | | Rx Err 0 0 N/A | | Tx Err 0 0 0 | |Rx No Mbuf 0 0 N/A
-
Warp server
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |Port 0 |Port 0: link UP, speed 10Gbps, duplex full(manual), TX: 0.00% RX: 0. |Port 1: link DOWN, speed 0Mbps, duplex N/A(manual), TX: 0.00% RX: 0. |TcId Type Criteria State Runtime Quick stats |==================================================================== |==================================================================== |============================================================================================ | CL s/s Established Closed Data | CL s/s Established Closed Data | 0 TCP SRV Srv Up: 1 PASSED 639.80s Srv Up: 1 | TCP/UDP 0 0 0 | TCP/UDP 0 0 0 |Port 1 | | |TcId Type Criteria State Runtime Quick stats |==================================================================== |==================================================================== |============================================================================================ |TCP SM Stats |TCP SM Stats | |==================================================================== |==================================================================== | |INIT : 0 LISTEN : 2 |INIT : 0 LISTEN : 0 | |SYN_SENT : 0 SYN_RECV : 0 |SYN_SENT : 0 SYN_RECV : 0 | |ESTAB : 0 FIN_WAIT_1 : 0 |ESTAB : 0 FIN_WAIT_1 : 0 | |FIN_WAIT_2 : 0 LAST_ACK : 0 |FIN_WAIT_2 : 0 LAST_ACK : 0 | |CLOSING : 0 TIME_WAIT : 0 |CLOSING : 0 TIME_WAIT : 0 | |CLOSE_WAIT : 0 CLOSED : 0 |CLOSE_WAIT : 0 CLOSED : 0 | | | | |==================================================================== |==================================================================== | |IP Stats |IP Stats | |==================================================================== |==================================================================== | |Rx Pkts : 1675674304 Invalid Cksum : 0 |Rx Pkts : 0 Invalid Cksum : 0 | |Rx Bytes : 120588448832 Small Mbuf : 0 |Rx Bytes : 0 Small Mbuf : 0 | |Rx ICMP : 0 Small Hdr : 0 |Rx ICMP : 0 Small Hdr : 0 | |Rx TCP : 1675674304 Invalid Len : 0 |Rx TCP : 0 Invalid Len : 0 | |Rx UDP : 0 Rx Frags : 0 |Rx UDP : 0 Rx Frags : 0 |---------------------------------------------------------------------------------------------|Rx Other : 0 |Rx Other : 0 |Port 0, Test Case 0 |Invalid Ver : 0 Res Bit : 0 |Invalid Ver : 0 Res Bit : 0 | | | |Statistics: |==================================================================== |==================================================================== | Estab/s Closed/s Data Send/s |Link Stats |Link Stats | 0 0 0 |==================================================================== |==================================================================== | Requests Replies | Link Rate SW | Link Rate SW | 836881590 836881259 | Rx Pkts 1675674309 0 1675674310 | Rx Pkts 0 0 0 | | Rx Bytes 149080546774 0 149080546834 | Rx Bytes 0 0 0 | | Tx Pkts 1676086405 0 1676086408 | Tx Pkts 0 0 0 | | Tx Bytes 149179173292 0 144151688272 | Tx Bytes 0 0 0 | | Rx Err 0 0 N/A | Rx Err 0 0 N/A | | Tx Err 0 0 0 | Tx Err 0 0 0 | |Rx No Mbuf 0 0 N/A |Rx No Mbuf 0 0 N/A | | |
The router is able to perform source NAT function for traffic rate of 5.4 Mpps for 0.9M simultaneous tcp connections.