Skip to content

Reference Switch

Jump to bottom
Yuta edited this page Aug 18, 2021 · 1 revision

Name

reference_switch

Location

hw/projects/reference_switch

IP Cores

Software

Description

The division of the hardware into modules was hinted at in the previous section. Understanding these modules is essential in making the most of the available designs. The reference projects in NetFPGA platform, including the Switch, all follow the same modular structure. This design is a pipeline where each stage is a separate module.

The CMAC subsystem on Xilinx open-nic-shell sends received packet to the input arbiter module via nf_mac_attachment. The input arbiter has three input interfaces: two from the CMAC subsystem on open-nic-shell and one from QDMA subsystem on open-nic-shell. Each input to the arbiter connects to an input queue, which is in fact a small fall-through FIFO. The simple arbiter rotates between all the input queues in a round robin manner, each time selecting a non-empty queue and writing one full packet from it to the next stage in the data-path, which is the output port lookup module.

The output port lookup module is responsible for deciding which port a packet goes out of. After that decision is made, the packet is then handed to the output queues module. The lookup module implements a simple learning CAM, implemented using Xilinx's CAM core. Packets with unknown destination MAC address are broadcasted.

Once a packet arrives to the output_queues module, it already has a marked destination (provided on a side channel - The TUSER field). According to the destination it is entered to a dedicated output queue. There are five such output queues: one per each 10G port and one to the DMA block. Note that a packet may be dropped if its output queue is full or almost full. When a packet reaches the head of its output queue, it is sent to the corresponding output port, being either CMAC subsystem or the QDMA subsystem. The output queues are arranged in an interleaved order: one physical Ethernet port, one DMA port etc. Even queues are therefore assigned to physical Ethernet ports, and odd queues are assigned to the virtual DMA ports.

The QDMA subsystem on open-nic-shell serves as a DMA engine for the reference switch design. It includes QDMA IP, a DMA engine and AXI4 Interconnect module.

Testing

  1. Make sure you clone the latest version of the NetFPGA package. Please ensure that you have the necessary packages installed. The current testing infrastructure is Python based.
$ git clone https://github.com/NetFPGA/NetFPGA-PLUS-dev.git
  1. Make sure the ${NFPLUS_FOLDER}/tools/settings.sh file is updated in your system.

Check if the variables like NFPLUS_FOLDER, NF_DESIGN_DIR, BOARD_NAME are set in the right path, e.g.:

export NF_PROJECT_NAME=reference_switch

To set up the environment variables, source both relevant settings files, e.g.:

source ~/NetFPGA-PLUS-dev/tools/settings.sh
source /tools/Xilinx/Vivado/2020.2/settings64.sh
  1. Do a make only the first time when you have a fresh repository. It is not necessary to repeat this make every time you want to build a new project unless you have made any changes to the IP cores.
[root@nf-test109 ~]# cd $NFPLUS_FOLDER 
[root@nf-test109 NetFPGA-PLUS-dev]# make

  1. Please follow the CAM instructions in the Building Your First Project page. Note: The CAM core must be generated prior to running this project.

  2. Program the FPGA with the right bitfile using xsdb (Xilinx tool).

  • If you want to run the Hardware tests with the pre-existing bitfile provided in the base repo:
[root@nf-test109 reference_switch]# cd bitfiles
[root@nf-test109 bitfiles]# xsdb

On the xsdb console, use fpga -f reference_switch.bit to program the FPGA with the bitfile. Reboot the machine.

  • If you want to create your own bitfile and run the Hardware tests:
[root@nf-test109 reference_switch]# make
[root@nf-test109 reference_switch]# cd bitfiles
[root@nf-test109 bitfiles]# xsdb

On the xsdb console, use fpga -f reference_switch.bit to program the FPGA with the bitfile. Reboot the machine.

  1. Check if the bit file is loaded using the following command.
[root@nf-test109 scripts]# lspci –vxx | grep Xilinx

If you don't see the Xilinx device, you need to reprogram the FPGA and reboot as mentioned earlier.

  1. Build the driver for the NetFPGA PLUS board.
[root@nf-test109 ]# cd ${NFPLUS_FOLDER}/sw/driver/
[root@nf-test109 ]# make
[root@nf-test109 ]# insmod open-nic-driver/onic.ko

Then run ifconfig -a to check if you are able to see the 'nfX' interfaces.

  1. Running the test

The top level file nf_test.py file can be found inside NetFPGA-PLUS-dev/tools/scripts. Tests are run using the nf_test.py command followed by the arguments indicating if it is a hardware or simulation test and what is the specific test that we would like to run. So when running the test, test mode should be specified (sim or hw). For instance:

[root@nf-test109 scripts]#./nf_test.py hw --major learning --minor sw

or

[root@nf-test109 scripts]#./nf_test.py sim --major learning --minor sw

For a complete list of arguments type ./nf_test.py --help.

You can find more information related to hardware and simulation tests here:

The test infrastructure is based on the python. You can find the tests inside the projects/{project_name}/test folder.

Testing hardware using two or more machines

To run the test, you need two machines, A and B. Let's say Machine A is equipped with NetFPGA and Machine B is equipped with a third-party 100G dual-port NIC.

Download the reference_switch bitfile from hw/projects/reference_switch/bitfiles/reference_switch.bit. Connect Machine A and Machine B using two 100G cables. Assume we use nf0 (the port farthest from the PCI Express) and nf1 on Machine A and eth1 and eth2 on Machine B. Generate packets from eth1 with given MAC addresses (let's say, dest-MAC = x and src-MAC = y). Check that all the NetFPGA physical ports send back the packet. Then, generate from eth2 packets with dest-MAC = y and src-MAC = x. You will be able to see that only nf0 will forward back packets.

Clone this wiki locally