iSCSI performace

[timansky]

Currently purestorage connected with 4x25Gbps links with different networks/vlans

• Prod: Proxmox nodes connected with 2x10Gbps in alb bond. (MB: X10DRFF, CPU: Xeon CPU E5-2695 "72 core", RAM: 512Gb)
• Test: Proxmox Connected with 2x1Gbps in alb bond (MB: X12STE-F-001, CPU: Xeon E-2334 (8 core), RAM: 32Gb)
• Purestorage: FA-X20R3

Qemu setup:

• 8gb ram
• 8 cores (kvm64)
• disk flags: aio=io_uring,cache=writeback,diskard=on, iothread=1
• mount options: ext4 nobarrier,noatime
• pve-qemu-kvm = 9.0.2-4
• qemu-server = 8.3.3

iSCSI setup:

• prod:
  • nr_sessions = 8
• test:
  • nr_sessions = 1

Multipath setup:

• path_selector: "queue-length 0"
• hardware_handler: "1 alua"
• path_grouping_policy: group_by_prio
• prio: alua
• path_checker: tur
• user_friendly_names: no
• features: 0
• recheck_wwid: yes
• fast_io_fail_tmo: 10
• dev_loss_tmo: 60
• failback: immediate
• no_path_retry: fail

Average iops from fio:

fio cmd: fio -ioengine=libaio -direct=1 -sync=1 -name=test -bs=4k -iodepth=1 -rw=randwrite -runtime=600 --size=50G with 100Gb volume

Prod mounted:

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=12.9MiB/s][w=3296 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=14109: Tue Jan 21 18:53:56 2025
  write: IOPS=3508, BW=13.7MiB/s (14.4MB/s)(8223MiB/600001msec); 0 zone resets
    slat (usec): min=11, max=29312, avg=47.41, stdev=41.12
    clat (usec): min=2, max=43706, avg=235.45, stdev=150.79
     lat (usec): min=56, max=43743, avg=282.87, stdev=156.54
    clat percentiles (usec):
     |  1.00th=[  196],  5.00th=[  210], 10.00th=[  217], 20.00th=[  221],
     | 30.00th=[  223], 40.00th=[  227], 50.00th=[  229], 60.00th=[  235],
     | 70.00th=[  239], 80.00th=[  247], 90.00th=[  262], 95.00th=[  277],
     | 99.00th=[  306], 99.50th=[  318], 99.90th=[  375], 99.95th=[  898],
     | 99.99th=[ 2278]
   bw (  KiB/s): min= 9976, max=48720, per=100.00%, avg=14043.14, stdev=1361.57, samples=1199
   iops        : min= 2494, max=12180, avg=3510.70, stdev=340.41, samples=1199
  lat (usec)   : 4=0.01%, 10=0.01%, 50=0.02%, 100=0.29%, 250=82.53%
  lat (usec)   : 500=17.10%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.02%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=2.04%, sys=14.95%, ctx=2430501, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2105020,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=13.7MiB/s (14.4MB/s), 13.7MiB/s-13.7MiB/s (14.4MB/s-14.4MB/s), io=8223MiB (8622MB), run=600001-600001msec

Disk stats (read/write):
  sdb: ios=86/6667414, sectors=688/90686464, merge=0/4391432, ticks=4/364352, in_queue=364356, util=51.48%

Test mounted:

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.36
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][w=33.1MiB/s][w=8485 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=16008: Tue Jan 21 18:51:27 2025
  write: IOPS=8744, BW=34.2MiB/s (35.8MB/s)(20.0GiB/600001msec); 0 zone resets
    slat (usec): min=4, max=3495, avg=28.01, stdev=22.22
    clat (nsec): min=1130, max=14503k, avg=85142.29, stdev=34240.52
     lat (usec): min=18, max=14516, avg=113.15, stdev=42.90
    clat percentiles (usec):
     |  1.00th=[   19],  5.00th=[   73], 10.00th=[   75], 20.00th=[   77],
     | 30.00th=[   79], 40.00th=[   82], 50.00th=[   84], 60.00th=[   87],
     | 70.00th=[   90], 80.00th=[   98], 90.00th=[  103], 95.00th=[  109],
     | 99.00th=[  127], 99.50th=[  139], 99.90th=[  231], 99.95th=[  359],
     | 99.99th=[  807]
   bw (  KiB/s): min=25200, max=152936, per=100.00%, avg=34993.12, stdev=10577.13, samples=1199
   iops        : min= 6300, max=38234, avg=8748.25, stdev=2644.29, samples=1199
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=2.14%, 50=1.56%
  lat (usec)   : 100=80.19%, 250=16.01%, 500=0.07%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
  cpu          : usr=3.19%, sys=18.23%, ctx=7446288, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5246790,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=34.2MiB/s (35.8MB/s), 34.2MiB/s-34.2MiB/s (35.8MB/s-35.8MB/s), io=20.0GiB (21.5GB), run=600001-600001msec

Disk stats (read/write):
  sdb: ios=0/17655730, sectors=0/224156872, merge=0/10363881, ticks=0/368119, in_queue=368120, util=51.49%

DIrect write inside vm to disk approximately ~2.5Gbps on both
DIrect write from proxmox to disk approximately ~2.5Gbps on both

PROBLEM: Why 1Gbps server is twice faster that 10Gbps on single VM