NAS upgrade
NAS
- Login
sudo synogear installiperf3 -s -p 5201
Workstation
- Login
iperf3 -c 192.168.129.3 -p 5201
Plan
NAS
- LAN: 192.168.129.3/24
- P2P: 192.168.130.9/30
Titan
- LAN: 192.168.129.217/24
- P2P: 192.168.130.10/30
Hardware
USB-C 10Gb NIC
enp4s0f4u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 34:c8:d6:b3:08:ac brd ff:ff:ff:ff:ff:ff
altname enx34c8d6b308ac
Baseline
Titan: 1Gb port / 1Gb link NAS: 1Gb port / 1Gb link
$ iperf3 -c 192.168.129.3 -p 5201
Connecting to host 192.168.129.3, port 5201
[ 4] local 192.168.129.217 port 35646 connected to 192.168.129.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 955 Mbits/sec 0 355 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 0 355 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 940 Mbits/sec 0 386 KBytes
[ 4] 3.00-4.00 sec 113 MBytes 946 Mbits/sec 0 386 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 404 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 404 KBytes
[ 4] 6.00-7.00 sec 112 MBytes 936 Mbits/sec 0 424 KBytes
[ 4] 7.00-8.00 sec 113 MBytes 947 Mbits/sec 0 424 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 937 Mbits/sec 0 424 KBytes
[ 4sa] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 424 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
Use USB 10Gb on Titan
Titan: 10Gb port / 1Gb link NAS: 1Gb port / 1Gb link
$ iperf3 -c 192.168.129.3 -p 5201
Connecting to host 192.168.129.3, port 5201
[ 4] local 192.168.129.231 port 37496 connected to 192.168.129.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 419 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 937 Mbits/sec 0 419 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 0 440 KBytes
[ 4] 3.00-4.00 sec 112 MBytes 943 Mbits/sec 0 440 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 0 440 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 937 Mbits/sec 0 440 KBytes
[ 4] 6.00-7.00 sec 113 MBytes 945 Mbits/sec 0 440 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 440 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 440 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 937 Mbits/sec 0 440 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
Install 10Gb on NAS
Titan: 10Gb port / 1Gb link NAS: 10Gb port / 1Gb link
$ iperf3 -c 192.168.130.9 -p 5201
Connecting to host 192.168.130.9, port 5201
[ 4] local 192.168.130.10 port 51120 connected to 192.168.130.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 4] 0.00-1.00 sec 564 MBytes 4.73 Gbits/sec 0 1.16 MBytes
[ 4] 1.00-2.00 sec 561 MBytes 4.70 Gbits/sec 0 1.16 MBytes
[ 4] 2.00-3.00 sec 561 MBytes 4.71 Gbits/sec 0 1.23 MBytes
[ 4] 3.00-4.00 sec 562 MBytes 4.71 Gbits/sec 0 1.23 MBytes
[ 4] 4.00-5.00 sec 561 MBytes 4.71 Gbits/sec 0 1.23 MBytes
[ 4] 5.00-6.00 sec 561 MBytes 4.71 Gbits/sec 0 1.23 MBytes
[ 4] 6.00-7.00 sec 561 MBytes 4.71 Gbits/sec 0 1.29 MBytes
[ 4] 7.00-8.00 sec 560 MBytes 4.70 Gbits/sec 0 1.29 MBytes
[ 4] 8.00-9.00 sec 562 MBytes 4.71 Gbits/sec 0 1.29 MBytes
[ 4] 9.00-10.00 sec 561 MBytes 4.70 Gbits/sec 0 1.29 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 4] 0.00-10.00 sec 5.48 GBytes 4.71 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 5.48 GBytes 4.71 Gbits/sec receiver
iperf Done.
File performance testing
Basic rsync from local NVME to $HOME/tmp/
- Over 1Gb NICs
$ rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ && sync
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 161.83MB/s 0:00:38 (xfr#1, to-chk=0/1)
- Over 10Gb NICs
$ rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ && sync
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 225.15MB/s 0:00:27 (xfr#1, to-chk=0/1)
Performance testing
Trying multiple iperf3 connections
Two connections
$ iperf3 -P 2 -p 5201 -c nas
Connecting to host nas, port 5201
[ 4] local 192.168.130.10 port 36980 connected to 192.168.130.9 port 5201
[ 6] local 192.168.130.10 port 36994 connected to 192.168.130.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 4] 0.00-1.00 sec 284 MBytes 2.38 Gbits/sec 0 559 KBytes
[ 6] 0.00-1.00 sec 282 MBytes 2.37 Gbits/sec 0 536 KBytes
[SUM] 0.00-1.00 sec 566 MBytes 4.75 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec 0 559 KBytes
[ 6] 1.00-2.00 sec 282 MBytes 2.36 Gbits/sec 0 564 KBytes
[SUM] 1.00-2.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec 0 559 KBytes
[ 6] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec 0 564 KBytes
[SUM] 2.00-3.00 sec 560 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec 0 559 KBytes
[ 6] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec 0 594 KBytes
[SUM] 3.00-4.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec 0 559 KBytes
[ 6] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 0 594 KBytes
[SUM] 4.00-5.00 sec 561 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 559 KBytes
[ 6] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 622 KBytes
[SUM] 5.00-6.00 sec 560 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec 0 559 KBytes
[ 6] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 622 KBytes
[SUM] 6.00-7.00 sec 561 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec 0 629 KBytes
[ 6] 7.00-8.00 sec 281 MBytes 2.36 Gbits/sec 0 622 KBytes
[SUM] 7.00-8.00 sec 561 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec 0 629 KBytes
[ 6] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 0 622 KBytes
[SUM] 8.00-9.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec 0 629 KBytes
[ 6] 9.00-10.00 sec 282 MBytes 2.36 Gbits/sec 0 622 KBytes
[SUM] 9.00-10.00 sec 561 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 4] 0.00-10.00 sec 2.74 GBytes 2.36 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver
[ 6] 0.00-10.00 sec 2.74 GBytes 2.36 Gbits/sec 0 sender
[ 6] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver
[SUM] 0.00-10.00 sec 5.49 GBytes 4.71 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 5.48 GBytes 4.71 Gbits/sec receiver
iperf Done.
Result: Same ~4.7 Gbits/sec performance.
Maybe it needs more cores/threads to see a difference?
Four connections
$ iperf3 -P 4 -p 5201 -c nas
Connecting to host nas, port 5201
[ 4] local 192.168.130.10 port 50562 connected to 192.168.130.9 port 5201
[ 6] local 192.168.130.10 port 50578 connected to 192.168.130.9 port 5201
[ 8] local 192.168.130.10 port 50590 connected to 192.168.130.9 port 5201
[ 10] local 192.168.130.10 port 50596 connected to 192.168.130.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 4] 0.00-1.00 sec 141 MBytes 1.18 Gbits/sec 0 324 KBytes
[ 6] 0.00-1.00 sec 142 MBytes 1.19 Gbits/sec 0 322 KBytes
[ 8] 0.00-1.00 sec 142 MBytes 1.19 Gbits/sec 0 321 KBytes
[ 10] 0.00-1.00 sec 142 MBytes 1.19 Gbits/sec 0 307 KBytes
[SUM] 0.00-1.00 sec 567 MBytes 4.75 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 140 MBytes 1.18 Gbits/sec 0 324 KBytes
[ 6] 1.00-2.00 sec 140 MBytes 1.18 Gbits/sec 0 322 KBytes
[ 8] 1.00-2.00 sec 140 MBytes 1.18 Gbits/sec 0 321 KBytes
[ 10] 1.00-2.00 sec 140 MBytes 1.17 Gbits/sec 0 322 KBytes
[SUM] 1.00-2.00 sec 560 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 140 MBytes 1.18 Gbits/sec 0 324 KBytes
[ 6] 2.00-3.00 sec 140 MBytes 1.18 Gbits/sec 0 338 KBytes
[ 8] 2.00-3.00 sec 141 MBytes 1.18 Gbits/sec 0 321 KBytes
[ 10] 2.00-3.00 sec 141 MBytes 1.18 Gbits/sec 0 322 KBytes
[SUM] 2.00-3.00 sec 562 MBytes 4.72 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 141 MBytes 1.18 Gbits/sec 0 341 KBytes
[ 6] 3.00-4.00 sec 140 MBytes 1.18 Gbits/sec 0 356 KBytes
[ 8] 3.00-4.00 sec 140 MBytes 1.18 Gbits/sec 0 355 KBytes
[ 10] 3.00-4.00 sec 140 MBytes 1.17 Gbits/sec 0 338 KBytes
[SUM] 3.00-4.00 sec 561 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 140 MBytes 1.17 Gbits/sec 0 341 KBytes
[ 6] 4.00-5.00 sec 141 MBytes 1.18 Gbits/sec 0 356 KBytes
[ 8] 4.00-5.00 sec 141 MBytes 1.18 Gbits/sec 0 355 KBytes
[ 10] 4.00-5.00 sec 140 MBytes 1.18 Gbits/sec 0 338 KBytes
[SUM] 4.00-5.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 140 MBytes 1.18 Gbits/sec 0 341 KBytes
[ 6] 5.00-6.00 sec 140 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 8] 5.00-6.00 sec 140 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 10] 5.00-6.00 sec 141 MBytes 1.18 Gbits/sec 0 338 KBytes
[SUM] 5.00-6.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 140 MBytes 1.17 Gbits/sec 0 341 KBytes
[ 6] 6.00-7.00 sec 140 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 8] 6.00-7.00 sec 140 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 10] 6.00-7.00 sec 140 MBytes 1.18 Gbits/sec 0 338 KBytes
[SUM] 6.00-7.00 sec 560 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 140 MBytes 1.18 Gbits/sec 0 341 KBytes
[ 6] 7.00-8.00 sec 140 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 8] 7.00-8.00 sec 140 MBytes 1.17 Gbits/sec 0 373 KBytes
[ 10] 7.00-8.00 sec 140 MBytes 1.18 Gbits/sec 0 338 KBytes
[SUM] 7.00-8.00 sec 561 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 140 MBytes 1.18 Gbits/sec 0 341 KBytes
[ 6] 8.00-9.00 sec 141 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 8] 8.00-9.00 sec 141 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 10] 8.00-9.00 sec 141 MBytes 1.18 Gbits/sec 0 338 KBytes
[SUM] 8.00-9.00 sec 562 MBytes 4.72 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 141 MBytes 1.18 Gbits/sec 0 434 KBytes
[ 6] 9.00-10.00 sec 140 MBytes 1.18 Gbits/sec 0 373 KBytes
[ 8] 9.00-10.00 sec 140 MBytes 1.17 Gbits/sec 0 373 KBytes
[ 10] 9.00-10.00 sec 140 MBytes 1.18 Gbits/sec 0 338 KBytes
[SUM] 9.00-10.00 sec 561 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 4] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec receiver
[ 6] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec 0 sender
[ 6] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec receiver
[ 8] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec 0 sender
[ 8] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec receiver
[ 10] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec 0 sender
[ 10] 0.00-10.00 sec 1.37 GBytes 1.18 Gbits/sec receiver
[SUM] 0.00-10.00 sec 5.49 GBytes 4.71 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 5.48 GBytes 4.71 Gbits/sec receiver
iperf Done.
Result: Same ~4.7 Gbits/sec performance.
Nope. But we'll give more threads a test...
(Insert obligitory "MOAR!!!" image here...)
Ten connections
$ iperf3 -P 10 -p 5201 -c nas
Connecting to host nas, port 5201
[ 4] local 192.168.130.10 port 48174 connected to 192.168.130.9 port 5201
[ 6] local 192.168.130.10 port 48188 connected to 192.168.130.9 port 5201
[ 8] local 192.168.130.10 port 48204 connected to 192.168.130.9 port 5201
[ 10] local 192.168.130.10 port 48208 connected to 192.168.130.9 port 5201
[ 12] local 192.168.130.10 port 48212 connected to 192.168.130.9 port 5201
[ 14] local 192.168.130.10 port 48224 connected to 192.168.130.9 port 5201
[ 16] local 192.168.130.10 port 48228 connected to 192.168.130.9 port 5201
[ 18] local 192.168.130.10 port 48230 connected to 192.168.130.9 port 5201
[ 20] local 192.168.130.10 port 48242 connected to 192.168.130.9 port 5201
[ 22] local 192.168.130.10 port 48250 connected to 192.168.130.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 4] 0.00-1.00 sec 56.8 MBytes 476 Mbits/sec 0 161 KBytes
[ 6] 0.00-1.00 sec 56.9 MBytes 477 Mbits/sec 0 161 KBytes
[ 8] 0.00-1.00 sec 56.8 MBytes 476 Mbits/sec 0 144 KBytes
[ 10] 0.00-1.00 sec 56.9 MBytes 477 Mbits/sec 0 161 KBytes
[ 12] 0.00-1.00 sec 56.8 MBytes 476 Mbits/sec 0 161 KBytes
[ 14] 0.00-1.00 sec 56.8 MBytes 476 Mbits/sec 0 161 KBytes
[ 16] 0.00-1.00 sec 56.9 MBytes 477 Mbits/sec 0 161 KBytes
[ 18] 0.00-1.00 sec 56.6 MBytes 475 Mbits/sec 0 156 KBytes
[ 20] 0.00-1.00 sec 56.8 MBytes 476 Mbits/sec 0 163 KBytes
[ 22] 0.00-1.00 sec 56.9 MBytes 477 Mbits/sec 0 161 KBytes
[SUM] 0.00-1.00 sec 568 MBytes 4.76 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 1.00-2.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 6] 1.00-2.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[ 8] 1.00-2.00 sec 56.0 MBytes 470 Mbits/sec 0 144 KBytes
[ 10] 1.00-2.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[ 12] 1.00-2.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[ 14] 1.00-2.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 16] 1.00-2.00 sec 56.0 MBytes 470 Mbits/sec 0 170 KBytes
[ 18] 1.00-2.00 sec 56.2 MBytes 472 Mbits/sec 0 156 KBytes
[ 20] 1.00-2.00 sec 56.1 MBytes 471 Mbits/sec 0 163 KBytes
[ 22] 1.00-2.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[SUM] 1.00-2.00 sec 561 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 2.00-3.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[ 6] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 8] 2.00-3.00 sec 56.0 MBytes 470 Mbits/sec 0 144 KBytes
[ 10] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 12] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 14] 2.00-3.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[ 16] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 0 170 KBytes
[ 18] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 0 156 KBytes
[ 20] 2.00-3.00 sec 56.1 MBytes 471 Mbits/sec 0 163 KBytes
[ 22] 2.00-3.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[SUM] 2.00-3.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[ 6] 3.00-4.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 8] 3.00-4.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 10] 3.00-4.00 sec 56.0 MBytes 470 Mbits/sec 0 170 KBytes
[ 12] 3.00-4.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[ 14] 3.00-4.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 16] 3.00-4.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 18] 3.00-4.00 sec 55.9 MBytes 469 Mbits/sec 0 156 KBytes
[ 20] 3.00-4.00 sec 56.2 MBytes 472 Mbits/sec 0 163 KBytes
[ 22] 3.00-4.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[SUM] 3.00-4.00 sec 561 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 4.00-5.00 sec 56.2 MBytes 471 Mbits/sec 0 161 KBytes
[ 6] 4.00-5.00 sec 55.9 MBytes 468 Mbits/sec 0 161 KBytes
[ 8] 4.00-5.00 sec 56.2 MBytes 471 Mbits/sec 0 161 KBytes
[ 10] 4.00-5.00 sec 56.2 MBytes 471 Mbits/sec 0 170 KBytes
[ 12] 4.00-5.00 sec 56.2 MBytes 471 Mbits/sec 0 161 KBytes
[ 14] 4.00-5.00 sec 56.1 MBytes 470 Mbits/sec 0 170 KBytes
[ 16] 4.00-5.00 sec 56.1 MBytes 470 Mbits/sec 0 170 KBytes
[ 18] 4.00-5.00 sec 56.2 MBytes 471 Mbits/sec 0 156 KBytes
[ 20] 4.00-5.00 sec 56.1 MBytes 470 Mbits/sec 0 163 KBytes
[ 22] 4.00-5.00 sec 56.0 MBytes 469 Mbits/sec 0 161 KBytes
[SUM] 4.00-5.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 5.00-6.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 6] 5.00-6.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 8] 5.00-6.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 10] 5.00-6.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 12] 5.00-6.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 14] 5.00-6.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 16] 5.00-6.00 sec 56.1 MBytes 471 Mbits/sec 0 177 KBytes
[ 18] 5.00-6.00 sec 56.1 MBytes 471 Mbits/sec 0 156 KBytes
[ 20] 5.00-6.00 sec 56.1 MBytes 471 Mbits/sec 0 163 KBytes
[ 22] 5.00-6.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[SUM] 5.00-6.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 6.00-7.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[ 6] 6.00-7.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[ 8] 6.00-7.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[ 10] 6.00-7.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 12] 6.00-7.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[ 14] 6.00-7.00 sec 55.9 MBytes 469 Mbits/sec 0 170 KBytes
[ 16] 6.00-7.00 sec 56.1 MBytes 471 Mbits/sec 0 177 KBytes
[ 18] 6.00-7.00 sec 56.0 MBytes 470 Mbits/sec 0 156 KBytes
[ 20] 6.00-7.00 sec 55.8 MBytes 468 Mbits/sec 0 163 KBytes
[ 22] 6.00-7.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[SUM] 6.00-7.00 sec 560 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 7.00-8.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 6] 7.00-8.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 8] 7.00-8.00 sec 56.0 MBytes 470 Mbits/sec 0 161 KBytes
[ 10] 7.00-8.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 12] 7.00-8.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 14] 7.00-8.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 16] 7.00-8.00 sec 56.2 MBytes 472 Mbits/sec 0 177 KBytes
[ 18] 7.00-8.00 sec 56.2 MBytes 472 Mbits/sec 0 156 KBytes
[ 20] 7.00-8.00 sec 56.2 MBytes 472 Mbits/sec 0 163 KBytes
[ 22] 7.00-8.00 sec 56.0 MBytes 470 Mbits/sec 0 161 KBytes
[SUM] 7.00-8.00 sec 562 MBytes 4.71 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 8.00-9.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 6] 8.00-9.00 sec 55.9 MBytes 469 Mbits/sec 0 161 KBytes
[ 8] 8.00-9.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[ 10] 8.00-9.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 12] 8.00-9.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[ 14] 8.00-9.00 sec 56.1 MBytes 471 Mbits/sec 0 170 KBytes
[ 16] 8.00-9.00 sec 55.8 MBytes 468 Mbits/sec 0 177 KBytes
[ 18] 8.00-9.00 sec 56.1 MBytes 471 Mbits/sec 0 156 KBytes
[ 20] 8.00-9.00 sec 56.1 MBytes 471 Mbits/sec 0 163 KBytes
[ 22] 8.00-9.00 sec 56.1 MBytes 471 Mbits/sec 0 161 KBytes
[SUM] 8.00-9.00 sec 561 MBytes 4.70 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 9.00-10.00 sec 55.9 MBytes 468 Mbits/sec 0 161 KBytes
[ 6] 9.00-10.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[ 8] 9.00-10.00 sec 56.6 MBytes 475 Mbits/sec 0 240 KBytes
[ 10] 9.00-10.00 sec 56.6 MBytes 475 Mbits/sec 0 249 KBytes
[ 12] 9.00-10.00 sec 56.6 MBytes 475 Mbits/sec 0 247 KBytes
[ 14] 9.00-10.00 sec 56.1 MBytes 470 Mbits/sec 0 170 KBytes
[ 16] 9.00-10.00 sec 56.1 MBytes 470 Mbits/sec 0 177 KBytes
[ 18] 9.00-10.00 sec 56.0 MBytes 469 Mbits/sec 0 156 KBytes
[ 20] 9.00-10.00 sec 56.1 MBytes 470 Mbits/sec 0 163 KBytes
[ 22] 9.00-10.00 sec 56.2 MBytes 472 Mbits/sec 0 161 KBytes
[SUM] 9.00-10.00 sec 563 MBytes 4.72 Gbits/sec 0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 4] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 6] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 6] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 8] 0.00-10.00 sec 562 MBytes 472 Mbits/sec 0 sender
[ 8] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 10] 0.00-10.00 sec 562 MBytes 472 Mbits/sec 0 sender
[ 10] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 12] 0.00-10.00 sec 562 MBytes 472 Mbits/sec 0 sender
[ 12] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 14] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 14] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 16] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 16] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 18] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 18] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 20] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 20] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[ 22] 0.00-10.00 sec 562 MBytes 471 Mbits/sec 0 sender
[ 22] 0.00-10.00 sec 561 MBytes 471 Mbits/sec receiver
[SUM] 0.00-10.00 sec 5.49 GBytes 4.71 Gbits/sec 0 sender
[SUM] 0.00-10.00 sec 5.48 GBytes 4.71 Gbits/sec receiver
iperf Done.
Result: Same ~4.7 Gbits/sec performance.
Trying with MTU of 9000
Both ports set to MTU=9000:
Local workstation:
$ ip address show dev enp4s0f4u1
3: enp4s0f4u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
link/ether 34:c8:d6:b3:08:ac brd ff:ff:ff:ff:ff:ff
altname enx34c8d6b308ac
inet 192.168.130.10/30 brd 192.168.130.11 scope global noprefixroute enp4s0f4u1
valid_lft forever preferred_lft forever
And on the NAS:
# # ip address show dev eth2
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc mq state DOWN group default qlen 1000
link/ether 90:09:d0:aa:83:d0 brd ff:ff:ff:ff:ff:ff
inet 192.168.130.9/30 brd 192.168.130.11 scope global eth2
valid_lft forever preferred_lft forever
$ iperf3 -p 5201 -c nas
Connecting to host nas, port 5201
[ 4] local 192.168.130.10 port 41914 connected to 192.168.130.9 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 4] 0.00-1.00 sec 507 MBytes 4.25 Gbits/sec 0 935 KBytes
[ 4] 1.00-2.00 sec 508 MBytes 4.26 Gbits/sec 0 987 KBytes
[ 4] 2.00-3.00 sec 505 MBytes 4.24 Gbits/sec 0 987 KBytes
[ 4] 3.00-4.00 sec 501 MBytes 4.20 Gbits/sec 0 1.02 MBytes
[ 4] 4.00-5.00 sec 500 MBytes 4.20 Gbits/sec 0 1.02 MBytes
[ 4] 5.00-6.00 sec 506 MBytes 4.25 Gbits/sec 0 1.07 MBytes
[ 4] 6.00-7.00 sec 502 MBytes 4.21 Gbits/sec 0 1.07 MBytes
[ 4] 7.00-8.00 sec 502 MBytes 4.21 Gbits/sec 0 1.07 MBytes
[ 4] 8.00-9.00 sec 498 MBytes 4.18 Gbits/sec 0 1.64 MBytes
[ 4] 9.00-10.00 sec 496 MBytes 4.15 Gbits/sec 0 1.64 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 4] 0.00-10.00 sec 4.91 GBytes 4.22 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 4.91 GBytes 4.21 Gbits/sec receiver
Result: Dang, a slight decrease, but possibly due to the larger overall packet size?
BUT!!! The "rsync" seems to be quite a bit faster, but also all over the place when it comes to the overall MB/s:
Write performance - local to NAS:
$ for C in $(seq 1 5) ; do rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && date && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ ; done
Sat Oct 31 02:55:27 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 808.13MB/s 0:00:07 (xfr#1, to-chk=0/1)
Sat Oct 31 02:56:02 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 440.86MB/s 0:00:14 (xfr#1, to-chk=0/1)
Sat Oct 31 02:56:57 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 680.04MB/s 0:00:09 (xfr#1, to-chk=0/1)
Sat Oct 31 02:57:46 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 733.48MB/s 0:00:08 (xfr#1, to-chk=0/1)
Sat Oct 31 02:58:37 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 650.65MB/s 0:00:09 (xfr#1, to-chk=0/1)
[dan@titan ~]$
The overall write speed comes out to 662MB/s.
Doing read performance from both locations:
Testing the local read:
$ time (for C in $(seq 1 5) ; do date ; time sha1sum /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso ; done ; echo Total time:)
Sat Oct 31 03:05:02 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso
real 0m3.271s
user 0m2.841s
sys 0m0.418s
Sat Oct 31 03:05:05 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso
real 0m3.266s
user 0m2.853s
sys 0m0.401s
Sat Oct 31 03:05:09 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso
real 0m3.269s
user 0m2.818s
sys 0m0.438s
Sat Oct 31 03:05:12 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso
real 0m3.269s
user 0m2.819s
sys 0m0.436s
Sat Oct 31 03:05:15 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso
real 0m3.282s
user 0m2.803s
sys 0m0.465s
Total time:
real 0m16.364s
user 0m14.135s
sys 0m2.164s
And now a read over NFS:
$ time (for C in $(seq 1 5) ; do date ; time sha1sum /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso ; done)
Sat Oct 31 03:03:33 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso
real 0m11.146s
user 0m3.501s
sys 0m0.894s
Sat Oct 31 03:03:44 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso
real 0m3.279s
user 0m2.858s
sys 0m0.406s
Sat Oct 31 03:03:48 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso
real 0m3.279s
user 0m2.816s
sys 0m0.449s
Sat Oct 31 03:03:51 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso
real 0m3.280s
user 0m2.807s
sys 0m0.459s
Sat Oct 31 03:03:54 PM CDT 2026
9fd52c962c2823dc3d5379b18c189b468b4d6d1b /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso
real 0m3.275s
user 0m2.841s
sys 0m0.420s
Total time:
real 0m24.267s
user 0m14.826s
sys 0m2.636s
Reading five times gives 24.267s over NFS, vs 16.364s local.
Heat problems?
I noticed when I went to re-run another test the speeds were decreasing:
$ time (for C in $(seq 1 5) ; do date ; rm -f /home/nfs/dan/tmp/ubuntu-26.04-desktop-amd64.iso && sync && rsync --progress /home/dan.local/iso/ubuntu-26.04-desktop-amd64.iso /home/nfs/dan/tmp/ ; done ; echo Total time:)
Sat Oct 31 03:10:55 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 449.69MB/s 0:00:13 (xfr#1, to-chk=0/1)
Sat Oct 31 03:11:47 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 289.55MB/s 0:00:21 (xfr#1, to-chk=0/1)
Sat Oct 31 03:12:25 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 307.83MB/s 0:00:20 (xfr#1, to-chk=0/1)
Sat Oct 31 03:13:19 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 251.13MB/s 0:00:24 (xfr#1, to-chk=0/1)
Sat Oct 31 03:14:06 PM CDT 2026
ubuntu-26.04-desktop-amd64.iso
6,518,974,464 100% 267.18MB/s 0:00:23 (xfr#1, to-chk=0/1)
Total time:
real 3m57.811s
user 0m9.038s
sys 0m25.878s
Admittedly, over the past ~10-15 minutes I've been doing a lot of big transfers and the metal case of the USB NIC is a bit warm, but not ultra hot.
I'll let this cool down and try later.