I need to do these few things.
Team the ports, set up vlans in the switch, enable jumbo frames.
The picture below illustrates what I am about to do...
 |
| Overview of the network involved in the teaming |
Teaming
Let's do the teaming in SG0684 first...
nmcli connection add type team con-name team0 ifname team0 \
config '{ "runner": {"name": "roundrobin"}, "link_watch": {"name": "ethtool"}}'
nmcli con mod team0 ipv4.addresses 10.8.8.204/24
nmcli con mod team0 ipv4.method manual
nmcli con add type team-slave con-name team0-slave1 ifname p5p1 master team0
nmcli con add type team-slave con-name team0-slave2 ifname p5p2 master team0
nmcli con add type team-slave con-name team0-slave3 ifname p5p3 master team0
nmcli con add type team-slave con-name team0-slave4 ifname p5p4 master team0
nmcli connection down team0
nmcli connection up team0
ip link set mtu 9000 dev team0When it is all done up, we can use the command...
teamdctl -v team0 state view
...to make sure that it looks OK. If you want to see what it looks like...
 |
| Output of "teamdctl -v team0 state view" |
Similarly in SG0685...
nmcli connection add type team con-name team0 ifname team0 \
config '{ "runner": {"name": "roundrobin"}, "link_watch": {"name": "ethtool"}}'
nmcli con mod team0 ipv4.addresses 10.8.8.205/24
nmcli con mod team0 ipv4.method manual
nmcli con add type team-slave con-name team0-slave1 ifname p5p1 master team0
nmcli con add type team-slave con-name team0-slave2 ifname p5p2 master team0
nmcli con add type team-slave con-name team0-slave3 ifname p5p3 master team0
nmcli con add type team-slave con-name team0-slave4 ifname p5p4 master team0
nmcli connection down team0
nmcli connection up team0
ip link set mtu 9000 dev team0
VLANs
Why do we need to set up VLANs in the switch? When the ports are teamed, a single MAC address will be used on all the ports. The switch needs to be configured so that it is not confused by the same MAC appearing in more than 1 port. Since there are 4 ports per team, we will need to create 4 VLANs. (We can so the same thing by installing 4 switches, but since we have only 1 switch, we will use VLANs to create 4 switches, so to speak.) The next 2 pictures illustrate what I did for my switch. (The MAC addresses are auto-detected by the switch. I manually set the VLAN IDs.)
 |
| Duplicate MAC addresses as seen by switch. Use VLAN to manage. |
 |
| MAC address from the other server's teamed device. Use VLAN to manage. |
 |
| Using 4 VLANs to create the effect of 4 switches. |
Jumbo Frames
We also need to enable jumbo frames. For the servers, the command...
ip link set mtu 9000 dev team0
... (which we already issued above) enables the jumbo frames. For the switch, it varies from switch to switch. The documentation of the switch should be consulted.
We can use the command...
ping -c 4 -s 8972 10.8.8.205
... to test whether the jumbo frames are properly set up. The -s 8972 flag sets the packet size. If jumbo frames are done up correctly in all the devices, the ping should go through and get a reply, like this...
 |
| Using ping to ensure jumbo frames are set up correctly. |
Tests
We are now ready to do some file transfer tests.
Source: 50GB file "50GBfile.dat" in SG0684.
Destination: SG0685 (10.8.8.205), folder "/ExtraSpace/share".
Test 1: rsync over SSH
Result: 1.202 Gb/sDetails...
[root@SG0684 temp]# rsync --stats --progress -e 'ssh' 50GBfile.dat root@10.8.8.205:/ExtraSpace/share
root@10.8.8.205's password:
50GBfile.dat
52,428,800,000 100% 150.29MB/s 0:05:32 (xfr#1, to-chk=0/1)
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 52,428,800,000 bytes
Total transferred file size: 52,428,800,000 bytes
Literal data: 52,428,800,000 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 52,441,600,084
Total bytes received: 35
sent 52,441,600,084 bytes received 35 bytes 155,844,279.70 bytes/sec
total size is 52,428,800,000 speedup is 1.00
[root@SG0684 temp]#... 150.29MB/s = 1.202 Gb/s. Not very impressive. (I suspect the ssh / encryption might have created some overhead which slowed the transfer speed somehow.)
Test 2: rsync over NFS
Result: 2.428 Gb/sDetails...
In this test, we NFS mount the space in the destination server on the source server. We do something like this in the source server...
mount -t nfs 10.8.8.205:/ExtraSpace/share /SG0685/share
... then do the rsync. Result...
[root@SG0684 temp]# rsync --stats --progress 50GBfile.dat /SG0685/share
50GBfile.dat
52,428,800,000 100% 303.50MB/s 0:02:44 (xfr#1, to-chk=0/1)
Number of files: 1 (reg: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 52,428,800,000 bytes
Total transferred file size: 52,428,800,000 bytes
Literal data: 52,428,800,000 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 52,441,600,084
Total bytes received: 35
sent 52,441,600,084 bytes received 35 bytes 309,389,971.20 bytes/sec
total size is 52,428,800,000 speedup is 1.00
[root@SG0684 temp]#.... 303.50MB/s = 2.428 Gb/s. Better.
Test 3: cp over NFS
Result: 2.741 Gb/sDetails...
[root@SG0684 temp]# date ; cp 50GBfile.dat /SG0685/share/ ; date
Sat Aug 3 12:41:35 +08 2019
Sat Aug 3 12:53:09 +08 2019
[root@SG0684 temp]#... (52428800000 bytes)/(2 minutes 33 seconds) = 2.741 Gb/s. Almost the same as rsync over NFS.
Test 4: Netcat
Result: 3.9548 Gb/sDetails...
It may not be practical to use netcat in a full backup scenario, but it does show in theory, what speeds can be achieved, without all the extra overhead like encryption, shared filesystem protocols, etc. Here's how it can be done.
Firstly, at the destination server, we cd to the folder that we want the file to be copied to...
cd /ExtraSpace/share
... then we set up the netcat (nc) in "listen" mode...
nc -l 7000 | pv | tar -xpf -
After the destination server is set up, we go the source server, and use netcat to transfer the file to the destination server.
tar -cf - * | pv | nc 10.8.8.205 7000
The video below shows the process...
From the video above, we get the speed...
48.8GiB / (1min 46seconds)
= 3.9548 Gb/s
In conclusion, Linux teaming can increased throughput. But it requires configuration at the switch. The method used in transferring the data can also affect the speed that we ultimately get.
