The wide adoption of 10 to 100Gbit Ethernet technology has supported the amazing development of the media and entertainment industry and has allowed for workflows that would otherwise have been extremely hard to pull off. But in order to achieve the best possible performance from your high speed ethernet setup, efforts in installation and maintenance are required. In this blog, we will go over the things that you should keep in mind in order to achieve maximum performance with a 10Gbit Ethernet connection.
Do a sanity check
First thing’s first – be sure to check if all the devices in the chain between the Server and the Client are capable of 10Gbit Ethernet.
1. Check the workstations and servers network cards.
2. Check all switches / used switch ports, bonds, teams ISLs, M-LAGs etc.
3. Check all patches.
4. Check all wall sockets.
Take care of your cabling
To deliver the performance to our clients, we rely on high-performance cables, but these can be a sensitive link in the performance chain. Fibre optic cables are sensitive to stress, especially bending and need to be taken care of, and while the good old copper cable is more robust, it is also more interference-sensitive.
Fibre optic cabling
1. If you have decided on a fibre optic infrastructure, make sure to use at least OM3 quality cables. For longer distances OM4.
Maximum reach at 10Gb/s:
OM3 50µm – 300m (850nm)
OM4 50µm – 550m (850nm)
Maximum reach at 40Gb/s and 100Gb/s:
OM3 50µm – 100m (850nm)
OM4 50µm – 125m (850nm)
2. Avoid bending fibre optic cables. Bending the cable can cause damage to the fibres and can result in refraction. Every cable has an established minimum bending radius that will be printed on the cable packing.
3. Any patches should be new or in near new conditions, handle with care.
4. Upon unpacking or re-packing the cables, make sure that the cable connectors have dust caps on. New cables, in some cases, have a connection check report or an inspection sheet in the packing.
5. Inspect and clean fibre cables before use by using compressed air or isopropyl alcohol.
6. Make sure to use only the connector body for connecting and disconnecting the cable. Never pull at the cable directly.
7. Make sure that the standard cable length of patches and installed lines are not longer than standards recommend.
1. If you have decided on a copper-based infrastructure, consider using the best shielded cables available.
2. Avoid crimping cable ends on your own as this can lead to performance drop.
3. Do not lay the cables near other electrical installations. This can lead to interference and induce errors into the transmitted signal. When the quality of the cable is in doubt, it can be tested with a tool like the Fluke DSP-2000 CableAnalyzer.
4. Avoid long distances. Don’t exceed 100m cable length with Cat.7 cables. When it comes to 40Gbit cabling, they are best used for interconnectivity of the storage devices. Direct attached copper cables are limited to 7m.
Maximum reach 10GBASE-T Copper, Twisted Pair:
Cat. 5e – 50m
Cat. 6a/7 – 100m
5. Cables without connector clips should not be used. Even if you fasten the connector with glue or tape, they will eventually come loose.
Use the right PCIe slot
1. Make sure that the Network Interface Card is in the right slot. Network adapter Cards require a certain throughput and need to be placed in the right PCIe Slot on the motherboard. If you have a multi-CPU workstation, be aware that the PCIe slot load is divided between the CPUs. The same goes for the RAM slots.
2. Read the manual of the board, sometimes the PCIe slots are not fully loaded with lanes – This means that a physical PCI-e x16 slot might only be electrically equipped with 4 or 8 lanes. In that case, the slot will not provide the expected throughput.
3. Thunderbolt is a mystery for several reasons: In Apples marketing, we see the receiver and transceiver speeds added together; when daisy-chaining devices, the speed must be divided by the number of devices; even some separated ports are bound to the same thunderbolt chip and thereby act as if they were daisy chained.
Check general network settings
If your entire network will support it, set packet size to jumbo frames which packs six times as much data per package. When enabling jumbo frames, one can expect improvement in performance if all network components fully support jumbo frames. If any of the network components do not fully support jumbo frames, it is recommended to use the default MTU 1500 setting. The switches, routers, and NICs from one end to the other must all support the same size of jumbo frame transmission for a successful jumbo frame communication session.
At Layer 2, not all gigabit switches support jumbo frames. Those that don’t will drop the frames with no warning. For a jumbo packet to pass through a router, both the ingress and egress interfaces must support the larger packet size. Sometimes, if the MTU isn’t supported, the data will have to be fragmented to a supported size and retransmitted, thereby reducing throughput.
Check general client settings
1. Update your macOS to at least 10.13.6. Earlier versions enable SMB signing by default, which can impede the performance of high speed for Ethernet networks. Update your macOS, or disable SMB signing. See the following Apple support link for further details.
2. To set Jumbo frames on macOS: access the System Preferences -> Network and select the 10G interface. Click the „Advanced…“ button in the lower right. In the Advanced window select the „Hardware“ tab and set „Configure:“ to „Manually“. Set MTU to „Custom“ and enter „9000“.
3. Set the mode to full-duplex, which allows the adapter to send and receive packages at the same time. To set full-duplex on macOS: go into the System Preferences > Network, and select the 10G interfaces. Click the „Advanced…“ button in the lower right. In the Advanced window select the „Hardware“ tab and set „Configure:“ to „Manually“. Set duplex to „full-duplex“.
1. To set Jumbo frames on Windows: Open the Network and Sharing Center and click Change adapter settings. Right-click the NIC for which you want to enable jumbo frames and select Properties. In the Networking tab, click the Configure button for the network adapter. In the Advanced tab select Jumbo Frame and change to 9014 Bytes. In the same settings menu change the value for Receive- and Send-Buffer-Size to be between 2 and 4GB.
2. To set full-duplex on Windows: navigate to the Device Manager. Open Properties menu of the adapter you would like to configure. Open the Link Speed tab. In the Speed and Duplex drop down menu select the appropriate speed and duplex.
3. Check that the Flow-control is enabled on both RX and TX.
4. Disable Windows Defender and/or other antivirus tools as well as the Firewall on Windows 10 Machines if possible. You can use the following registry key patch for this task.
Avoid network congestions
When combining different network speed infrastructures within the same network, it is of the utmost importance that the necessary steps are taken to avoid network congestions. When, for example, a server is connected by a 100Gbit link to the switch and a number of clients are connected using a 10Gbit connection, network congestion issues are likely to arise. The reason for this is that the server doesn’t know that the client is only connected with 10% of its own speed. The server will try to provide the data as fast as possible and the client will not be able to receive it with the same speed. Enabling Flow control on the network interfaces of the client, server and the connected switch ports will allow the client to communicate with the server to reduce the sending speed. To be on the safe side, in multi switch environments also enable Flow Control on the inter-switch linking ports. There are several mechanisms for handling network congestions from different switch vendors, but flow control is the most common.
In very big switching environments Flow Control might cause other issues. In this case it is recommended that a plan is created that shows the expected data traffic and that other mechanisms are used to avoid congestions.
1. Iperf – One of the most helpful tools for checking transmission speed between two network knots without utilising anything other than RAM and the network interfaces. Before starting the tests, make sure to have a proper infrastructure documentation in order to execute the tests without any possible enhancements.
One direction: Server to client with a TCP window size of 1024 kB:
Server side: iperf -w 1024k -s Client Side: iperf -w 1024k -c "servers IP Address"
Second direction: Client to server with a TCP window size of 1024 kB:
Client side: iperf -w 1024k -s Server Side: iperf -w 1024k -c "clients IP Address"
After executing the test run, turn on the enhancements and redo both tests, one in each direction. It is usually a good idea to do just one enhancement setting at a time and repeat the test before moving on to the next setting.
2. Iftop – A tool that lets you see traffic flows on a network. It is recommended to run it on a dedicated network interface and with an upper bandwidth limit.
Example iftop command for a 10G Interface:
iftop -i "NIC Name" -m 10G
Be sure to know what you want to measure and choose the right testing method for it. If you wish to measure pure network performance, use a tool like iperf. If you have special workload needs, use tools that emulate these kinds of workloads.
Check to not measure disk performance using file-read/write based tests. A 10G line can easily maxout the performance of underlying disk systems.