On 5/12/23 09:41, Roger Heflin wrote:
I have work experience with Intel 10Gbit, Older Emulex 10GB (be[23]net driver), current Broadcom 10/25G and Mellanox 10/25G. Note that for it to be useful you*MUST* have multiple interrupts. 1Gbit interfaces used to hit a limit at around 50Mbit (the cpu was not fast enough to do the required data moves on a single core to go much faster). Given the cpus are faster now the limit is probably up to 1Gbit/sec/interrupt. I have had to set nic adaptors to use a layer-3 data -> interrupt so that you can get higher rates (ethtool setting) only available on some nic cards. I have also had to set the interrupt number (ethtool setting), note that it is rather pointless to have the interrupt count higher than the number of real cores. And likely you want the interrupt count for the disk controllers being used + the nic interrupts <= the number of cores. At home I question the value of it. You might even simply test your nfs home setup and see if you can even get close to Gbit speeds, unless you have a newer machine and a good underlying disk setup you probably aren't going to get there. And you also need a decent Gbit card, but if you do not have a decent gbit card, you can buy a used enterprise grade dual gbit card for around $25 (branded Dell or branded HPE). With enterprise grade hw and good disk and other parts one can hit 115-125MBytes/sec. With a new machine at home and a 7-disk raid-6 setup with a good gbit nic card I can get 115-125MB/sec when data is in cache, but not coming directly off of disk, unless the disk is an ssd and reading large files. My prior machine could not hit that rate but was 10 year old hw. To use a 10Gbit interface you will have to have multiple machines doing large file sequential io (assuming they are wireless or gbit interfaces) at the same time. I doubt you are going to gain enough (or possibly any) to make it worth the cost or the trouble to get it working. Install sar and configure it to sample at 1minute and sar -n DEV will show you your network rates, if you aren't currently sustaining 115MByte/sec every so often then 10gbit is not going to do anything for you.
My home office has a couple of HP Proliant servers with big (12 SAS drives) RAID arrays and dual CPU E5-2697 v2 processors with 24 cores each (48 cores total in the machine). I can sustain 1.6Gb/sec reads and writes, tested via dd with oflag=dsync and also with fio.
So with consumer grade NICs on the Proliants and my workstation (https://www.amazon.com/TP-Link-TX401-Ethernet-Supports-Including/dp/B08D71PVXG/), I suspect it will give me better performance than the 1Gb ethernet I'm currently using. Will it be as amazing as high end enterprise gear? Most likely not. Will it be better than 1Gb ethernet I'm currently using? I suspect so.
I appreciate the advice, I'll definitely play around with the settings to make sure I'm using enough cores.
Thomas _______________________________________________ users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
