And also given your hw, limit your interrupt count to the number of cores on a single socket, the interrupts will typically only happen on the local socket the given pci bus is attached to, and you would need to determine if different cards have their PCI buses connected to different sockets. Which model of proliant is that? Note I have used/debugged/optimized dl360/dl380/dl560/dl580/dl980 gen7/8/9/10/11(testing). On Fri, May 12, 2023 at 10:14 AM Roger Heflin <rogerheflin@xxxxxxxxx> wrote: > > Given that hardware, buy something like this instead. > > used, but better class of card. > > https://www.newegg.com/intel-e10g42bt/p/N82E16833106075?Description=10gbit%20card&cm_re=10gbit_card-_-33-106-075-_-Product&quicklink=true > > or something similar from the used sellers. There seem to be a decent > variety of cards under $100. > > Manufacturers that are good are HPE/DELL/Intel/Broadcom/IBM/Lenovo. > But you do need to do a bit of research on the given cards to see what > the real chipset is. Avoid Emulex/Be2net variants they have "issues". > Intel/Broadcom/Mellanox based cards are good. > > On Fri, May 12, 2023 at 10:06 AM Thomas Cameron > <thomas.cameron@xxxxxxxxxxxxxxx> wrote: > > > > On 5/12/23 09:41, Roger Heflin wrote: > > > I have work experience with Intel 10Gbit, Older Emulex 10GB (be[23]net > > > driver), current Broadcom 10/25G and Mellanox 10/25G. > > > > > > Note that for it to be useful you*MUST* have multiple interrupts. > > > 1Gbit interfaces used to hit a limit at around 50Mbit (the cpu was not > > > fast enough to do the required data moves on a single core to go much > > > faster). Given the cpus are faster now the limit is probably up to > > > 1Gbit/sec/interrupt. > > > > > > I have had to set nic adaptors to use a layer-3 data -> interrupt so > > > that you can get higher rates (ethtool setting) only available on some > > > nic cards. > > > > > > I have also had to set the interrupt number (ethtool setting), note > > > that it is rather pointless to have the interrupt count higher than > > > the number of real cores. And likely you want the interrupt count for > > > the disk controllers being used + the nic interrupts <= the number of > > > cores. > > > > > > At home I question the value of it. You might even simply test your > > > nfs home setup and see if you can even get close to Gbit speeds, > > > unless you have a newer machine and a good underlying disk setup you > > > probably aren't going to get there. And you also need a decent Gbit > > > card, but if you do not have a decent gbit card, you can buy a used > > > enterprise grade dual gbit card for around $25 (branded Dell or > > > branded HPE). > > > > > > With enterprise grade hw and good disk and other parts one can hit > > > 115-125MBytes/sec. > > > > > > With a new machine at home and a 7-disk raid-6 setup with a good gbit > > > nic card I can get 115-125MB/sec when data is in cache, but not coming > > > directly off of disk, unless the disk is an ssd and reading large > > > files. My prior machine could not hit that rate but was 10 year old > > > hw. > > > > > > To use a 10Gbit interface you will have to have multiple machines > > > doing large file sequential io (assuming they are wireless or gbit > > > interfaces) at the same time. > > > > > > I doubt you are going to gain enough (or possibly any) to make it > > > worth the cost or the trouble to get it working. > > > > > > Install sar and configure it to sample at 1minute and sar -n DEV will > > > show you your network rates, if you aren't currently sustaining > > > 115MByte/sec every so often then 10gbit is not going to do anything > > > for you. > > > > My home office has a couple of HP Proliant servers with big (12 SAS > > drives) RAID arrays and dual CPU E5-2697 v2 processors with 24 cores > > each (48 cores total in the machine). I can sustain 1.6Gb/sec reads and > > writes, tested via dd with oflag=dsync and also with fio. > > > > So with consumer grade NICs on the Proliants and my workstation > > (https://www.amazon.com/TP-Link-TX401-Ethernet-Supports-Including/dp/B08D71PVXG/), > > I suspect it will give me better performance than the 1Gb ethernet I'm > > currently using. Will it be as amazing as high end enterprise gear? Most > > likely not. Will it be better than 1Gb ethernet I'm currently using? I > > suspect so. > > > > I appreciate the advice, I'll definitely play around with the settings > > to make sure I'm using enough cores. > > > > Thomas > > _______________________________________________ > > users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx > > To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx > > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue _______________________________________________ users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
