Re: Experience with 10Gb ethernet adapters?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have work experience with Intel 10Gbit, Older Emulex 10GB (be[23]net
driver), current Broadcom 10/25G and Mellanox 10/25G.

Note that for it to be useful you *MUST* have multiple interrupts.
1Gbit interfaces used to hit a limit at around 50Mbit (the cpu was not
fast enough to do the required data moves on a single core to go much
faster).  Given the cpus are faster now the limit is probably up to
1Gbit/sec/interrupt.

I have had to set nic adaptors to use a layer-3 data -> interrupt so
that you can get higher rates (ethtool setting) only available on some
nic cards.

I have also had to set the interrupt number (ethtool setting), note
that it is rather pointless to have the interrupt count higher than
the number of real cores.  And likely you want the interrupt count for
the disk controllers being used + the nic interrupts <= the number of
cores.

At home I question the value of it.   You might even simply test your
nfs home setup and see if you can even get close to Gbit speeds,
unless you have a newer machine and a good underlying disk setup you
probably aren't going to get there.  And you also need a decent Gbit
card, but if you do not have a decent gbit card, you can buy a used
enterprise grade dual gbit card for around $25 (branded Dell or
branded HPE).

With enterprise grade hw and good disk and other parts one can hit
115-125MBytes/sec.

With a new machine at home and a 7-disk raid-6 setup with a good gbit
nic card I can get 115-125MB/sec when data is in cache, but not coming
directly off of disk, unless the disk is an ssd and reading large
files.  My prior machine could not hit that rate but was 10 year old
hw.

To use a 10Gbit interface you will have to have multiple machines
doing large file sequential io (assuming they are wireless or gbit
interfaces) at the same time.

I doubt you are going to gain enough (or possibly any) to make it
worth the cost or the trouble to get it working.

Install sar and configure it to sample at 1minute and sar -n DEV will
show you your network rates, if you aren't currently sustaining
115MByte/sec every so often then 10gbit is not going to do anything
for you.


On Fri, May 12, 2023 at 8:36 AM Thomas Cameron
<thomas.cameron@xxxxxxxxxxxxxxx> wrote:
>
> Howdy, all -
>
> I use an NFS server export to mount my /home directory on my desktop.
> I've got the itch to go to 10Gb ethernet, but I am reading that the
> tp-link tx401 has a problem with bridging, and I use bridging for KVM
> virtual machines on my desktop. I *think* that you can just disable
> using the command "ethtool -K <ethX> lro off," but I wondered if anyone
> had any experience with NICs that work with bridging out of the box.
>
> Anyone got any experience with 10Gb ethernet cards? Good? Bad? Hassles?
>
> Thomas
> _______________________________________________
> users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue



[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [EPEL Devel]     [Fedora Magazine]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Desktop]     [Fedora Fonts]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Fedora Sparc]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux