I think the primary area where people are concerned about latency are rbd and 4k block size access. OTOH 2.3us latency seems to be 2 orders of magnitude below of what seems to be realistically achievable on a real world cluster anyway (http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-July/011731.html) so I don't really think the basic latency difference from copper vs fiber as listed make much of a difference at this point
On Thu, 22 Mar 2018 at 17:14, Subhachandra Chandra <schandra@xxxxxxxxxxxx> wrote:
Latency is a concern if your application is sending one packet at a time and waiting for a reply. If you are streaming large blocks of data, the first packet is delayed by the network latency but after that you will receive a 10Gbps stream continuously. The latency for jumbo frames vs 1500 byte frames depends upon the switch type. On a cut-through switch there is very little difference but on a store-and-forward switch it will be proportional to packet size. Most modern switching ASICs are capable of cut-through operation.Subhachandra_______________________________________________On Wed, Mar 21, 2018 at 7:15 AM, Willem Jan Withagen <wjw@xxxxxxxxxxx> wrote:On 21-3-2018 13:47, Paul Emmerich wrote:
> Hi,
>
> 2.3µs is a typical delay for a 10GBASE-T connection. But fiber or SFP+
> DAC connections should be faster: switches are typically in the range of
> ~500ns to 1µs.
>
>
> But you'll find that this small difference in latency induced by the
> switch will be quite irrelevant in the grand scheme of things when using
> the Linux network stack...
But I think it does when people start to worry about selecting High
clock speed CPUS versus packages with more cores...
900ns is quite a lot if you have that mindset.
And probably 1800ns at that, because the delay will be a both ends.
Or perhaps even 3600ns because the delay is added at every ethernet
connector???
But I'm inclined to believe you that the network stack could take quite
some time...
--WjW
> Paul
>
> 2018-03-21 12:16 GMT+01:00 Willem Jan Withagen <wjw@xxxxxxxxxxx
> <mailto:wjw@xxxxxxxxxxx>>:
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>
> Hi,
>
> I just ran into this table for a 10G Netgear switch we use:
>
> Fiberdelays:
> 10 Gbps vezelvertraging (64 bytepakketten): 1.827 µs
> 10 Gbps vezelvertraging (512 bytepakketten): 1.919 µs
> 10 Gbps vezelvertraging (1024 bytepakketten): 1.971 µs
> 10 Gbps vezelvertraging (1518 bytepakketten): 1.905 µs
>
> Copperdelays:
> 10 Gbps kopervertraging (64 bytepakketten): 2.728 µs
> 10 Gbps kopervertraging (512 bytepakketten): 2.85 µs
> 10 Gbps kopervertraging (1024 bytepakketten): 2.904 µs
> 10 Gbps kopervertraging (1518 bytepakketten): 2.841 µs
>
> Fiberdelays:
> 1 Gbps vezelvertraging (64 bytepakketten) 2.289 µs
> 1 Gbps vezelvertraging (512 bytepakketten) 2.393 µs
> 1 Gbps vezelvertraging (1024 bytepakketten) 2.423 µs
> 1 Gbps vezelvertraging (1518 bytepakketten) 2.379 µs
>
> Copperdelays:
> 1 Gbps kopervertraging (64 bytepakketten) 2.707 µs
> 1 Gbps kopervertraging (512 bytepakketten) 2.821 µs
> 1 Gbps kopervertraging (1024 bytepakketten) 2.866 µs
> 1 Gbps kopervertraging (1518 bytepakketten) 2.826 µs
>
> So the difference is serious: 900ns on a total of 1900ns for a 10G
> pakket.
> Other strange thing is that 1K packets are slower than 1518 bytes.
>
> So that might warrant connecting boxes preferably with optics
> instead of CAT cableing if you are trying to squeeze the max out of
> a setup.
>
> Sad thing is that they do not report for jumbo frames, and doing these
> measurements your self is not easy...
>
> --WjW
>
> _______________________________________________
> ceph-users mailing list
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
>
>
>
>
> --
> --
> Paul Emmerich
>
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io <http://www.croit.io>
> Tel: +49 89 1896585 90
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
