Hi Stefan, can you provide a link to or copy of the contents of the tuned-profile so others can also profit from it? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Bauer <sb@xxxxxxx> Sent: Wednesday, May 22, 2024 10:51 AM To: Anthony D'Atri; ceph-users@xxxxxxx Subject: Re: How network latency affects ceph performance really with NVME only storage? Hi Anthony and others, thank you for your reply. To be honest, I'm not even looking for a solution, i just wanted to ask if latency affects the performance at all in my case and how others handle this ;) One of our partners delivered a solution with a latency-optimized profile for tuned-daemon. Now the latency is much better: apt install tuned tuned-adm profile network-latency # ping 10.1.4.13 PING 10.1.4.13 (10.1.4.13) 56(84) bytes of data. 64 bytes from 10.1.4.13: icmp_seq=1 ttl=64 time=0.047 ms 64 bytes from 10.1.4.13: icmp_seq=2 ttl=64 time=0.028 ms 64 bytes from 10.1.4.13: icmp_seq=3 ttl=64 time=0.025 ms 64 bytes from 10.1.4.13: icmp_seq=4 ttl=64 time=0.020 ms 64 bytes from 10.1.4.13: icmp_seq=5 ttl=64 time=0.023 ms 64 bytes from 10.1.4.13: icmp_seq=6 ttl=64 time=0.026 ms 64 bytes from 10.1.4.13: icmp_seq=7 ttl=64 time=0.024 ms 64 bytes from 10.1.4.13: icmp_seq=8 ttl=64 time=0.023 ms 64 bytes from 10.1.4.13: icmp_seq=9 ttl=64 time=0.033 ms 64 bytes from 10.1.4.13: icmp_seq=10 ttl=64 time=0.021 ms ^C --- 10.1.4.13 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9001ms rtt min/avg/max/mdev = 0.020/0.027/0.047/0.007 ms Am 21.05.24 um 15:08 schrieb Anthony D'Atri: > Check the netmask on your interfaces, is it possible that you're sending inter-node traffic up and back down needlessly? > >> On May 21, 2024, at 06:02, Stefan Bauer <sb@xxxxxxx> wrote: >> >> Dear Users, >> >> i recently setup a new ceph 3 node cluster. Network is meshed between all nodes (2 x 25G with DAC). >> Storage is flash only (Kioxia 3.2 TBBiCS FLASH 3D TLC, KCMYXVUG3T20) >> >> The latency with ping tests between the nodes shows: >> >> # ping 10.1.3.13 >> PING 10.1.3.13 (10.1.3.13) 56(84) bytes of data. >> 64 bytes from 10.1.3.13: icmp_seq=1 ttl=64 time=0.145 ms >> 64 bytes from 10.1.3.13: icmp_seq=2 ttl=64 time=0.180 ms >> 64 bytes from 10.1.3.13: icmp_seq=3 ttl=64 time=0.180 ms >> 64 bytes from 10.1.3.13: icmp_seq=4 ttl=64 time=0.115 ms >> 64 bytes from 10.1.3.13: icmp_seq=5 ttl=64 time=0.110 ms >> 64 bytes from 10.1.3.13: icmp_seq=6 ttl=64 time=0.120 ms >> 64 bytes from 10.1.3.13: icmp_seq=7 ttl=64 time=0.124 ms >> 64 bytes from 10.1.3.13: icmp_seq=8 ttl=64 time=0.140 ms >> 64 bytes from 10.1.3.13: icmp_seq=9 ttl=64 time=0.127 ms >> 64 bytes from 10.1.3.13: icmp_seq=10 ttl=64 time=0.143 ms >> 64 bytes from 10.1.3.13: icmp_seq=11 ttl=64 time=0.129 ms >> --- 10.1.3.13 ping statistics --- >> 11 packets transmitted, 11 received, 0% packet loss, time 10242ms >> rtt min/avg/max/mdev = 0.110/0.137/0.180/0.022 ms >> >> >> On another cluster i have much better values, with 10G SFP+ and fibre-cables: >> >> 64 bytes from large-ipv6-ip: icmp_seq=42 ttl=64 time=0.081 ms >> 64 bytes from large-ipv6-ip: icmp_seq=43 ttl=64 time=0.078 ms >> 64 bytes from large-ipv6-ip: icmp_seq=44 ttl=64 time=0.084 ms >> 64 bytes from large-ipv6-ip: icmp_seq=45 ttl=64 time=0.075 ms >> 64 bytes from large-ipv6-ip: icmp_seq=46 ttl=64 time=0.071 ms >> 64 bytes from large-ipv6-ip: icmp_seq=47 ttl=64 time=0.081 ms >> 64 bytes from large-ipv6-ip: icmp_seq=48 ttl=64 time=0.074 ms >> 64 bytes from large-ipv6-ip: icmp_seq=49 ttl=64 time=0.085 ms >> 64 bytes from large-ipv6-ip: icmp_seq=50 ttl=64 time=0.077 ms >> 64 bytes from large-ipv6-ip: icmp_seq=51 ttl=64 time=0.080 ms >> 64 bytes from large-ipv6-ip: icmp_seq=52 ttl=64 time=0.084 ms >> 64 bytes from large-ipv6-ip: icmp_seq=53 ttl=64 time=0.084 ms >> ^C >> --- long-ipv6-ip ping statistics --- >> 53 packets transmitted, 53 received, 0% packet loss, time 53260ms >> rtt min/avg/max/mdev = 0.071/0.082/0.111/0.006 ms >> >> If i want best performance, does the latency difference matter at all? Should i change DAC to SFP-transceivers wwith fibre-cables to improve overall ceph performance or is this nitpicking? >> >> Thanks a lot. >> >> Stefan >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx -- Mit freundlichen Grüßen Stefan Bauer Schulstraße 5 83308 Trostberg 0179-1194767 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |