Hi Stefan,
ahh OK, misunderstood your e-mail. It sounded like it was a custom profile, not a standard one shipped with tuned.
Thanks for the clarification!
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Stefan Bauer <sb@xxxxxxx>
Sent: Wednesday, May 22, 2024 12:44 PM
To: ceph-users@xxxxxxx
Subject: Re: How network latency affects ceph performance really with NVME only storage?
Hi Frank,
it's pretty straightforward. Just follow the steps:
apt install tuned
tuned-adm profile network-latency
According to [1]:
network-latency
A server profile focused on lowering network latency.
This profile favors performance over power savings by setting
|intel_pstate| and |min_perf_pct=100|. It disables transparent huge
pages, and automatic NUMA balancing. It also uses *cpupower* to set
the |performance| cpufreq governor, and requests a
/|cpu_dma_latency|/ value of |1|. It also sets /|busy_read|/ and
/|busy_poll|/ times to |50| μs, and /|tcp_fastopen|/ to |3|.
[1]
https://access.redhat.com/documentation/de-de/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-tool_reference-tuned_adm
Cheers.
Stefan
Am 22.05.24 um 12:18 schrieb Frank Schilder:
> Hi Stefan,
>
> can you provide a link to or copy of the contents of the tuned-profile so others can also profit from it?
>
> Thanks!
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Stefan Bauer<sb@xxxxxxx>
> Sent: Wednesday, May 22, 2024 10:51 AM
> To: Anthony D'Atri;ceph-users@xxxxxxx
> Subject: Re: How network latency affects ceph performance really with NVME only storage?
>
> Hi Anthony and others,
>
> thank you for your reply. To be honest, I'm not even looking for a
> solution, i just wanted to ask if latency affects the performance at all
> in my case and how others handle this ;)
>
> One of our partners delivered a solution with a latency-optimized
> profile for tuned-daemon. Now the latency is much better:
>
> apt install tuned
>
> tuned-adm profile network-latency
>
> # ping 10.1.4.13
> PING 10.1.4.13 (10.1.4.13) 56(84) bytes of data.
> 64 bytes from 10.1.4.13: icmp_seq=1 ttl=64 time=0.047 ms
> 64 bytes from 10.1.4.13: icmp_seq=2 ttl=64 time=0.028 ms
> 64 bytes from 10.1.4.13: icmp_seq=3 ttl=64 time=0.025 ms
> 64 bytes from 10.1.4.13: icmp_seq=4 ttl=64 time=0.020 ms
> 64 bytes from 10.1.4.13: icmp_seq=5 ttl=64 time=0.023 ms
> 64 bytes from 10.1.4.13: icmp_seq=6 ttl=64 time=0.026 ms
> 64 bytes from 10.1.4.13: icmp_seq=7 ttl=64 time=0.024 ms
> 64 bytes from 10.1.4.13: icmp_seq=8 ttl=64 time=0.023 ms
> 64 bytes from 10.1.4.13: icmp_seq=9 ttl=64 time=0.033 ms
> 64 bytes from 10.1.4.13: icmp_seq=10 ttl=64 time=0.021 ms
> ^C
> --- 10.1.4.13 ping statistics ---
> 10 packets transmitted, 10 received, 0% packet loss, time 9001ms
> rtt min/avg/max/mdev = 0.020/0.027/0.047/0.007 ms
>
> Am 21.05.24 um 15:08 schrieb Anthony D'Atri:
>> Check the netmask on your interfaces, is it possible that you're sending inter-node traffic up and back down needlessly?
>>
>>> On May 21, 2024, at 06:02, Stefan Bauer<sb@xxxxxxx> wrote:
>>>
>>> Dear Users,
>>>
>>> i recently setup a new ceph 3 node cluster. Network is meshed between all nodes (2 x 25G with DAC).
>>> Storage is flash only (Kioxia 3.2 TBBiCS FLASH 3D TLC, KCMYXVUG3T20)
>>>
>>> The latency with ping tests between the nodes shows:
>>>
>>> # ping 10.1.3.13
>>> PING 10.1.3.13 (10.1.3.13) 56(84) bytes of data.
>>> 64 bytes from 10.1.3.13: icmp_seq=1 ttl=64 time=0.145 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=2 ttl=64 time=0.180 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=3 ttl=64 time=0.180 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=4 ttl=64 time=0.115 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=5 ttl=64 time=0.110 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=6 ttl=64 time=0.120 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=7 ttl=64 time=0.124 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=8 ttl=64 time=0.140 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=9 ttl=64 time=0.127 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=10 ttl=64 time=0.143 ms
>>> 64 bytes from 10.1.3.13: icmp_seq=11 ttl=64 time=0.129 ms
>>> --- 10.1.3.13 ping statistics ---
>>> 11 packets transmitted, 11 received, 0% packet loss, time 10242ms
>>> rtt min/avg/max/mdev = 0.110/0.137/0.180/0.022 ms
>>>
>>>
>>> On another cluster i have much better values, with 10G SFP+ and fibre-cables:
>>>
>>> 64 bytes from large-ipv6-ip: icmp_seq=42 ttl=64 time=0.081 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=43 ttl=64 time=0.078 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=44 ttl=64 time=0.084 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=45 ttl=64 time=0.075 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=46 ttl=64 time=0.071 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=47 ttl=64 time=0.081 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=48 ttl=64 time=0.074 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=49 ttl=64 time=0.085 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=50 ttl=64 time=0.077 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=51 ttl=64 time=0.080 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=52 ttl=64 time=0.084 ms
>>> 64 bytes from large-ipv6-ip: icmp_seq=53 ttl=64 time=0.084 ms
>>> ^C
>>> --- long-ipv6-ip ping statistics ---
>>> 53 packets transmitted, 53 received, 0% packet loss, time 53260ms
>>> rtt min/avg/max/mdev = 0.071/0.082/0.111/0.006 ms
>>>
>>> If i want best performance, does the latency difference matter at all? Should i change DAC to SFP-transceivers wwith fibre-cables to improve overall ceph performance or is this nitpicking?
>>>
>>> Thanks a lot.
>>>
>>> Stefan
>>> _______________________________________________
>>> ceph-users mailing list --ceph-users@xxxxxxx
>>> To unsubscribe send an email toceph-users-leave@xxxxxxx
> --
> Mit freundlichen Grüßen
>
> Stefan Bauer
> Schulstraße 5
> 83308 Trostberg
> 0179-1194767
> _______________________________________________
> ceph-users mailing list --ceph-users@xxxxxxx
> To unsubscribe send an email toceph-users-leave@xxxxxxx
--
Mit freundlichen Grüßen
Stefan Bauer
Schulstraße 5
83308 Trostberg
0179-1194767
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
