Let me share some outputs about my cluster.
root@ud-01:~# ceph fs status
ud-data - 84 clients
=======
RANK STATE MDS ACTIVITY DNS INOS DIRS
CAPS
0 active ud-data.ud-02.xcoojt Reqs: 31 /s 3022k 3021k 52.6k
385k
POOL TYPE USED AVAIL
cephfs.ud-data.meta metadata 136G 44.4T
cephfs.ud-data.data data 45.2T 44.4T
STANDBY MDS
ud-data.ud-03.lhwkml
ud-data.ud-05.rnhcfe
ud-data.ud-01.uatjle
ud-data.ud-04.seggyv
--------------------------------------------------------------------------
This is "ceph tell mds.ud-data.ud-02.xcoojt session ls" output for the
reported client for cache pressure warning.
{
"id": 1282205,
"entity": {
"name": {
"type": "client",
"num": 1282205
},
"addr": {
"type": "v1",
"addr": "172.16.3.48:0",
"nonce": 2169935642
}
},
"state": "open",
"num_leases": 0,
"num_caps": 52092,
"request_load_avg": 1,
"uptime": 75754.745608647994,
"requests_in_flight": 0,
"num_completed_requests": 0,
"num_completed_flushes": 1,
"reconnecting": false,
"recall_caps": {
"value": 2577232.0049106553,
"halflife": 60
},
"release_caps": {
"value": 1.4093491463510395,
"halflife": 60
},
"recall_caps_throttle": {
"value": 63733.985544098425,
"halflife": 1.5
},
"recall_caps_throttle2o": {
"value": 19452.428409271757,
"halflife": 0.5
},
"session_cache_liveness": {
"value": 14.100272208890081,
"halflife": 300
},
"cap_acquisition": {
"value": 0,
"halflife": 10
},
"delegated_inos": [
{
"start": "0x10004a1c031",
"length": 282
},
{
"start": "0x10004a1c33f",
"length": 207
},
{
"start": "0x10004a1cdda",
"length": 6
},
{
"start": "0x10004a3c12e",
"length": 3
},
{
"start": "0x1000f9831fe",
"length": 2
}
],
"inst": "client.1282205 v1:172.16.3.48:0/2169935642",
"completed_requests": [],
"prealloc_inos": [
{
"start": "0x10004a1c031",
"length": 282
},
{
"start": "0x10004a1c33f",
"length": 207
},
{
"start": "0x10004a1cdda",
"length": 6
},
{
"start": "0x10004a3c12e",
"length": 3
},
{
"start": "0x1000f9831fe",
"length": 2
},
{
"start": "0x1000fa86e5f",
"length": 54
},
{
"start": "0x1000faa069c",
"length": 501
}
],
"client_metadata": {
"client_features": {
"feature_bits": "0x0000000000007bff"
},
"metric_spec": {
"metric_flags": {
"feature_bits": "0x00000000000003ff"
}
},
"entity_id": "admin",
"hostname": "bennevis-2",
"kernel_version": "5.15.0-91-generic",
"root": "/volumes/babblians"
}
}
Özkan Göksu <ozkangksu@xxxxxxxxx>, 17 Oca 2024 Çar, 07:22 tarihinde şunu
yazdı:
Hello Eugen.
Thank you for the answer.
According to knowledge and test results at this issue:
https://github.com/ceph/ceph/pull/38574
I've tried their advice and I've applied the following changes.
max_mds = 4
standby_mds = 1
mds_cache_memory_limit = 16GB
mds_recall_max_caps = 40000
When I set these parameters, 1 day later I saw this log:
[8531248.982954] Out of memory: Killed process 1580586 (ceph-mds)
total-vm:70577592kB, anon-rss:70244236kB, file-rss:0kB, shmem-rss:0kB,
UID:167 pgtables:137832kB oom_score_adj:0
All the MDS services created memory leak and killed by kernel.
Because of this I changed it as below and it is stable now but
performance
is very poor and I still get cache pressure alerts.
max_mds = 1
standby_mds = 5
mds_cache_memory_limit = 8GB
mds_recall_max_caps = 30000
I'm very surprised that you are advising to decrease
"mds_recall_max_caps"
because it is the opposite of what developers advised in the issue I've
sended.
It is very hard to play around with MDS parameters without expert
level of
understanding what these parameters stands for and how it will
effect the
behavior.
Because of this I'm trying to understand the MDS code flow and I'm very
interested with learning more and tuning my system by debugging and
understanding my own data flow and MDS usage.
I have a very unique data flow and I think I need to configure the
system
for this case.
I have 80+ clients and via all of these clients my users are requesting
Read a range of objects and compare them in GPU, they generate new
data and
Write the new data back in the cluster.
So it means my clients usually reads objects only one time and do
not read
the same object again. Sometimes same user runs multiple service in
multiple clients and these services can read the same data from
different
clients.
So having a large cache is useless for my use case. I need to setup MDS
and Cephfs Client for this data flow.
When I debug the MDS ram usage, I see high allocation all the time
and I
wonder why? If any of my client does not read any object why MDS
does not
remove that data from ram allocation?
I need to configure MDS for reading the data and removing it very
fast if
the data is constantly requested from clients. In this case ofc I
want a
ram cache tier.
I'm little confused and I need to learn more about how MDS works and
how
should I make multiple active MDS faster for my subvolumes and
client data
flow.
Best regards.
Eugen Block <eblock@xxxxxx>, 16 Oca 2024 Sal, 11:36 tarihinde şunu
yazdı:
Hi,
I have dealt with this topic multiple times, the SUSE team helped
understanding what's going on under the hood. The summary can be found
in this thread [1].
What helped in our case was to reduce the mds_recall_max_caps from 30k
(default) to 3k. We tried it in steps of 1k IIRC. So I suggest to
reduce that value step by step (maybe start with 20k or something) to
find the optimal value.
Regards,
Eugen
[1] https://www.spinics.net/lists/ceph-users/msg73188.html
Zitat von Özkan Göksu <ozkangksu@xxxxxxxxx>:
> Hello.
>
> I have 5 node ceph cluster and I'm constantly having "clients
failing to
> respond to cache pressure" warning.
>
> I have 84 cephfs kernel clients (servers) and my users are accessing
their
> personal subvolumes located on one pool.
>
> My users are software developers and the data is home and user data.
(Git,
> python projects, sample data and generated new data)
>
>
---------------------------------------------------------------------------------
> --- RAW STORAGE ---
> CLASS SIZE AVAIL USED RAW USED %RAW USED
> ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71
> TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71
>
> --- POOLS ---
> POOL ID PGS STORED OBJECTS USED %USED MAX
AVAIL
> .mgr 1 1 356 MiB 90 1.0 GiB
0 30
TiB
> cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB
0.15 45
TiB
> cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB
32.97 45
TiB
>
---------------------------------------------------------------------------------
> root@ud-01:~# ceph fs status
> ud-data - 84 clients
> =======
> RANK STATE MDS ACTIVITY DNS INOS DIRS
> CAPS
> 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k
303k
> 720k
> POOL TYPE USED AVAIL
> cephfs.ud-data.meta metadata 137G 44.9T
> cephfs.ud-data.data data 44.2T 44.9T
> STANDBY MDS
> ud-data.ud-02.xcoojt
> ud-data.ud-05.rnhcfe
> ud-data.ud-03.lhwkml
> ud-data.ud-01.uatjle
> MDS version: ceph version 17.2.6
(d7ff0d10654d2280e08f1ab989c7cdf3064446a5)
> quincy (stable)
>
>
-----------------------------------------------------------------------------------
> My MDS settings are below:
>
> mds_cache_memory_limit | 8589934592
> mds_cache_trim_threshold | 524288
> mds_recall_global_max_decay_threshold | 131072
> mds_recall_max_caps | 30000
> mds_recall_max_decay_rate | 1.500000
> mds_recall_max_decay_threshold | 131072
> mds_recall_warning_threshold | 262144
>
>
> I have 2 questions:
> 1- What should I do to prevent cache pressue warning ?
> 2- What can I do to increase speed ?
>
> - Thanks
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
