Re: 1 clients failing to respond to cache pressure (quincy:17.2.6)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I understand that your MDS shows a high CPU usage, but other than that what is your performance issue? Do users complain? Do some operations take longer than expected? Are OSDs saturated during those phases? Because the cache pressure messages don’t necessarily mean that users will notice. MDS daemons are single-threaded so that might be a bottleneck. In that case multi-active mds might help, which you already tried and experienced OOM killers. But you might have to disable the mds balancer as someone else mentioned. And then you could think about pinning, is it possible to split the CephFS into multiple subdirectories and pin them to different ranks?
But first I’d still like to know what the performance issue really is.

Zitat von Özkan Göksu <ozkangksu@xxxxxxxxx>:

I will try my best to explain my situation.

I don't have a separate mds server. I have 5 identical nodes, 3 of them
mons, and I use the other 2 as active and standby mds. (currently I have
left overs from max_mds 4)

root@ud-01:~# ceph -s
  cluster:
    id:     e42fd4b0-313b-11ee-9a00-31da71873773
    health: HEALTH_WARN
            1 clients failing to respond to cache pressure

  services:
    mon: 3 daemons, quorum ud-01,ud-02,ud-03 (age 9d)
    mgr: ud-01.qycnol(active, since 8d), standbys: ud-02.tfhqfd
    mds: 1/1 daemons up, 4 standby
    osd: 80 osds: 80 up (since 9d), 80 in (since 5M)

  data:
    volumes: 1/1 healthy
    pools:   3 pools, 2305 pgs
    objects: 106.58M objects, 25 TiB
    usage:   45 TiB used, 101 TiB / 146 TiB avail
    pgs:     2303 active+clean
             2    active+clean+scrubbing+deep

  io:
    client:   16 MiB/s rd, 3.4 MiB/s wr, 77 op/s rd, 23 op/s wr

------------------------------
root@ud-01:~# ceph fs status
ud-data - 84 clients
=======
RANK  STATE           MDS              ACTIVITY     DNS    INOS   DIRS
CAPS
 0    active  ud-data.ud-02.xcoojt  Reqs:   40 /s  2579k  2578k   169k
 3048k
        POOL           TYPE     USED  AVAIL
cephfs.ud-data.meta  metadata   136G  44.9T
cephfs.ud-data.data    data    44.3T  44.9T

------------------------------
root@ud-01:~# ceph health detail
HEALTH_WARN 1 clients failing to respond to cache pressure
[WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure
    mds.ud-data.ud-02.xcoojt(mds.0): Client bmw-m4 failing to respond to
cache pressure client_id: 1275577

------------------------------
When I check the failing client with session ls I see only "num_caps: 12298"

ceph tell mds.ud-data.ud-02.xcoojt session ls | jq -r '.[] | "clientid:
\(.id)= num_caps: \(.num_caps), num_leases: \(.num_leases),
request_load_avg: \(.request_load_avg), num_completed_requests:
\(.num_completed_requests), num_completed_flushes:
\(.num_completed_flushes)"' | sort -n -t: -k3

clientid: 1275577= num_caps: 12298, num_leases: 0, request_load_avg: 0,
num_completed_requests: 0, num_completed_flushes: 1
clientid: 1294542= num_caps: 13000, num_leases: 12, request_load_avg: 105,
num_completed_requests: 0, num_completed_flushes: 6
clientid: 1282187= num_caps: 16869, num_leases: 1, request_load_avg: 0,
num_completed_requests: 0, num_completed_flushes: 1
clientid: 1275589= num_caps: 18943, num_leases: 0, request_load_avg: 52,
num_completed_requests: 0, num_completed_flushes: 1
clientid: 1282154= num_caps: 24747, num_leases: 1, request_load_avg: 57,
num_completed_requests: 2, num_completed_flushes: 2
clientid: 1275553= num_caps: 25120, num_leases: 2, request_load_avg: 116,
num_completed_requests: 2, num_completed_flushes: 8
clientid: 1282142= num_caps: 27185, num_leases: 6, request_load_avg: 128,
num_completed_requests: 0, num_completed_flushes: 8
clientid: 1275535= num_caps: 40364, num_leases: 6, request_load_avg: 111,
num_completed_requests: 2, num_completed_flushes: 8
clientid: 1282130= num_caps: 41483, num_leases: 0, request_load_avg: 135,
num_completed_requests: 0, num_completed_flushes: 1
clientid: 1275547= num_caps: 42953, num_leases: 4, request_load_avg: 119,
num_completed_requests: 2, num_completed_flushes: 6
clientid: 1282139= num_caps: 45435, num_leases: 27, request_load_avg: 84,
num_completed_requests: 2, num_completed_flushes: 34
clientid: 1282136= num_caps: 48374, num_leases: 8, request_load_avg: 0,
num_completed_requests: 1, num_completed_flushes: 1
clientid: 1275532= num_caps: 48664, num_leases: 7, request_load_avg: 115,
num_completed_requests: 2, num_completed_flushes: 8
clientid: 1191789= num_caps: 130319, num_leases: 0, request_load_avg: 1753,
num_completed_requests: 0, num_completed_flushes: 0
clientid: 1275571= num_caps: 139488, num_leases: 0, request_load_avg: 2,
num_completed_requests: 0, num_completed_flushes: 1
clientid: 1282133= num_caps: 145487, num_leases: 0, request_load_avg: 8,
num_completed_requests: 1, num_completed_flushes: 1
clientid: 1534496= num_caps: 1041316, num_leases: 0, request_load_avg: 0,
num_completed_requests: 0, num_completed_flushes: 1

------------------------------
When I check the dashboard/service/mds I see %120+ CPU usage on active MDS
but on the host everything is almost idle and disk waits are very low.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.61    0.00    0.38    0.41    0.00   98.60

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s
  wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm
d_await dareq-sz     f/s f_await  aqu-sz  %util
sdc              2.00      0.01     0.00   0.00    0.50     6.00   20.00
   0.04     0.00   0.00    0.50     2.00    0.00      0.00     0.00   0.00
   0.00     0.00   10.00    0.60    0.02   1.20
sdd              3.00      0.02     0.00   0.00    0.67     8.00  285.00
   1.84    77.00  21.27    0.44     6.61    0.00      0.00     0.00   0.00
   0.00     0.00  114.00    0.83    0.22  22.40
sde              1.00      0.01     0.00   0.00    1.00     8.00   36.00
   0.08     3.00   7.69    0.64     2.33    0.00      0.00     0.00   0.00
   0.00     0.00   18.00    0.67    0.04   1.60
sdf              5.00      0.04     0.00   0.00    0.40     7.20   40.00
   0.09     3.00   6.98    0.53     2.30    0.00      0.00     0.00   0.00
   0.00     0.00   20.00    0.70    0.04   2.00
sdg             11.00      0.08     0.00   0.00    0.73     7.27   36.00
   0.09     4.00  10.00    0.50     2.44    0.00      0.00     0.00   0.00
   0.00     0.00   18.00    0.72    0.04   3.20
sdh              5.00      0.03     0.00   0.00    0.60     5.60   46.00
   0.10     2.00   4.17    0.59     2.17    0.00      0.00     0.00   0.00
   0.00     0.00   23.00    0.83    0.05   2.80
sdi              7.00      0.04     0.00   0.00    0.43     6.29   36.00
   0.07     1.00   2.70    0.47     2.11    0.00      0.00     0.00   0.00
   0.00     0.00   18.00    0.61    0.03   2.40
sdj              5.00      0.04     0.00   0.00    0.80     7.20   42.00
   0.09     1.00   2.33    0.67     2.10    0.00      0.00     0.00   0.00
   0.00     0.00   21.00    0.81    0.05   3.20

------------------------------
Other than this 5x node cluster, I also have a 3x node cluster with
identical hardware but it serves for a different purpose and data workload.
In this cluster I don't have any problem and MDS default settings seems
enough.
The only difference between two cluster is, 5x node cluster used directly
by users, 3x node cluster used heavily to read and write data via projects
not by users. So allocate and de-allocate will be better.

I guess I just have a problematic use case on the 5x node cluster and as I
mentioned above, I might have the similar problem but I don't know how to
debug it.

https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/YO4SGL4DJQ6EKUBUIHKTFSW72ZJ3XLZS/
quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic
caps recall eventually starts recalling those but the (el7 kernel) client
won't release them. Stopping Codium seems to be the only way to release."

------------------------------
Before reading the osd df you should know that I created 2x
OSD/per"CT4000MX500SSD1"
# ceph osd df tree
ID   CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE  DATA     OMAP      META
    AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
 -1         145.54321         -  146 TiB   45 TiB   44 TiB   119 GiB  333
GiB  101 TiB  30.81  1.00    -          root default
 -3          29.10864         -   29 TiB  8.9 TiB  8.8 TiB    25 GiB   66
GiB   20 TiB  30.54  0.99    -              host ud-01
  0    ssd    1.81929   1.00000  1.8 TiB  616 GiB  610 GiB   1.4 GiB  4.5
GiB  1.2 TiB  33.04  1.07   61      up          osd.0
  1    ssd    1.81929   1.00000  1.8 TiB  527 GiB  521 GiB   1.5 GiB  4.0
GiB  1.3 TiB  28.28  0.92   53      up          osd.1
  2    ssd    1.81929   1.00000  1.8 TiB  595 GiB  589 GiB   2.3 GiB  4.0
GiB  1.2 TiB  31.96  1.04   63      up          osd.2
  3    ssd    1.81929   1.00000  1.8 TiB  527 GiB  521 GiB   1.8 GiB  4.2
GiB  1.3 TiB  28.30  0.92   55      up          osd.3
  4    ssd    1.81929   1.00000  1.8 TiB  525 GiB  520 GiB   1.3 GiB  3.9
GiB  1.3 TiB  28.21  0.92   52      up          osd.4
  5    ssd    1.81929   1.00000  1.8 TiB  592 GiB  586 GiB   1.8 GiB  3.8
GiB  1.2 TiB  31.76  1.03   61      up          osd.5
  6    ssd    1.81929   1.00000  1.8 TiB  559 GiB  553 GiB   1.8 GiB  4.3
GiB  1.3 TiB  30.03  0.97   57      up          osd.6
  7    ssd    1.81929   1.00000  1.8 TiB  602 GiB  597 GiB   836 MiB  4.4
GiB  1.2 TiB  32.32  1.05   58      up          osd.7
  8    ssd    1.81929   1.00000  1.8 TiB  614 GiB  609 GiB   1.2 GiB  4.5
GiB  1.2 TiB  32.98  1.07   60      up          osd.8
  9    ssd    1.81929   1.00000  1.8 TiB  571 GiB  565 GiB   2.2 GiB  4.2
GiB  1.3 TiB  30.67  1.00   61      up          osd.9
 10    ssd    1.81929   1.00000  1.8 TiB  528 GiB  522 GiB   1.3 GiB  4.1
GiB  1.3 TiB  28.33  0.92   52      up          osd.10
 11    ssd    1.81929   1.00000  1.8 TiB  551 GiB  546 GiB   1.5 GiB  3.6
GiB  1.3 TiB  29.57  0.96   56      up          osd.11
 12    ssd    1.81929   1.00000  1.8 TiB  594 GiB  588 GiB   1.8 GiB  4.4
GiB  1.2 TiB  31.91  1.04   61      up          osd.12
 13    ssd    1.81929   1.00000  1.8 TiB  561 GiB  555 GiB   1.1 GiB  4.3
GiB  1.3 TiB  30.10  0.98   55      up          osd.13
 14    ssd    1.81929   1.00000  1.8 TiB  616 GiB  609 GiB   1.9 GiB  4.2
GiB  1.2 TiB  33.04  1.07   64      up          osd.14
 15    ssd    1.81929   1.00000  1.8 TiB  525 GiB  520 GiB   1.1 GiB  4.0
GiB  1.3 TiB  28.20  0.92   51      up          osd.15
 -5          29.10864         -   29 TiB  9.0 TiB  8.9 TiB    22 GiB   67
GiB   20 TiB  30.89  1.00    -              host ud-02
 16    ssd    1.81929   1.00000  1.8 TiB  617 GiB  611 GiB   1.7 GiB  4.7
GiB  1.2 TiB  33.12  1.08   63      up          osd.16
 17    ssd    1.81929   1.00000  1.8 TiB  582 GiB  577 GiB   1.6 GiB  4.0
GiB  1.3 TiB  31.26  1.01   59      up          osd.17
 18    ssd    1.81929   1.00000  1.8 TiB  583 GiB  578 GiB   418 MiB  4.0
GiB  1.3 TiB  31.29  1.02   54      up          osd.18
 19    ssd    1.81929   1.00000  1.8 TiB  550 GiB  544 GiB   1.5 GiB  4.0
GiB  1.3 TiB  29.50  0.96   56      up          osd.19
 20    ssd    1.81929   1.00000  1.8 TiB  551 GiB  546 GiB   1.1 GiB  4.1
GiB  1.3 TiB  29.57  0.96   54      up          osd.20
 21    ssd    1.81929   1.00000  1.8 TiB  616 GiB  610 GiB   1.3 GiB  4.4
GiB  1.2 TiB  33.04  1.07   60      up          osd.21
 22    ssd    1.81929   1.00000  1.8 TiB  573 GiB  567 GiB   1.6 GiB  4.1
GiB  1.3 TiB  30.75  1.00   58      up          osd.22
 23    ssd    1.81929   1.00000  1.8 TiB  616 GiB  610 GiB   1.3 GiB  4.3
GiB  1.2 TiB  33.06  1.07   60      up          osd.23
 24    ssd    1.81929   1.00000  1.8 TiB  539 GiB  534 GiB   844 MiB  3.8
GiB  1.3 TiB  28.92  0.94   51      up          osd.24
 25    ssd    1.81929   1.00000  1.8 TiB  583 GiB  576 GiB   2.1 GiB  4.1
GiB  1.3 TiB  31.27  1.02   61      up          osd.25
 26    ssd    1.81929   1.00000  1.8 TiB  617 GiB  611 GiB   1.3 GiB  4.6
GiB  1.2 TiB  33.12  1.08   61      up          osd.26
 27    ssd    1.81929   1.00000  1.8 TiB  537 GiB  532 GiB   1.2 GiB  4.1
GiB  1.3 TiB  28.84  0.94   53      up          osd.27
 28    ssd    1.81929   1.00000  1.8 TiB  527 GiB  522 GiB   1.3 GiB  4.2
GiB  1.3 TiB  28.29  0.92   53      up          osd.28
 29    ssd    1.81929   1.00000  1.8 TiB  594 GiB  588 GiB   1.5 GiB  4.6
GiB  1.2 TiB  31.91  1.04   59      up          osd.29
 30    ssd    1.81929   1.00000  1.8 TiB  528 GiB  523 GiB   1.4 GiB  4.1
GiB  1.3 TiB  28.35  0.92   53      up          osd.30
 31    ssd    1.81929   1.00000  1.8 TiB  594 GiB  589 GiB   1.6 GiB  3.8
GiB  1.2 TiB  31.89  1.03   61      up          osd.31
 -7          29.10864         -   29 TiB  8.9 TiB  8.8 TiB    23 GiB   67
GiB   20 TiB  30.66  1.00    -              host ud-03
 32    ssd    1.81929   1.00000  1.8 TiB  593 GiB  588 GiB   1.1 GiB  4.3
GiB  1.2 TiB  31.84  1.03   57      up          osd.32
 33    ssd    1.81929   1.00000  1.8 TiB  617 GiB  611 GiB   1.8 GiB  4.4
GiB  1.2 TiB  33.13  1.08   63      up          osd.33
 34    ssd    1.81929   1.00000  1.8 TiB  537 GiB  532 GiB   2.0 GiB  3.8
GiB  1.3 TiB  28.84  0.94   59      up          osd.34
 35    ssd    1.81929   1.00000  1.8 TiB  562 GiB  556 GiB   1.7 GiB  4.2
GiB  1.3 TiB  30.16  0.98   58      up          osd.35
 36    ssd    1.81929   1.00000  1.8 TiB  529 GiB  523 GiB   1.3 GiB  3.9
GiB  1.3 TiB  28.38  0.92   52      up          osd.36
 37    ssd    1.81929   1.00000  1.8 TiB  527 GiB  521 GiB   1.7 GiB  4.2
GiB  1.3 TiB  28.28  0.92   55      up          osd.37
 38    ssd    1.81929   1.00000  1.8 TiB  574 GiB  568 GiB   1.2 GiB  4.3
GiB  1.3 TiB  30.79  1.00   55      up          osd.38
 39    ssd    1.81929   1.00000  1.8 TiB  605 GiB  599 GiB   1.6 GiB  4.2
GiB  1.2 TiB  32.48  1.05   61      up          osd.39
 40    ssd    1.81929   1.00000  1.8 TiB  573 GiB  567 GiB   1.2 GiB  4.4
GiB  1.3 TiB  30.76  1.00   56      up          osd.40
 41    ssd    1.81929   1.00000  1.8 TiB  526 GiB  520 GiB   1.7 GiB  3.9
GiB  1.3 TiB  28.21  0.92   54      up          osd.41
 42    ssd    1.81929   1.00000  1.8 TiB  613 GiB  608 GiB  1010 MiB  4.4
GiB  1.2 TiB  32.91  1.07   58      up          osd.42
 43    ssd    1.81929   1.00000  1.8 TiB  606 GiB  600 GiB   1.7 GiB  4.3
GiB  1.2 TiB  32.51  1.06   61      up          osd.43
 44    ssd    1.81929   1.00000  1.8 TiB  583 GiB  577 GiB   1.6 GiB  4.2
GiB  1.3 TiB  31.29  1.02   60      up          osd.44
 45    ssd    1.81929   1.00000  1.8 TiB  618 GiB  613 GiB   1.4 GiB  4.3
GiB  1.2 TiB  33.18  1.08   62      up          osd.45
 46    ssd    1.81929   1.00000  1.8 TiB  550 GiB  544 GiB   1.5 GiB  4.2
GiB  1.3 TiB  29.50  0.96   54      up          osd.46
 47    ssd    1.81929   1.00000  1.8 TiB  526 GiB  522 GiB   692 MiB  3.7
GiB  1.3 TiB  28.25  0.92   50      up          osd.47
 -9          29.10864         -   29 TiB  9.0 TiB  8.9 TiB    26 GiB   68
GiB   20 TiB  31.04  1.01    -              host ud-04
 48    ssd    1.81929   1.00000  1.8 TiB  540 GiB  534 GiB   2.2 GiB  3.6
GiB  1.3 TiB  28.96  0.94   58      up          osd.48
 49    ssd    1.81929   1.00000  1.8 TiB  617 GiB  611 GiB   1.4 GiB  4.5
GiB  1.2 TiB  33.11  1.07   61      up          osd.49
 50    ssd    1.81929   1.00000  1.8 TiB  618 GiB  612 GiB   1.2 GiB  4.8
GiB  1.2 TiB  33.17  1.08   61      up          osd.50
 51    ssd    1.81929   1.00000  1.8 TiB  618 GiB  612 GiB   1.5 GiB  4.5
GiB  1.2 TiB  33.19  1.08   61      up          osd.51
 52    ssd    1.81929   1.00000  1.8 TiB  526 GiB  521 GiB   1.4 GiB  4.1
GiB  1.3 TiB  28.25  0.92   53      up          osd.52
 53    ssd    1.81929   1.00000  1.8 TiB  618 GiB  611 GiB   2.4 GiB  4.3
GiB  1.2 TiB  33.17  1.08   66      up          osd.53
 54    ssd    1.81929   1.00000  1.8 TiB  550 GiB  544 GiB   1.5 GiB  4.3
GiB  1.3 TiB  29.54  0.96   55      up          osd.54
 55    ssd    1.81929   1.00000  1.8 TiB  527 GiB  522 GiB   1.3 GiB  4.0
GiB  1.3 TiB  28.29  0.92   52      up          osd.55
 56    ssd    1.81929   1.00000  1.8 TiB  525 GiB  519 GiB   1.2 GiB  4.1
GiB  1.3 TiB  28.16  0.91   52      up          osd.56
 57    ssd    1.81929   1.00000  1.8 TiB  615 GiB  609 GiB   2.3 GiB  4.2
GiB  1.2 TiB  33.03  1.07   65      up          osd.57
 58    ssd    1.81929   1.00000  1.8 TiB  527 GiB  522 GiB   1.6 GiB  3.7
GiB  1.3 TiB  28.31  0.92   55      up          osd.58
 59    ssd    1.81929   1.00000  1.8 TiB  615 GiB  609 GiB   1.2 GiB  4.6
GiB  1.2 TiB  33.01  1.07   60      up          osd.59
 60    ssd    1.81929   1.00000  1.8 TiB  594 GiB  588 GiB   1.2 GiB  4.4
GiB  1.2 TiB  31.88  1.03   59      up          osd.60
 61    ssd    1.81929   1.00000  1.8 TiB  616 GiB  610 GiB   1.9 GiB  4.1
GiB  1.2 TiB  33.04  1.07   64      up          osd.61
 62    ssd    1.81929   1.00000  1.8 TiB  620 GiB  614 GiB   1.9 GiB  4.4
GiB  1.2 TiB  33.27  1.08   63      up          osd.62
 63    ssd    1.81929   1.00000  1.8 TiB  527 GiB  522 GiB   1.5 GiB  4.0
GiB  1.3 TiB  28.30  0.92   53      up          osd.63
-11          29.10864         -   29 TiB  9.0 TiB  8.9 TiB    23 GiB   65
GiB   20 TiB  30.91  1.00    -              host ud-05
 64    ssd    1.81929   1.00000  1.8 TiB  608 GiB  601 GiB   2.3 GiB  4.5
GiB  1.2 TiB  32.62  1.06   65      up          osd.64
 65    ssd    1.81929   1.00000  1.8 TiB  606 GiB  601 GiB   628 MiB  4.2
GiB  1.2 TiB  32.53  1.06   57      up          osd.65
 66    ssd    1.81929   1.00000  1.8 TiB  583 GiB  578 GiB   1.3 GiB  4.3
GiB  1.2 TiB  31.31  1.02   57      up          osd.66
 67    ssd    1.81929   1.00000  1.8 TiB  537 GiB  533 GiB   436 MiB  3.6
GiB  1.3 TiB  28.82  0.94   50      up          osd.67
 68    ssd    1.81929   1.00000  1.8 TiB  541 GiB  535 GiB   2.5 GiB  3.8
GiB  1.3 TiB  29.04  0.94   59      up          osd.68
 69    ssd    1.81929   1.00000  1.8 TiB  606 GiB  601 GiB   1.1 GiB  4.4
GiB  1.2 TiB  32.55  1.06   59      up          osd.69
 70    ssd    1.81929   1.00000  1.8 TiB  604 GiB  598 GiB   1.8 GiB  4.1
GiB  1.2 TiB  32.44  1.05   63      up          osd.70
 71    ssd    1.81929   1.00000  1.8 TiB  606 GiB  600 GiB   1.9 GiB  4.5
GiB  1.2 TiB  32.53  1.06   62      up          osd.71
 72    ssd    1.81929   1.00000  1.8 TiB  602 GiB  598 GiB   612 MiB  4.1
GiB  1.2 TiB  32.33  1.05   57      up          osd.72
 73    ssd    1.81929   1.00000  1.8 TiB  571 GiB  565 GiB   1.8 GiB  4.5
GiB  1.3 TiB  30.65  0.99   58      up          osd.73
 74    ssd    1.81929   1.00000  1.8 TiB  608 GiB  602 GiB   1.8 GiB  4.2
GiB  1.2 TiB  32.62  1.06   61      up          osd.74
 75    ssd    1.81929   1.00000  1.8 TiB  536 GiB  531 GiB   1.9 GiB  3.5
GiB  1.3 TiB  28.80  0.93   57      up          osd.75
 76    ssd    1.81929   1.00000  1.8 TiB  605 GiB  599 GiB   1.4 GiB  4.5
GiB  1.2 TiB  32.48  1.05   60      up          osd.76
 77    ssd    1.81929   1.00000  1.8 TiB  537 GiB  532 GiB   1.2 GiB  3.9
GiB  1.3 TiB  28.84  0.94   52      up          osd.77
 78    ssd    1.81929   1.00000  1.8 TiB  525 GiB  520 GiB   1.3 GiB  3.8
GiB  1.3 TiB  28.20  0.92   52      up          osd.78
 79    ssd    1.81929   1.00000  1.8 TiB  536 GiB  531 GiB   1.1 GiB  3.3
GiB  1.3 TiB  28.76  0.93   53      up          osd.79
                          TOTAL  146 TiB   45 TiB   44 TiB   119 GiB  333
GiB  101 TiB  30.81
MIN/MAX VAR: 0.91/1.08  STDDEV: 1.90



Eugen Block <eblock@xxxxxx>, 25 Oca 2024 Per, 16:52 tarihinde şunu yazdı:

There is no definitive answer wrt mds tuning. As it is everywhere
mentioned, it's about finding the right setup for your specific
workload. If you can synthesize your workload (maybe scale down a bit)
try optimizing it in a test cluster without interrupting your
developers too much.
But what you haven't explained yet is what are you experiencing as a
performance issue? Do you have numbers or a detailed description?
 From the fs status output you didn't seem to have too much activity
going on (around 140 requests per second), but that's probably not the
usual traffic? What does ceph report in its client IO output?
Can you paste the 'ceph osd df' output as well?
Do you have dedicated MDS servers or are they colocated with other
services?

Zitat von Özkan Göksu <ozkangksu@xxxxxxxxx>:

> Hello  Eugen.
>
> I read all of your MDS related topics and thank you so much for your
effort
> on this.
> There is not much information and I couldn't find a MDS tuning guide at
> all. It  seems that you are the correct person to discuss mds debugging
and
> tuning.
>
> Do you have any documents or may I learn what is the proper way to debug
> MDS and clients ?
> Which debug logs will guide me to understand the limitations and will
help
> to tune according to the data flow?
>
> While searching, I find this:
>
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/YO4SGL4DJQ6EKUBUIHKTFSW72ZJ3XLZS/
> quote:"A user running VSCodium, keeping 15k caps open.. the opportunistic
> caps recall eventually starts recalling those but the (el7 kernel) client
> won't release them. Stopping Codium seems to be the only way to release."
>
> Because of this I think I also need to play around with the client side
too.
>
> My main goal is increasing the speed and reducing the latency and I
wonder
> if these ideas are correct or not:
> - Maybe I need to increase client side cache size because via each
client,
> multiple users request a lot of objects and clearly the
> client_cache_size=16 default is not enough.
> -  Maybe I need to increase client side maximum cache limit for
> object "client_oc_max_objects=1000 to 10000" and data
"client_oc_size=200mi
> to 400mi"
> - The client cache cleaning threshold is not aggressive enough to keep
the
> free cache size in the desired range. I need to make it aggressive but
this
> should not reduce speed and increase latency.
>
> mds_cache_memory_limit=4gi to 16gi
> client_oc_max_objects=1000 to 10000
> client_oc_size=200mi to 400mi
> client_permissions=false #to reduce latency.
> client_cache_size=16 to 128
>
>
> What do you think?






_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux