Re: High RAM usage in OSD servers

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Can it be this bug :  http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-October/021676.html

In most of the OSDs buffer anon is high
  },
    "buffer_anon": {
        "items": 268443,
        "bytes": 1421912265


Karun Josy

On Sun, Feb 4, 2018 at 7:03 AM, Karun Josy <karunjosy1@xxxxxxxxx> wrote:
And can see this in error log :

Feb  2 16:41:28 ceph-las1-a4-osd kernel: bstore_kv_sync: page allocation stalls for 14188ms, order:0, mode:0x14280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=(null)


Karun Josy

On Sun, Feb 4, 2018 at 6:19 AM, Karun Josy <karunjosy1@xxxxxxxxx> wrote:
Hi,

We are using EC profile in our cluster.
We are seeing very high RAM usage in 1 OSD server.
Sometimes it goes too low and server hangs. We have to restart the daemons which frees up the memory, but in very short time get used up again

Memory usage of daemons from issue server 
-------------
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND

16918 ceph      20   0 15.780g 0.013t   7928 S  28.2  21.9  67:29.09 ceph-osd
18568 ceph      20   0 25.833g 0.023t  26096 S  24.9 36.8   9:15.58 ceph-osd
22630 ceph      20   0 12.520g 0.011t  26660 S  22.3 18.3   5:49.03 ceph-osd
 2796 ceph      20   0 11.091g 9.851g   8900 S  13.6 15.7  25:17.68 ceph-osd


Memory usage from another server : 
-------------------------------
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
11649 ceph      20   0 12.788g 9.563g  25068 S 107.6  7.6  12285:54 ceph-osd
18295 ceph      20   0 11.028g 6.069g  26212 S  54.0  4.8   2122:18 ceph-osd
30974 ceph      20   0 13.860g 0.010t  24956 S  46.4  8.1  10984:47 ceph-osd


We are using ec profile 5/3. And there are 2 failed disks in the cluster in another nodes, (I have marked them down, but not out) so cannot turn this node off as it will force some pgs to be incomplete state.

And help would be really appreciated. 


Karun Josy


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux