Re: something wrong with my monitor database ?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




Le 13/06/2022 à 17:54, Eric Le Lay a écrit :
Le 10/06/2022 à 11:58, Stefan Kooman a écrit :
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

On 6/10/22 11:41, Eric Le Lay wrote:
Hello list,

my ceph cluster was upgraded from nautilus to octopus last October,
causing snaptrims
to overload OSDs so I had to disable them (bluefs_buffered_io=false|true
didn't help).

Now I've copied data elsewhere and removed all clients and try to fix
the cluster.
Scraping it and starting over is possible, but it would be wonderful if
we could
figure out what's wrong with it...

FYI: osd snap trim sleep <- adding some sleep might help alleviate the
impact on the cluster.

If HEALTH is OK I would not expect anything wrong with your cluster.

Does " ceph osd dump |grep require_osd_release" give you
require_osd_release octopus?

Gr. Stefan
Hi Stefan,

thank you for your answer.
Even osd_snap_trim_sleep=10 was not sustainable with normal cluster load.| Following your email I've tested bluefs_buffered_io=true again and indeed it dramatically reduces disk load, but not cpu nor slow ceph io.

Yes, require_osd_release=octopus.

What worries me is the pool is now void of rbd images, but still has 14TiB of object data.
Here is my pool contents. rbd_directory, rbd_trash are empty.

   rados -p storage ls | sed 's/\(.*\..*\)\..*/\1/'|sort|uniq -c
      1 rbd_children
      6 rbd_data.13fc0d1d63c52b
   2634 rbd_data.15ab844f62d5
    258 rbd_data.15f1f2e2398dc7
    133 rbd_data.17d93e1c5a4855
    258 rbd_data.1af03e352ec460
   2987 rbd_data.236cfc2474b020
 206872 rbd_data.31c55ee49f0abb
 604593 rbd_data.5b423b48a4643f
     90 rbd_data.7b06b7abcc9441
  81576 rbd_data.913b398f28d1
     18 rbd_data.9662ade11235a
  16051 rbd_data.e01609a7a07e20
    278 rbd_data.e6b6f855b5172c
     90 rbd_data.e85da37e044922
      1 rbd_directory
      1 rbd_info
      1 rbd_trash

Eric

Those objects are deleted but have snapshots, even if the pool itself doesn't have snapshots.
What could cause that?


root@hpc1a:~# rados -p storage stat rbd_data.5b423b48a4643f.000000000006a4e5
 error stat-ing storage/rbd_data.5b423b48a4643f.000000000006a4e5: (2) No such file or directory
root@hpc1a:~# rados -p storage lssnap
0 snaps
root@hpc1a:~# rados -p storage listsnaps rbd_data.5b423b48a4643f.000000000006a4e5
rbd_data.5b423b48a4643f.000000000006a4e5:
cloneid    snaps    size    overlap
1160    1160    4194304 [1048576~32768,1097728~16384,1228800~16384,1409024~16384,1441792~16384,1572864~16384,1720320~16384,1900544~16384,2310144~16384]
1364    1364    4194304    []


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux