Re: Ceph newbie(?) issues

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 05. mars 2018 14:45, Jan Marquardt wrote:
Am 05.03.18 um 13:13 schrieb Ronny Aasen:
i had some similar issues when i started my proof of concept. especialy
the snapshot deletion i remember well.

the rule of thumb for filestore that i assume you are running is 1GB ram
per TB of osd. so with 8 x 4TB osd's you are looking at 32GB of ram for
osd's + some  GB's for the mon service, + some GB's  for the os itself.

i suspect if you inspect your dmesg log and memory graphs you will find
that the out of memory killer ends your osd's when the snap deletion (or
any other high load task) runs.

I ended up reducing the number of osd's per node, since the old
mainboard i used was maxed for memory.

Well, thanks for the broad hint. Somehow I assumed we fulfill the
recommendations, but of course you are right. We'll check if our boards
support 48 GB RAM. Unfortunately, there are currently no corresponding
messages. But I can't rule out that there haven't been any.

corruptions occured for me as well. and they was normaly associated with
disks dying or giving read errors. ceph often managed to fix them but
sometimes i had to just remove the hurting OSD disk.

hage some graph's  to look at. personaly i used munin/munin-node since
it was just an apt-get away from functioning graphs

also i used smartmontools to send me emails about hurting disks.
and smartctl to check all disks for errors.

I'll check S.M.A.R.T stuff. I am wondering if scrubbing errors are
always caused by disk problems or if they also could be triggered
by flapping OSDs or other circumstances.

good luck with ceph !

Thank you!


in my not that extensive experience, schrub errors come mainly from 2 issues. Either disk's giving read errors (should be visible both in the log and dmesg.) or having pools with size=2/min_size=1 instead of the default and recomended size=3/min_size=2 but i can not say that they do not come from crashing OSD's but my case the osd kept crashing due to bad disk and/or low memory.




If you have scrub errors you can not get rid of on filestore (not bluestore!) you can read the two following urls.


http://ceph.com/geen-categorie/ceph-manually-repair-object/ and on http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/

basicaly the steps are:

- find the pg      ::  rados list-inconsistent-pg [pool]
- find the problem :: rados list-inconsistent-obj 0.6 --format=json-pretty ; give you the object name look for hints to what is the bad object - find the object on disks :: manually check the objects on each osd for the given pg, check the object metadata (size/date/etc), run md5sum on them all and compare. check objects on the nonrunning osd's and compare there as well. anything to try to determine what object is ok and what is bad. - fix the problem :: assuming you find the bad object, stop the affected osd with the bad object, remove the object manually, restart osd. issue repair command.


Once i fixed my min_size=1 misconfiguration, and pulled the dying (but functional) disks from my cluster, and reduced osd count to prevent dying osd's all of those scrub errors went away. have not seen one in 6 months now.


kinds regards
Ronny Aasen
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux