Hi, Just to update the mailing list, we ended up going back to default ceph.conf without any additional settings than what is mandatory. We are now reaching speeds we never reached before, both in recovery and in regular usage. There was definitely something we set in the ceph.conf bogging everything down. On 2015-08-20 4:06 AM, Christian Balzer wrote: > > Hello, > > from all the pertinent points by Somnath, the one about pre-conditioning > would be pretty high on my list, especially if this slowness persists and > nothing else (scrub) is going on. > > This might be "fixed" by doing a fstrim. > > Additionally the levelDB's per OSD are of course sync'ing heavily during > reconstruction, so that might not be the favorite thing for your type of > SSDs. > > But ultimately situational awareness is very important, as in "what" is > actually going and slowing things down. > As usual my recommendations would be to use atop, iostat or similar on all > your nodes and see if your OSD SSDs are indeed the bottleneck or if it is > maybe just one of them or something else entirely. > > Christian > > On Wed, 19 Aug 2015 20:54:11 +0000 Somnath Roy wrote: > >> Also, check if scrubbing started in the cluster or not. That may >> considerably slow down the cluster. >> >> -----Original Message----- >> From: Somnath Roy >> Sent: Wednesday, August 19, 2015 1:35 PM >> To: 'J-P Methot'; ceph-users@xxxxxxxx >> Subject: RE: Bad performances in recovery >> >> All the writes will go through the journal. >> It may happen your SSDs are not preconditioned well and after a lot of >> writes during recovery IOs are stabilized to lower number. This is quite >> common for SSDs if that is the case. >> >> Thanks & Regards >> Somnath >> >> -----Original Message----- >> From: J-P Methot [mailto:jpmethot@xxxxxxxxxx] >> Sent: Wednesday, August 19, 2015 1:03 PM >> To: Somnath Roy; ceph-users@xxxxxxxx >> Subject: Re: Bad performances in recovery >> >> Hi, >> >> Thank you for the quick reply. However, we do have those exact settings >> for recovery and it still strongly affects client io. I have looked at >> various ceph logs and osd logs and nothing is out of the ordinary. >> Here's an idea though, please tell me if I am wrong. >> >> We use intel SSDs for journaling and samsung SSDs as proper OSDs. As was >> explained several times on this mailing list, Samsung SSDs suck in ceph. >> They have horrible O_dsync speed and die easily, when used as journal. >> That's why we're using Intel ssds for journaling, so that we didn't end >> up putting 96 samsung SSDs in the trash. >> >> In recovery though, what is the ceph behaviour? What kind of write does >> it do on the OSD SSDs? Does it write directly to the SSDs or through the >> journal? >> >> Additionally, something else we notice: the ceph cluster is MUCH slower >> after recovery than before. Clearly there is a bottleneck somewhere and >> that bottleneck does not get cleared up after the recovery is done. >> >> >> On 2015-08-19 3:32 PM, Somnath Roy wrote: >>> If you are concerned about *client io performance* during recovery, >>> use these settings.. >>> >>> osd recovery max active = 1 >>> osd max backfills = 1 >>> osd recovery threads = 1 >>> osd recovery op priority = 1 >>> >>> If you are concerned about *recovery performance*, you may want to >>> bump this up, but I doubt it will help much from default settings.. >>> >>> Thanks & Regards >>> Somnath >>> >>> -----Original Message----- >>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf >>> Of J-P Methot >>> Sent: Wednesday, August 19, 2015 12:17 PM >>> To: ceph-users@xxxxxxxx >>> Subject: Bad performances in recovery >>> >>> Hi, >>> >>> Our setup is currently comprised of 5 OSD nodes with 12 OSD each, for >>> a total of 60 OSDs. All of these are SSDs with 4 SSD journals on each. >>> The ceph version is hammer v0.94.1 . There is a performance overhead >>> because we're using SSDs (I've heard it gets better in infernalis, but >>> we're not upgrading just yet) but we can reach numbers that I would >>> consider "alright". >>> >>> Now, the issue is, when the cluster goes into recovery it's very fast >>> at first, but then slows down to ridiculous levels as it moves >>> forward. You can go from 7% to 2% to recover in ten minutes, but it >>> may take 2 hours to recover the last 2%. While this happens, the >>> attached openstack setup becomes incredibly slow, even though there is >>> only a small fraction of objects still recovering (less than 1%). The >>> settings that may affect recovery speed are very low, as they are by >>> default, yet they still affect client io speed way more than it should. >>> >>> Why would ceph recovery become so slow as it progress and affect >>> client io even though it's recovering at a snail's pace? And by a >>> snail's pace, I mean a few kb/second on 10gbps uplinks. -- >>> ====================== Jean-Philippe Méthot >>> Administrateur système / System administrator GloboTech Communications >>> Phone: 1-514-907-0050 >>> Toll Free: 1-(888)-GTCOMM1 >>> Fax: 1-(514)-907-0750 >>> jpmethot@xxxxxxxxxx >>> http://www.gtcomm.net >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> ________________________________ >>> >>> PLEASE NOTE: The information contained in this electronic mail message >>> is intended only for the use of the designated recipient(s) named >>> above. If the reader of this message is not the intended recipient, >>> you are hereby notified that you have received this message in error >>> and that any review, dissemination, distribution, or copying of this >>> message is strictly prohibited. If you have received this >>> communication in error, please notify the sender by telephone or >>> e-mail (as shown above) immediately and destroy any and all copies of >>> this message in your possession (whether hard copies or electronically >>> stored copies). >>> >> >> >> -- >> ====================== >> Jean-Philippe Méthot >> Administrateur système / System administrator GloboTech Communications >> Phone: 1-514-907-0050 >> Toll Free: 1-(888)-GTCOMM1 >> Fax: 1-(514)-907-0750 >> jpmethot@xxxxxxxxxx >> http://www.gtcomm.net >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > -- ====================== Jean-Philippe Méthot Administrateur système / System administrator GloboTech Communications Phone: 1-514-907-0050 Toll Free: 1-(888)-GTCOMM1 Fax: 1-(514)-907-0750 jpmethot@xxxxxxxxxx http://www.gtcomm.net _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |