Thanks for the config, few comments inline:, not really related to the issue > On 21 Aug 2015, at 15:12, J-P Methot <jpmethot@xxxxxxxxxx> wrote: > > Hi, > > First of all, we are sure that the return to the default configuration > fixed it. As soon as we restarted only one of the ceph nodes with the > default configuration, it sped up recovery tremedously. We had already > restarted before with the old conf and recovery was never that fast. > > Regarding the configuration, here's the old one with comments : > > [global] > fsid = ************************* > mon_initial_members = cephmon1 > mon_host = ******************* > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > filestore_xattr_use_omap = true // > Let's you use xattributes of xfs/ext4/btrfs filesystems This actually did the opposite, but this option doesn't exist anymore > osd_pool_default_pgp_num = 450 // > default pgp number for new pools > osd_pg_bits = 12 // > number of bits used to designate pgps. Lets you have 2^12 pgps Could someone comment on those? What exactly does it do? What if I have more PGs than num_osds*osd_pg_bits? > osd_pool_default_size = 3 // > default copy number for new pools > osd_pool_default_pg_num = 450 // > default pg number for new pools > public_network = ************* > cluster_network = *************** > osd_pgp_bits = 12 // > number of bits used to designate pgps. Let's you have 2^12 pgps > > [osd] > filestore_queue_max_ops = 5000 // set to 500 by default Defines the > maximum number of in progress operations the file store accepts before > blocking on queuing new operations. > filestore_fd_cache_random = true // ???? No docs, I don't see this in my ancient cluster :-) > journal_queue_max_ops = 1000000 // set > to 500 by default. Number of operations allowed in the journal queue > filestore_omap_header_cache_size = 1000000 // Determines > the size of the LRU used to cache object omap headers. Larger values use > more memory but may reduce lookups on omap. > filestore_fd_cache_size = 1000000 // You don't really need to set this so high, but not sure what the implications are if you go too high (it probably doesn't eat more memory until it opens so many files). If you have 4MB object on a 1TB drive than you really only need 250K to keep all files open. > not in the ceph documentation. Seems to be a common tweak for SSD > clusters though. > max_open_files = 1000000 // > lets ceph set the max file descriptor in the OS to prevent running out > of file descriptors This is too low if you were really using all of the fd_cache. There are going to be thousands of tcp connection which need to be accounted for as well. (in my experience there can be hundreds to thousands tcp connection from just one RBD client and 200 OSDs, which is a lot). > osd_journal_size = 10000 // > journal max size for each OSD > > New conf: > > [global] > fsid = ************************* > mon_initial_members = cephmon1 > mon_host = ************ > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > public_network = ****************** > cluster_network = ****************** > > You might notice, I have a few undocumented settings in the old > configuration. These are settings I took from a certain openstack summit > presentation and they may have contributed to this whole problem. Here's > a list of settings that I think might be a possible cause for these > speed issues: > > filestore_fd_cache_random = true > filestore_fd_cache_size = 1000000 > > Additionally, my colleague thinks these settings may have contributed : > > filestore_queue_max_ops = 5000 > journal_queue_max_ops = 1000000 > > We will do further tests on these settings once we have our lab ceph > test environment as we are also curious as to exactly what caused this. > > > On 2015-08-20 11:43 AM, Alex Gorbachev wrote: >>> >>> Just to update the mailing list, we ended up going back to default >>> ceph.conf without any additional settings than what is mandatory. We are >>> now reaching speeds we never reached before, both in recovery and in >>> regular usage. There was definitely something we set in the ceph.conf >>> bogging everything down. >> >> Could you please share the old and new ceph.conf, or the section that >> was removed? >> >> Best regards, >> Alex >> >>> >>> >>> On 2015-08-20 4:06 AM, Christian Balzer wrote: >>>> >>>> Hello, >>>> >>>> from all the pertinent points by Somnath, the one about pre-conditioning >>>> would be pretty high on my list, especially if this slowness persists and >>>> nothing else (scrub) is going on. >>>> >>>> This might be "fixed" by doing a fstrim. >>>> >>>> Additionally the levelDB's per OSD are of course sync'ing heavily during >>>> reconstruction, so that might not be the favorite thing for your type of >>>> SSDs. >>>> >>>> But ultimately situational awareness is very important, as in "what" is >>>> actually going and slowing things down. >>>> As usual my recommendations would be to use atop, iostat or similar on all >>>> your nodes and see if your OSD SSDs are indeed the bottleneck or if it is >>>> maybe just one of them or something else entirely. >>>> >>>> Christian >>>> >>>> On Wed, 19 Aug 2015 20:54:11 +0000 Somnath Roy wrote: >>>> >>>>> Also, check if scrubbing started in the cluster or not. That may >>>>> considerably slow down the cluster. >>>>> >>>>> -----Original Message----- >>>>> From: Somnath Roy >>>>> Sent: Wednesday, August 19, 2015 1:35 PM >>>>> To: 'J-P Methot'; ceph-users@xxxxxxxx >>>>> Subject: RE: Bad performances in recovery >>>>> >>>>> All the writes will go through the journal. >>>>> It may happen your SSDs are not preconditioned well and after a lot of >>>>> writes during recovery IOs are stabilized to lower number. This is quite >>>>> common for SSDs if that is the case. >>>>> >>>>> Thanks & Regards >>>>> Somnath >>>>> >>>>> -----Original Message----- >>>>> From: J-P Methot [mailto:jpmethot@xxxxxxxxxx] >>>>> Sent: Wednesday, August 19, 2015 1:03 PM >>>>> To: Somnath Roy; ceph-users@xxxxxxxx >>>>> Subject: Re: Bad performances in recovery >>>>> >>>>> Hi, >>>>> >>>>> Thank you for the quick reply. However, we do have those exact settings >>>>> for recovery and it still strongly affects client io. I have looked at >>>>> various ceph logs and osd logs and nothing is out of the ordinary. >>>>> Here's an idea though, please tell me if I am wrong. >>>>> >>>>> We use intel SSDs for journaling and samsung SSDs as proper OSDs. As was >>>>> explained several times on this mailing list, Samsung SSDs suck in ceph. >>>>> They have horrible O_dsync speed and die easily, when used as journal. >>>>> That's why we're using Intel ssds for journaling, so that we didn't end >>>>> up putting 96 samsung SSDs in the trash. >>>>> >>>>> In recovery though, what is the ceph behaviour? What kind of write does >>>>> it do on the OSD SSDs? Does it write directly to the SSDs or through the >>>>> journal? >>>>> >>>>> Additionally, something else we notice: the ceph cluster is MUCH slower >>>>> after recovery than before. Clearly there is a bottleneck somewhere and >>>>> that bottleneck does not get cleared up after the recovery is done. >>>>> >>>>> >>>>> On 2015-08-19 3:32 PM, Somnath Roy wrote: >>>>>> If you are concerned about *client io performance* during recovery, >>>>>> use these settings.. >>>>>> >>>>>> osd recovery max active = 1 >>>>>> osd max backfills = 1 >>>>>> osd recovery threads = 1 >>>>>> osd recovery op priority = 1 >>>>>> >>>>>> If you are concerned about *recovery performance*, you may want to >>>>>> bump this up, but I doubt it will help much from default settings.. >>>>>> >>>>>> Thanks & Regards >>>>>> Somnath >>>>>> >>>>>> -----Original Message----- >>>>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf >>>>>> Of J-P Methot >>>>>> Sent: Wednesday, August 19, 2015 12:17 PM >>>>>> To: ceph-users@xxxxxxxx >>>>>> Subject: Bad performances in recovery >>>>>> >>>>>> Hi, >>>>>> >>>>>> Our setup is currently comprised of 5 OSD nodes with 12 OSD each, for >>>>>> a total of 60 OSDs. All of these are SSDs with 4 SSD journals on each. >>>>>> The ceph version is hammer v0.94.1 . There is a performance overhead >>>>>> because we're using SSDs (I've heard it gets better in infernalis, but >>>>>> we're not upgrading just yet) but we can reach numbers that I would >>>>>> consider "alright". >>>>>> >>>>>> Now, the issue is, when the cluster goes into recovery it's very fast >>>>>> at first, but then slows down to ridiculous levels as it moves >>>>>> forward. You can go from 7% to 2% to recover in ten minutes, but it >>>>>> may take 2 hours to recover the last 2%. While this happens, the >>>>>> attached openstack setup becomes incredibly slow, even though there is >>>>>> only a small fraction of objects still recovering (less than 1%). The >>>>>> settings that may affect recovery speed are very low, as they are by >>>>>> default, yet they still affect client io speed way more than it should. >>>>>> >>>>>> Why would ceph recovery become so slow as it progress and affect >>>>>> client io even though it's recovering at a snail's pace? And by a >>>>>> snail's pace, I mean a few kb/second on 10gbps uplinks. -- >>>>>> ====================== Jean-Philippe Méthot >>>>>> Administrateur système / System administrator GloboTech Communications >>>>>> Phone: 1-514-907-0050 >>>>>> Toll Free: 1-(888)-GTCOMM1 >>>>>> Fax: 1-(514)-907-0750 >>>>>> jpmethot@xxxxxxxxxx >>>>>> http://www.gtcomm.net >>>>>> _______________________________________________ >>>>>> ceph-users mailing list >>>>>> ceph-users@xxxxxxxxxxxxxx >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>>> >>>>>> ________________________________ >>>>>> >>>>>> PLEASE NOTE: The information contained in this electronic mail message >>>>>> is intended only for the use of the designated recipient(s) named >>>>>> above. If the reader of this message is not the intended recipient, >>>>>> you are hereby notified that you have received this message in error >>>>>> and that any review, dissemination, distribution, or copying of this >>>>>> message is strictly prohibited. If you have received this >>>>>> communication in error, please notify the sender by telephone or >>>>>> e-mail (as shown above) immediately and destroy any and all copies of >>>>>> this message in your possession (whether hard copies or electronically >>>>>> stored copies). >>>>>> >>>>> >>>>> >>>>> -- >>>>> ====================== >>>>> Jean-Philippe Méthot >>>>> Administrateur système / System administrator GloboTech Communications >>>>> Phone: 1-514-907-0050 >>>>> Toll Free: 1-(888)-GTCOMM1 >>>>> Fax: 1-(514)-907-0750 >>>>> jpmethot@xxxxxxxxxx >>>>> http://www.gtcomm.net >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>>> >>> >>> >>> -- >>> ====================== >>> Jean-Philippe Méthot >>> Administrateur système / System administrator >>> GloboTech Communications >>> Phone: 1-514-907-0050 >>> Toll Free: 1-(888)-GTCOMM1 >>> Fax: 1-(514)-907-0750 >>> jpmethot@xxxxxxxxxx >>> http://www.gtcomm.net >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -- > ====================== > Jean-Philippe Méthot > Administrateur système / System administrator > GloboTech Communications > Phone: 1-514-907-0050 > Toll Free: 1-(888)-GTCOMM1 > Fax: 1-(514)-907-0750 > jpmethot@xxxxxxxxxx > http://www.gtcomm.net > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |