If that pool is your metadata pool, it looks at a quick glance like
it's timing out somewhere while reading and building up the omap
contents (ie, the contents of a directory). Which might make sense if,
say, you have very fragmented leveldb stores combined with very large
CephFS directories. Trying to make the leveldbs happier (I think there
are some options to compact on startup, etc?) might help; otherwise
you might be running into the same "too-large omap collections" thing
that Brandon referred to. Which in CephFS can be fixed by either
having smaller folders or (if you're very nervy, and ready to turn on
something we think works but don't test enough) enabling directory
fragmentation.
-Greg
On Wed, Jun 1, 2016 at 2:14 PM, Adam Tygart <mozes@xxxxxxx> wrote:
> I've been attempting to work through this, finding the pgs that are
> causing hangs, determining if they are "safe" to remove, and removing
> them with ceph-objectstore-tool on osd 16.
>
> I'm now getting hangs (followed by suicide timeouts) referencing pgs
> that I've just removed, so this doesn't seem to be all there is to the
> issue.
>
> --
> Adam
>
> On Wed, Jun 1, 2016 at 12:00 PM, Brandon Morris, PMP
> <brandon.morris.pmp@xxxxxxxxx> wrote:
>> Adam,
>>
>> We ran into similar issues when we get too many objects in bucket
>> (around 300 million). The .rgw.buckets.index pool became unable to complete
>> backfill operations. The only way we were able to get past it was to
>> export the offending placement group with the ceph-objectstore-tool and
>> re-import it into another OSD to complete the backfill. For us, the export
>> operation seemed to hang and took 8 hours to export, so if you do choose to
>> go down this route, be patient.
>>
>> From your logs, it appears that pg 32.10c is the offending PG on OSD.16. If
>> you are running into the same issue we did, when you go to export it there
>> will be a file that will hang. For whatever reason the leveldb metadata for
>> that file hangs and causes the backfill operation to suicide the OSD.
>>
>> If anyone from the community has an explanation for why this happens I would
>> love to know. We have run into this twice now on the Infernalis codebase.
>> We are in the process of rebuilding our cluster to Jewel, so can't say
>> whether or not it happens there as well.
>>
>> ---------
>> Here is the pertinent lines from your log.
>>
>> 2016-06-01 09:26:54.683922 7f34c5e41700 7 osd.16 pg_epoch: 497663
>> pg[32.10c( v 477010'1607561 (459778'1604561,477010'1607561] local-les=493771
>> n=3917 ec=44014 les/c/f 493771/486667/0 497332/497662/497662)
>> [214,143,448]/[16] r=0 lpr=497662 pi=483321-497661/190 rops=1
>> bft=143,214,448 crt=0'0 lcod 0'0 mlcod 0'0
>> undersized+degraded+remapped+backfilling+peered] send_push_op
>> 32:30966cd6:::100042c76a0.00000000:head v 250315'1040233 size 0
>> recovery_info:
>> ObjectRecoveryInfo(32:30966cd6:::100042c76a0.00000000:head@250315'1040233,
>> size: 0, copy_subset: [], clone_subset: {})
>> [...]
>> 2016-06-01 09:27:25.091411 7f34cf856700 1 heartbeat_map is_healthy
>> 'OSD::recovery_tp thread 0x7f34c5e41700' had timed out after 30
>> [...]
>> 2016-06-01 09:31:57.201645 7f3510669700 1 heartbeat_map is_healthy
>> 'OSD::recovery_tp thread 0x7f34c5e41700' had timed out after 30
>> 2016-06-01 09:31:57.201671 7f3510669700 1 heartbeat_map is_healthy
>> 'OSD::recovery_tp thread 0x7f34c5e41700' had suicide timed out after 300
>> common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(const
>> ceph::heartbeat_handle_d*, const char*, time_t)' thread 7f3510669700 time
>> 2016-06-01 09:31:57.201687
>> common/HeartbeatMap.cc: 86: FAILED assert(0 == "hit suicide timeout")
>> ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x85) [0x7f35167bb5b5]
>> 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char
>> const*, long)+0x2e1) [0x7f35166f7bf1]
>> 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0x7f35166f844e]
>> 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0x7f35166f8c2c]
>> 5: (CephContextServiceThread::entry()+0x15b) [0x7f35167d331b]
>> 6: (()+0x7dc5) [0x7f35146ecdc5]
>> 7: (clone()+0x6d) [0x7f3512d77ced]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
>> interpret this.
>> 2016-06-01 09:31:57.205990 7f3510669700 -1 common/HeartbeatMap.cc: In
>> function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*,
>> const char*, time_t)' thread 7f3510669700 time 2016-06-01 09:31:57.201687
>> common/HeartbeatMap.cc: 86: FAILED assert(0 == "hit suicide timeout")
>>
>> Brandon
>>
>>
>> On Wed, Jun 1, 2016 at 9:13 AM, Adam Tygart <mozes@xxxxxxx> wrote:
>>>
>>> Hello all,
>>>
>>> I'm running into an issue with ceph osds crashing over the last 4
>>> days. I'm running Jewel (10.2.1) on CentOS 7.2.1511.
>>>
>>> A little setup information:
>>> 26 hosts
>>> 2x 400GB Intel DC P3700 SSDs
>>> 12x6TB spinning disks
>>> 4x4TB spinning disks.
>>>
>>> The SSDs are used for both journals and as an OSD (for the cephfs
>>> metadata pool).
>>>
>>> We were running Ceph with some success in this configuration
>>> (upgrading ceph from hammer to infernalis to jewel) for the past 8-10
>>> months.
>>>
>>> Up through Friday, we were healthy.
>>>
>>> Until Saturday. On Saturday, the OSDs on the SSDs started flapping and
>>> then finally dying off, hitting their suicide timeout due to missing
>>> heartbeats. At the time, we were running Infernalis, getting ready to
>>> upgrade to Jewel.
>>>
>>> I spent the weekend and Monday, attempting to stabilize those OSDs,
>>> unfortunately failing. As part of the stabilzation attempts, I check
>>> iostat -x, the SSDs were seeing 1000 iops each. I checked wear levels,
>>> and overall SMART health of the SSDs, everything looks normal. I
>>> checked to make sure the time was in sync between all hosts.
>>>
>>> I also tried to move the metadata pool to the spinning disks (to
>>> remove some dependence on the SSDs, just in case). The suicide timeout
>>> issues followed the pool migration. The spinning disks started timing
>>> out. This was at a time when *all* of client the IOPs to the ceph
>>> cluster were in the low 100's as reported to by ceph -s. I was
>>> restarting failed OSDs as fast as they were dying and I couldn't keep
>>> up. I checked the switches and NICs for errors and drops. No changes
>>> in the frequency of them. We're talking an error every 20-25 minutes.
>>> I would expect network issues to affect other OSDs (and pools) in the
>>> system, too.
>>>
>>> On Tuesday, I got together with my coworker, and we tried together to
>>> stabilize the cluster. We finally went into emergency maintenance
>>> mode, as we could not get the metadata pool healthy. We stopped the
>>> MDS, we tried again to let things stabilize, with no client IO to the
>>> pool. Again more suicide timeouts.
>>>
>>> Then, we rebooted the ceph nodes, figuring there *might* be something
>>> stuck in a hardware IO queue or cache somewhere. Again more crashes
>>> when the machines came back up.
>>>
>>> We figured at this point, there was nothing to lose by performing the
>>> update to Jewel, and, who knows, maybe we were hitting a bug that had
>>> been fixed. Reboots were involved again (kernel updates, too).
>>>
>>> More crashes.
>>>
>>> I finally decided, that there *could* be an unlikely chance that jumbo
>>> frames might suddenly be an issue (after years of using them with
>>> these switches). I turned down the MTUs on the ceph nodes to the
>>> standard 1500.
>>>
>>> More crashes.
>>>
>>> We decided to try and let things settle out overnight, with no IO.
>>> That brings us to today:
>>>
>>> We have 51 Intel P3700 SSDs driving this pool, and now 26 of them have
>>> crashed due to the suicide timeout. I've tried starting them one at a
>>> time, they're still dying off with suicide timeouts.
>>>
>>> I've gathered the logs I could think to:
>>> A crashing OSD: http://people.cs.ksu.edu/~mozes/osd.16.log
>>> CRUSH Tree: http://people.cs.ksu.edu/~mozes/crushtree.txt
>>> OSD Tree: http://people.cs.ksu.edu/~mozes/osdtree.txt
>>> Pool Definitions: http://people.cs.ksu.edu/~mozes/pools.txt
>>>
>>> At the moment, we're dead in the water. I would appreciate any
>>> pointers to getting this fixed.
>>>
>>> --
>>> Adam Tygart
>>> Beocat Sysadmin
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
