Hi David, i think this should be the problem - form a new log from today: 2018-10-13 20:57:20.367326 mon.a [WRN] Health check update: 4 osds down (OSD_DOWN) ... 2018-10-13 20:57:41.268674 mon.a [WRN] Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) ... 2018-10-13 20:58:08.684451 mon.a [WRN] Health check failed: 1 osds down (OSD_DOWN) ... 2018-10-13 20:58:22.841210 mon.a [WRN] Health check failed: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY) .... 2018-10-13 20:58:47.570017 mon.a [WRN] Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) ... 2018-10-13 20:58:49.142108 osd.19 [WRN] Monitor daemon marked osd.19 down, but it is still running 2018-10-13 20:58:53.750164 mon.a [WRN] Health check update: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) ... so there is a timeframe of > 90s whee PGs are inactive and unavail - this would at least explain stalled I/O to me? Greets, Stefan Am 12.10.2018 um 15:59 schrieb David Turner: > The PGs per OSD does not change unless the OSDs are marked out. You > have noout set, so that doesn't change at all during this test. All of > your PGs peered quickly at the beginning and then were active+undersized > the rest of the time, you never had any blocked requests, and you always > had 100MB/s+ client IO. I didn't see anything wrong with your cluster > to indicate that your clients had any problems whatsoever accessing data. > > Can you confirm that you saw the same problems while you were running > those commands? The next thing would seem that possibly a client isn't > getting an updated OSD map to indicate that the host and its OSDs are > down and it's stuck trying to communicate with host7. That would > indicate a potential problem with the client being unable to communicate > with the Mons maybe? Have you completely ruled out any network problems > between all nodes and all of the IPs in the cluster. What does your > client log show during these times? > > On Fri, Oct 12, 2018 at 8:35 AM Nils Fahldieck - Profihost AG > <n.fahldieck@xxxxxxxxxxxx <mailto:n.fahldieck@xxxxxxxxxxxx>> wrote: > > Hi, in our `ceph.conf` we have: > > mon_max_pg_per_osd = 300 > > While the host is offline (9 OSDs down): > > 4352 PGs * 3 / 62 OSDs ~ 210 PGs per OSD > > If all OSDs are online: > > 4352 PGs * 3 / 71 OSDs ~ 183 PGs per OSD > > ... so this doesn't seem to be the issue. > > If I understood you right, that's what you've meant. If I got you wrong, > would you mind to point to one of those threads you mentioned? > > Thanks :) > > Am 12.10.2018 um 14:03 schrieb Burkhard Linke: > > Hi, > > > > > > On 10/12/2018 01:55 PM, Nils Fahldieck - Profihost AG wrote: > >> I rebooted a Ceph host and logged `ceph status` & `ceph health > detail` > >> every 5 seconds. During this I encountered 'PG_AVAILABILITY > Reduced data > >> availability: pgs peering'. At the same time some VMs hung as > described > >> before. > > > > Just a wild guess... you have 71 OSDs and about 4500 PG with size=3. > > 13500 PG instance overall, resulting in ~190 PGs per OSD under normal > > circumstances. > > > > If one host is down and the PGs have to re-peer, you might reach the > > limit of 200 PG/OSDs on some of the OSDs, resulting in stuck peering. > > > > You can try to raise this limit. There are several threads on the > > mailing list about this. > > > > Regards, > > Burkhard > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |