I have often seen a problem where a single osd in an eternal deep scrup
will hang any client trying to connect. Stopping or restarting that
single OSD fixes the problem.
Do you use snapshots?
Here's what the scrub bug looks like (where that many seconds is 14 hours):
> ceph daemon "osd.$osd_number" dump_blocked_ops
> {
> "description": "osd_op(client.6480719.0:2000419292 4.a27969ae
> rbd_data.46820b238e1f29.000000000000aa70 [set-alloc-hint object_size
> 524288 write_size 524288,write 0~4096] snapc 16ec0=[16ec0]
> ack+ondisk+write+known_if_redirected e148441)",
> "initiated_at": "2017-09-12 20:04:27.987814",
> "age": 49315.666393,
> "duration": 49315.668515,
> "type_data": [
> "delayed",
> {
> "client": "client.6480719",
> "tid": 2000419292
> },
> [
> {
> "time": "2017-09-12 20:04:27.987814",
> "event": "initiated"
> },
> {
> "time": "2017-09-12 20:04:27.987862",
> "event": "queued_for_pg"
> },
> {
> "time": "2017-09-12 20:04:28.004142",
> "event": "reached_pg"
> },
> {
> "time": "2017-09-12 20:04:28.004219",
> "event": "waiting for scrub"
> }
> ]
> ]
> }
On 11/09/17 17:20, Matteo Dacrema wrote:
> Update: I noticed that there was a pg that remained scrubbing from the first day I found the issue to when I reboot the node and problem disappeared.
> Can this cause the behaviour I described before?
>
>
>> Il giorno 09 nov 2017, alle ore 15:55, Matteo Dacrema <mdacrema@xxxxxxxx> ha scritto:
>>
>> Hi all,
>>
>> I’ve experienced a strange issue with my cluster.
>> The cluster is composed by 10 HDDs nodes with 20 nodes + 4 journal each plus 4 SSDs nodes with 5 SSDs each.
>> All the nodes are behind 3 monitors and 2 different crush maps.
>> All the cluster is on 10.2.7
>>
>> About 20 days ago I started to notice that long backups hangs with "task jbd2/vdc1-8:555 blocked for more than 120 seconds” on the HDD crush map.
>> About few days ago another VM start to have high iowait without doing iops also on the HDD crush map.
>>
>> Today about a hundreds VMs wasn’t able to read/write from many volumes all of them on HDD crush map. Ceph health was ok and no significant log entries were found.
>> Not all the VMs experienced this problem and in the meanwhile the iops on the journal and HDDs was very low even if I was able to do significant iops on the working VMs.
>>
>> After two hours of debug I decided to reboot one of the OSD nodes and the cluster start to respond again. Now the OSD node is back in the cluster and the problem is disappeared.
>>
>> Can someone help me to understand what happened?
>> I see strange entries in the log files like:
>>
>> accept replacing existing (lossy) channel (new one lossy=1)
>> fault with nothing to send, going to standby
>> leveldb manual compact
>>
>> I can share all the logs that can help to identify the issue.
>>
>> Thank you.
>> Regards,
>>
>> Matteo
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>> --
>> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non infetto.
>> Seguire il link qui sotto per segnalarlo come spam:
>> http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=12EAC4481A.A6F60
>>
>>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
--------------------------------------------
Peter Maloney
Brockmann Consult
Max-Planck-Str. 2
21502 Geesthacht
Germany
Tel: +49 4152 889 300
Fax: +49 4152 889 333
E-mail: peter.maloney@xxxxxxxxxxxxxxxxxxxx
Internet: http://www.brockmann-consult.de
--------------------------------------------
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
