That's the issue I remember (#20763)!
The hang happened to me once, on this cluster, after upgrade from jewel
to 12.2.2; then on Friday I disabled automatic bucket resharding due to
some other problems - didn't get any logrotate-related hangs through the
weekend. I wonder if these could be related?
Graham
On 12/11/2017 02:01 AM, Martin Emrich wrote:
Hi!
This sounds like http://tracker.ceph.com/issues/20763 (or indeed http://tracker.ceph.com/issues/20866).
It is still present in 12.2.2 (just tried it). My workaround is to exclude radosgw from logrotate (remove "radosgw" from /etc/logrotate.d/ceph) from being SIGHUPed, and to rotate the logs manually from time to time and completely restarting the radosgw processes one after the other on my radosgw cluster.
Regards,
Martin
Am 08.12.17, 18:58 schrieb "ceph-users im Auftrag von Graham Allan" <ceph-users-bounces@xxxxxxxxxxxxxx im Auftrag von gta@xxxxxxx>:
I noticed this morning that all four of our rados gateways (luminous
12.2.2) hung at logrotate time overnight. The last message logged was:
> 2017-12-08 03:21:01.897363 7fac46176700 0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
one of the 3 nodes recorded more detail:
> 2017-12-08 06:51:04.452108 7f80fbfdf700 1 rgw realm reloader: Pausing frontends for realm update...
> 2017-12-08 06:51:04.452126 7f80fbfdf700 1 rgw realm reloader: Frontends paused
> 2017-12-08 06:51:04.452891 7f8202436700 0 ERROR: failed to clone shard, completion_mgr.get_next() returned ret=-125
I remember seeing this happen on our test cluster a while back with
Kraken. I can't find the tracker issue I originally found related to
this, but it also sounds like it could be a reversion of bug #20339 or
#20686?
I recorded some strace output from one of the radosgw instances before
restarting, if it's useful to open an issue.
--
Graham Allan
Minnesota Supercomputing Institute - gta@xxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Graham Allan
Minnesota Supercomputing Institute - gta@xxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
