I have a 3 node Ceph 0.87 cluster. After a while I see an error in radosgw and I don’t find references in the list archives
heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc4eac2d700' had timed out after 600
The only solution is restart radosgw and for a while it works just fine
Any idea?
Thanks
ceph.conf
[global]
fsid = fc0e2e09-ade3-4ff6-b23e-f789775b2515
mon initial members = nodo-3
mon host = 192.168.2.200, 192.168.2.201, 192.168.2.202
mon addr = 192.168.2.200:6789, 192.168.2.201:6789, 192.168.2.202:6789
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd pool default size = 3
osd pool default min_size = 1
osd pool default pg_num = 128
osd pool default pgp_num = 128
osd recovery delay start = 15
log file = /dev/stdout
mon clock drift allowed = 1
[client.radosgw.gateway]
host = deis-store-gateway
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/radosgw.log
Full trace:
2014-12-15 21:59:27.976981 7fc70cb1c840 0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process radosgw, pid 127
2014-12-15 21:59:28.005388 7fc70cb1c840 0 framework: fastcgi
2014-12-15 21:59:28.005393 7fc70cb1c840 0 framework: civetweb
2014-12-15 21:59:28.005398 7fc70cb1c840 0 framework conf key: port, val: 7480
2014-12-15 21:59:28.005402 7fc70cb1c840 0 starting handler: civetweb
2014-12-15 21:59:28.010659 7fc70cb1c840 0 starting handler: fastcgi
2014-12-15 21:59:39.961503 7fc55cd11700 1 ====== starting new request req=0x7fc6ec1148e0 =====
2014-12-15 21:59:39.965239 7fc55cd11700 1 ====== req done req=0x7fc6ec1148e0 http_status=200 ======
2014-12-15 21:59:40.033219 7fc554500700 1 ====== starting new request req=0x7fc6ec11c190 =====
2014-12-15 21:59:40.038634 7fc554500700 0 WARNING: couldn't find acl header for object, generating default
2014-12-15 21:59:40.348267 7fc554500700 1 ====== req done req=0x7fc6ec11c190 http_status=200 ======
2014-12-15 22:00:42.522831 7fc554500700 1 ====== starting new request req=0x7fc6ec11c220 =====
2014-12-15 22:00:42.786590 7fc554500700 1 ====== req done req=0x7fc6ec11c220 http_status=200 ======
2014-12-15 22:04:41.906676 7fc55cd11700 1 ====== starting new request req=0x7fc6ec11c4c0 =====
2014-12-15 22:04:42.077969 7fc55cd11700 1 ====== req done req=0x7fc6ec11c4c0 http_status=200 ======
2014-12-15 22:09:42.270387 7fc554500700 1 ====== starting new request req=0x7fc6ec11bb90 =====
2014-12-15 22:09:42.634896 7fc554500700 1 ====== req done req=0x7fc6ec11bb90 http_status=200 ======
2014-12-15 22:14:42.812094 7fc554500700 1 ====== starting new request req=0x7fc6ec11a2c0 =====
2014-12-15 22:14:43.027164 7fc554500700 1 ====== req done req=0x7fc6ec11a2c0 http_status=200 ======
2014-12-15 22:19:43.330578 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec11a560 =====
2014-12-15 22:19:43.505847 7fc5acdb1700 1 ====== req done req=0x7fc6ec11a560 http_status=200 ======
2014-12-15 22:24:31.664914 7fc6fb7fe700 0 monclient: hunting for new mon
2014-12-15 22:24:31.691258 7fc70cb14700 0 -- 192.168.2.201:0/1000131 >> 192.168.2.202:6800/1 pipe(0x7fc6f0120610 sd=9 :0 s=1 pgs=0 cs=0 l=1 c=0x7fc6f01208a0).fault
2014-12-15 22:24:43.653020 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec11a3b0 =====
2014-12-15 22:24:49.093981 7fc55cd11700 1 ====== starting new request req=0x7fc6ec119d60 =====
2014-12-15 22:24:55.165618 7fc51347e700 1 ====== starting new request req=0x7fc6ec121290 =====
2014-12-15 22:25:04.181370 7fc57cd51700 1 ====== starting new request req=0x7fc6ec125fa0 =====
2014-12-15 22:25:11.936946 7fc531cbb700 1 ====== starting new request req=0x7fc6ec12ad20 =====
2014-12-15 22:25:12.401848 7fc5acdb1700 1 ====== req done req=0x7fc6ec11a3b0 http_status=200 ======
2014-12-15 22:25:12.402031 7fc57cd51700 1 ====== req done req=0x7fc6ec125fa0 http_status=200 =====
2014-12-15 22:25:12.402164 7fc51347e700 1 ====== req done req=0x7fc6ec121290 http_status=200 ======
2014-12-15 22:25:12.402286 7fc531cbb700 1 ====== req done req=0x7fc6ec12ad20 http_status=200 ======
2014-12-15 22:25:12.574183 7fc55cd11700 1 ====== req done req=0x7fc6ec119d60 http_status=200 ======
2014-12-15 22:28:44.138277 7fc531cbb700 1 ====== starting new request req=0x7fc6ec12fa80 =====
2014-12-15 22:28:44.277586 7fc531cbb700 1 ====== req done req=0x7fc6ec12fa80 http_status=200 ======
2014-12-15 22:29:44.023631 7fc531cbb700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 22:29:44.233772 7fc531cbb700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 22:34:43.458371 7fc51347e700 1 ====== starting new request req=0x7fc6ec119dc0 =====
2014-12-15 22:34:43.618785 7fc51347e700 1 ====== req done req=0x7fc6ec119dc0 http_status=200 ======
2014-12-15 22:39:43.772838 7fc531cbb700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 22:39:43.954160 7fc531cbb700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 22:44:44.131498 7fc55cd11700 1 ====== starting new request req=0x7fc6ec119dc0 =====
2014-12-15 22:44:44.272247 7fc55cd11700 1 ====== req done req=0x7fc6ec119dc0 http_status=200 ======
2014-12-15 22:49:44.585957 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 22:49:44.958454 7fc5acdb1700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 22:54:44.960858 7fc51347e700 1 ====== starting new request req=0x7fc6ec119dc0 =====
2014-12-15 22:54:45.161036 7fc51347e700 1 ====== req done req=0x7fc6ec119dc0 http_status=200 ======
2014-12-15 22:59:45.371507 7fc531cbb700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 22:59:45.530839 7fc531cbb700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 23:04:45.750120 7fc57cd51700 1 ====== starting new request req=0x7fc6ec119dc0 =====
2014-12-15 23:04:45.965856 7fc57cd51700 1 ====== req done req=0x7fc6ec119dc0 http_status=200 ======
2014-12-15 23:09:46.112771 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 23:09:46.323984 7fc5acdb1700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 23:14:46.519341 7fc51347e700 1 ====== starting new request req=0x7fc6ec119dc0 =====
2014-12-15 23:14:46.737396 7fc51347e700 1 ====== req done req=0x7fc6ec119dc0 http_status=200 ======
2014-12-15 23:19:46.906210 7fc554500700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 23:19:47.206524 7fc554500700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 23:24:46.328850 7fc51347e700 1 ====== starting new request req=0x7fc6ec119dc0 =====
2014-12-15 23:24:46.573374 7fc51347e700 1 ====== req done req=0x7fc6ec119dc0 http_status=200 ======
2014-12-15 23:29:46.651333 7fc5cedf5700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 23:29:46.972141 7fc5cedf5700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 23:34:47.110883 7fc5a9dab700 1 ====== starting new request req=0x7fc6ec11a110 =====
2014-12-15 23:34:47.528464 7fc5a9dab700 1 ====== req done req=0x7fc6ec11a110 http_status=200 ======
2014-12-15 23:39:47.484598 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec11c560 =====
2014-12-15 23:39:47.801606 7fc5acdb1700 1 ====== req done req=0x7fc6ec11c560 http_status=200 ======
2014-12-15 23:44:47.919073 7fc531cbb700 1 ====== starting new request req=0x7fc6ec1207e0 =====
2014-12-15 23:44:48.100640 7fc531cbb700 1 ====== req done req=0x7fc6ec1207e0 http_status=200 ======
2014-12-15 23:49:47.345552 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec11de50 =====
2014-12-15 23:49:47.541658 7fc5acdb1700 1 ====== req done req=0x7fc6ec11de50 http_status=200 ======
2014-12-15 23:54:47.768162 7fc531cbb700 1 ====== starting new request req=0x7fc6ec11f840 =====
2014-12-15 23:54:47.921098 7fc531cbb700 1 ====== req done req=0x7fc6ec11f840 http_status=200 ======
2014-12-15 23:59:48.206632 7fc5a9dab700 1 ====== starting new request req=0x7fc6ec11de50 =====
2014-12-15 23:59:48.401686 7fc5a9dab700 1 ====== req done req=0x7fc6ec11de50 http_status=200 ======
2014-12-16 00:04:48.601401 7fc5244a0700 1 ====== starting new request req=0x7fc6ec11f840 =====
2014-12-16 00:04:48.737402 7fc5244a0700 1 ====== req done req=0x7fc6ec11f840 http_status=200 ======
2014-12-16 00:09:48.912230 7fc51347e700 1 ====== starting new request req=0x7fc6ec125ea0 =====
2014-12-16 00:09:49.128728 7fc51347e700 1 ====== req done req=0x7fc6ec125ea0 http_status=200 ======
2014-12-16 00:14:48.314372 7fc5244a0700 1 ====== starting new request req=0x7fc6ec1231a0 =====
2014-12-16 00:14:48.447242 7fc5244a0700 1 ====== req done req=0x7fc6ec1231a0 http_status=200 ======
2014-12-16 00:19:48.696777 7fc51347e700 1 ====== starting new request req=0x7fc6ec0008c0 =====
2014-12-16 00:19:49.192244 7fc51347e700 1 ====== req done req=0x7fc6ec0008c0 http_status=200 ======
2014-12-16 00:24:49.103908 7fc576544700 1 ====== starting new request req=0x7fc6ec121970 ====
2014-12-16 00:24:49.351608 7fc576544700 1 ====== req done req=0x7fc6ec121970 http_status=200 ======
2014-12-16 00:29:49.478087 7fc590578700 1 ====== starting new request req=0x7fc6ec119a30 =====
2014-12-16 00:29:49.797458 7fc590578700 1 ====== req done req=0x7fc6ec119a30 http_status=200 ======
2014-12-16 00:34:49.896796 7fc51347e700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 00:34:50.088958 7fc51347e700 1 ====== req done req=0x7fc6ec12b1c0 http_status=200 ======
2014-12-16 00:39:49.282836 7fc590578700 1 ====== starting new request req=0x7fc6ec120af0 =====
2014-12-16 00:39:49.474155 7fc590578700 1 ====== req done req=0x7fc6ec120af0 http_status=200 ======
2014-12-16 00:44:49.669267 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec1148e0 =====
2014-12-16 00:44:49.931567 7fc5acdb1700 1 ====== req done req=0x7fc6ec1148e0 http_status=200 ======
2014-12-16 00:49:50.071880 7fc531cbb700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 00:49:50.278672 7fc531cbb700 1 ====== req done req=0x7fc6ec12b1c0 http_status=200 ======
2014-12-16 00:54:50.434177 7fc50d472700 1 ====== starting new request req=0x7fc6ec120af0 =====
2014-12-16 00:54:50.636824 7fc50d472700 1 ====== req done req=0x7fc6ec120af0 http_status=200 ======
2014-12-16 00:59:50.860987 7fc5244a0700 1 ====== starting new request req=0x7fc6ec1148e0 =====
2014-12-16 00:59:51.104947 7fc5244a0700 1 ====== req done req=0x7fc6ec1148e0 http_status=200 ======
2014-12-16 01:04:51.227055 7fc51347e700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 01:04:51.684523 7fc51347e700 1 ====== req done req=0x7fc6ec12b1c0 http_status=200 ======
2014-12-16 01:09:51.625787 7fc5b8dc9700 1 ====== starting new request req=0x7fc6ec120af0 =====
2014-12-16 01:09:51.832569 7fc5b8dc9700 1 ====== req done req=0x7fc6ec120af0 http_status=200 ======
2014-12-16 01:14:52.039519 7fc590578700 1 ====== starting new request req=0x7fc6ec1148e0 =====
2014-12-16 01:14:52.327627 7fc590578700 1 ====== req done req=0x7fc6ec1148e0 http_status=200 ======
2014-12-16 01:16:48.177117 7fc5244a0700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 01:16:48.328715 7fc5244a0700 1 ====== req done req=0x7fc6ec12b1c0 http_status=200 ======
2014-12-16 01:16:52.294757 7fc5244a0700 1 ====== starting new request req=0x7fc6ec120af0 =====
2014-12-16 01:16:52.574996 7fc5244a0700 1 ====== req done req=0x7fc6ec120af0 http_status=200 ======
2014-12-16 01:16:57.864873 7fc51347e700 1 ====== starting new request req=0x7fc6ec1148e0 ====
2014-12-16 01:16:58.488740 7fc51347e700 1 ====== req done req=0x7fc6ec1148e0 http_status=200 ======
2014-12-16 01:16:58.806320 7fc59f596700 1 ====== starting new request req=0x7fc6ec120af0 =====
2014-12-16 01:16:58.953147 7fc59f596700 1 ====== req done req=0x7fc6ec120af0 http_status=200 ======
2014-12-16 01:16:59.190285 7fc5b8dc9700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 01:16:59.351468 7fc5b8dc9700 1 ====== req done req=0x7fc6ec12b1c0 http_status=200 ======
2014-12-16 01:16:59.600994 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec1148e0 =====
2014-12-16 01:16:59.738354 7fc5acdb1700 1 ====== req done req=0x7fc6ec1148e0 http_status=200 ======
2014-12-16 01:16:59.848306 7fc51347e700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 01:16:59.855980 7fc51347e700 1 ====== req done req=0x7fc6ec12b1c0 http_status=200 ======
2014-12-16 01:16:59.893245 7fc5244a0700 1 ====== starting new request req=0x7fc6ec120af0 =====
2014-12-16 01:16:59.900728 7fc5244a0700 1 ====== req done req=0x7fc6ec120af0 http_status=200 ======
2014-12-16 01:16:59.904951 7fc50d472700 1 ====== starting new request req=0x7fc6ec1148e0 =====
2014-12-16 01:17:00.010587 7fc50d472700 1 ====== req done req=0x7fc6ec1148e0 http_status=200 ======
2014-12-16 01:17:00.214589 7fc590578700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 01:17:00.262242 7fc590578700 1 ====== req done req=0x7fc6ec12b1c0 http_status=200 ======
2014-12-16 01:17:00.326445 7fc59f596700 1 ====== starting new request req=0x7fc6ec120af0 =====
2014-12-16 01:17:00.336068 7fc59f596700 1 ====== req done req=0x7fc6ec120af0 http_status=200 ======
2014-12-16 01:17:00.961307 7fc4eac2d700 1 ====== starting new request req=0x7fc6ec1148e0 =====
2014-12-16 01:17:01.451900 7fc4eac2d700 0 RGWObjManifest::operator++(): result: ofs=4718592 stripe_ofs=4718592 part_ofs=0 rule->part_size=0
2014-12-16 01:17:01.451903 7fc4eac2d700 0 RGWObjManifest::operator++(): result: ofs=4933786 stripe_ofs=4933786 part_ofs=0 rule->part_size=0
2014-12-16 01:17:05.995727 7fc50d472700 1 ====== starting new request req=0x7fc6ec12b1c0 =====
2014-12-16 01:17:12.195228 7fc590578700 1 ====== starting new request req=0x7fc6ec1bacc0 =====
2014-12-16 01:17:20.134475 7fc59f596700 1 ====== starting new request req=0x7fc6ec1d0120 =====
2014-12-16 01:17:25.874526 7fc51347e700 1 ====== starting new request req=0x7fc6ec202b10 =====
2014-12-16 01:17:35.797966 7fc5acdb1700 1 ====== starting new request req=0x7fc6ec1ffe60 =====
2014-12-16 01:17:53.420953 7fc5b8dc9700 1 ====== starting new request req=0x7fc6ec1aab90 =====
2014-12-16 01:18:43.448919 7fc531cbb700 1 ====== starting new request req=0x7fc6ec250400 =====
2014-12-16 01:18:48.570450 7fc5c4de1700 1 ====== starting new request req=0x7fc6ec26d6b0 =====
2014-12-16 01:18:54.710198 7fc510c79700 1 ====== starting new request req=0x7fc6ec27b990 =====
2014-12-16 01:19:02.010002 7fc5244a0700 1 ====== starting new request req=0x7fc6ec31cd20 =====
2014-12-16 01:19:12.257436 7fc57dd53700 1 ====== starting new request req=0x7fc6ec324d40 =====
2014-12-16 01:19:24.180882 7fc58e574700 1 ====== starting new request req=0x7fc6ec34af40 ====
2014-12-16 01:19:42.452047 7fc5d05f8700 1 ====== starting new request req=0x7fc6ec3644a0 =====
2014-12-16 01:19:49.958032 7fc56dd33700 1 ====== starting new request req=0x7fc6ec384c90 =====
2014-12-16 01:19:55.879375 7fc4e5c23700 1 ====== starting new request req=0x7fc6ec3c69d0 =====
2014-12-16 01:20:02.435453 7fc54dcf3700 1 ====== starting new request req=0x7fc6ec412b20 =====
2014-12-16 01:20:10.858710 7fc5b85c8700 1 ====== starting new request req=0x7fc6ec420bf0 =====
2014-12-16 01:20:21.714292 7fc5d4600700 1 ====== starting new request req=0x7fc6ec43eb40 =====
2014-12-16 01:20:42.605250 7fc511c7b700 1 ====== starting new request req=0x7fc6ec18c3d0 =====
2014-12-16 01:21:15.000252 7fc51fc97700 1 ====== starting new request req=0x7fc6ec222c60 =====
2014-12-16 01:21:34.445231 7fc52acad700 1 ====== starting new request req=0x7fc6ec1cb440 ====
2014-12-16 01:21:39.566784 7fc56bd2f700 1 ====== starting new request req=0x7fc6ec4bdc10 =====
2014-12-16 01:21:45.294918 7fc50946a700 1 ====== starting new request req=0x7fc6ec4d0d90 ====
2014-12-16 01:21:50.938308 7fc51dc93700 1 ====== starting new request req=0x7fc6ec4f0580 =====
2014-12-16 01:21:54.967639 7fc4eac2d700 0 RGWObjManifest::operator++(): result: ofs=2548475 stripe_ofs=2548475 part_ofs=0 rule->part_size=0
2014-12-16 01:21:58.637426 7fc51ec95700 1 ====== starting new request req=0x7fc6ec4d23a0 =====
2014-12-16 01:22:00.661785 7fc57f556700 1 ====== starting new request req=0x7fc6ec2de870 ====
2014-12-16 01:22:04.345771 7fc518c89700 1 ====== starting new request req=0x7fc6ec504220 =====
2014-12-16 01:22:05.102161 7fc51ec95700 1 ====== req done req=0x7fc6ec4d23a0 http_status=200 ======
2014-12-16 01:22:09.415817 7fc518c89700 1 ====== req done req=0x7fc6ec504220 http_status=200 ======
2014-12-16 01:22:10.929547 7fc51ec95700 1 ====== starting new request req=0x7fc6ec312b00 =====
2014-12-16 01:22:15.859385 7fc51ec95700 1 ====== req done req=0x7fc6ec312b00 http_status=200 ======
2014-12-16 01:22:19.780891 7fc51ec95700 1 ====== starting new request req=0x7fc6ec121610 =====
2014-12-16 01:22:44.216635 7fc518c89700 1 ====== starting new request req=0x7fc6ec2b6510 =====
2014-12-16 01:23:11.876875 7fc594d81700 1 ====== starting new request req=0x7fc6ec269590 =====
2014-12-16 01:23:17.682509 7fc58ad6d700 1 ====== starting new request req=0x7fc6ec3bd560 =====
2014-12-16 01:23:23.794044 7fc52b4ae700 1 ====== starting new request req=0x7fc6ec34a490 =====
2014-12-16 01:23:29.713837 7fc53f4d6700 1 ====== starting new request req=0x7fc6ec53cb10 =====
2014-12-16 01:23:40.117459 7fc58255c700 1 ====== starting new request req=0x7fc6ec4735b0 =====
2014-12-16 01:23:52.376745 7fc4e942a700 1 ====== starting new request req=0x7fc6ec1e4480 =====
2014-12-16 01:24:09.483995 7fc51d492700 1 ====== starting new request req=0x7fc6ec204890 =====
2014-12-16 01:24:48.732829 7fc554d01700 1 ====== starting new request req=0x7fc6ec48bf10 =====
2014-12-16 01:24:53.942076 7fc53dcd3700 1 ====== starting new request req=0x7fc6ec298b10 =====
2014-12-16 01:25:00.289719 7fc56c530700 1 ====== starting new request req=0x7fc6ec2a0440 =====
2014-12-16 01:25:09.125304 7fc50a46c700 1 ====== starting new request req=0x7fc6ec266af0 =====
2014-12-16 01:25:22.028771 7fc57e554700 1 ====== starting new request req=0x7fc6ec4f4f40 =====
2014-12-16 01:25:34.284041 7fc5284a8700 1 ====== starting new request req=0x7fc6ec4e9780 =====
2014-12-16 01:25:56.919045 7fc53d4d2700 1 ====== starting new request req=0x7fc6ec33a360 =====
2014-12-16 01:26:23.332489 7fc506464700 1 ====== starting new request req=0x7fc6ec4e2480 =====
2014-12-16 01:26:29.061974 7fc5b9dcb700 1 ====== starting new request req=0x7fc6ec1227e0 =====
2014-12-16 01:26:34.569411 7fc52ccb1700 1 ====== starting new request req=0x7fc6ec3b2120 =====
2014-12-16 01:26:43.281399 7fc55950a700 1 ====== starting new request req=0x7fc6ec5faf70 =====
2014-12-16 01:26:55.300503 7fc51ac8d700 1 ====== starting new request req=0x7fc6ec60e260 =====
2014-12-16 01:26:59.103552 7fc55fd17700 1 ====== starting new request req=0x7fc6ec3777d0 =====
2014-12-16 01:27:01.212222 7fc5bfdd7700 1 ====== starting new request req=0x7fc6ec4c6cc0 ====
2014-12-16 01:27:03.167234 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc4eac2d700' had timed out after 600
2014-12-16 01:27:04.940160 7fc5d5e03700 1 ====== starting new request req=0x7fc6ec24c0f0 =====
2014-12-16 01:27:08.170973 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc50d472700' had timed out after 600
2014-12-16 01:27:08.171004 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc4eac2d700' had timed out after 600
2014-12-16 01:27:08.872852 7fc55fd17700 1 ====== req done req=0x7fc6ec3777d0 http_status=200 ======
2014-12-16 01:27:08.874417 7fc5d5e03700 1 ====== req done req=0x7fc6ec24c0f0 http_status=200 ======
2014-12-16 01:27:13.178668 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc50d472700' had timed out after 600
2014-12-16 01:27:13.178688 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc4eac2d700' had timed out after 600
2014-12-16 01:27:13.178742 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc590578700' had timed out after 600
2014-12-16 01:27:18.182477 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc50d472700' had timed out after 600
2014-12-16 01:27:18.182516 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc4eac2d700' had timed out after 600
2014-12-16 01:27:18.182630 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc590578700' had timed out after 600
2014-12-16 01:27:23.190302 7fc7019db700 1 heartbeat_map is_healthy 'RGWProcess::m_tp thread 0x7fc50d472700' had timed out after 600
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
