|
I noticed that my RGW Replication wasn't doing anything. RGW
Replication logs were full of: 2014-03-26T11:29:01.802 18698:ERROR:root:Could not retrieve region map from destination Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/radosgw_agent/cli.py", line 262, in main region_map = client.get_region_map(dest_conn) <snip> I tried to get a list of buckets in the slave zone, and it hangs. The simpliest operation I can run that hangs is: radosgw-admin --name=client.radosgw.ceph1c regions list This cluster has 2 nodes. Watching ceph -w, I see 2 OSDs get kicked out of the cluster. OSDs 4 and 8 are both flapping. An excerpt from ceph -w: 2014-03-26 11:24:33.582757 osd.9 [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.838374 secs 2014-03-26 11:24:33.582762 osd.9 [WRN] slow request 30.838374 seconds old, received at 2014-03-26 11:24:02.744331: osd_op(client.41087064.0:8 notify.0 [watch add cookie 1 ver 0] 9.4322fa9f e8270) v4 currently waiting for subops from [4] 2014-03-26 11:24:35.583064 osd.9 [WRN] 2 slow requests, 1 included below; oldest blocked for > 32.838703 secs 2014-03-26 11:24:35.583068 osd.9 [WRN] slow request 30.012828 seconds old, received at 2014-03-26 11:24:05.570206: osd_op(client.41087080.0:8 notify.0 [watch add cookie 1 ver 0] 9.4322fa9f e8270) v4 currently waiting for subops from [4] 2014-03-26 11:28:10.612684 osd.9 [WRN] 4 slow requests, 1 included below; oldest blocked for > 247.868328 secs 2014-03-26 11:28:10.612692 osd.9 [WRN] slow request 240.276922 seconds old, received at 2014-03-26 11:24:10.335737: osd_op(client.41087104.0:8 notify.0 [watch add cookie 1 ver 0] 9.4322fa9f e8270) v4 currently waiting for subops from [4] 2014-03-26 11:31:08.198483 mon.0 [INF] osd.4 marked itself down 2014-03-26 11:31:08.266406 mon.0 [INF] osdmap e8271: 16 osds: 15 up, 16 in 2014-03-26 11:31:21.294695 mon.0 [INF] osd.4 10.193.0.6:6812/2215 boot 2014-03-26 11:33:55.856801 mon.0 [INF] osd.8 marked down after no pg stats for 900.157048seconds 2014-03-26 11:33:55.894864 mon.0 [INF] osdmap e8275: 16 osds: 15 up, 16 in 2014-03-26 11:47:30.952634 mon.0 [INF] osd.8 out (down for 815.058447) I set the noout flag. This is a production slave zone. I can play with the cluster as long as replication is off. By playing with OSD 4 and 8, I found that I can make the cluster work fine with OSD 4 up, and OSD 8 down. If I start ODS 8, and stop OSD 4, RGW is still hung. If I start both, RGW is still hung. I saw some mailing list posts where somebody had a similar issue, and fixing the time solved it. I verified that my times are correct on both machines. I haven't seen any XFS issues, but just to be sure, I umounted both devices, and ran xfs_check. No errors were reported. Since all the easy things appear to be in order, I bumped up logging on both nodes: [osd] osd journal size = 6144 osd max backfills = 1 osd mkfs options xfs = -l size=1024m -n size=64k -i size=2048 -s size=4096 osd mkfs type = xfs osd mount options xfs = rw,noatime,nodiratime,nosuid,noexec,inode64 osd recovery op priority = 1 debug osd = 5 debug filestore = 5 debug journal = 1 debug monc = 20 At 14:14:00, I started OSD 4, and waited for ceph-w to stabilize. CPU usage was normal. At 14:15:10, I ran radosgw-admin --name=client.radosgw.ceph1c regions list && radosgw-admin --name=client.radosgw.ceph1c regionmap get. It returned successfully. At 14:16:00, I started OSD 8, and waited for ceph -w to stabilize. CPU usage started out normal, but went to 100% before 14:16:40. At 14:17:25, I ran radosgw-admin --name=client.radosgw.ceph1c regions list && radosgw-admin --name=client.radosgw.ceph1c regionmap get. regions list hung, and I killed At 14:18:15, I stopped ceph-osd id=8. At 14:18:45, I ran radosgw-admin --name=client.radosgw.ceph1c regions list && radosgw-admin --name=client.radosgw.ceph1c regionmap get. It returned successfully. At 14:19:10, I stopped ceph-osd id=8. Looking at the logs, nothing jumps out at me between 14:16:00 and 14:16:40. The logs are busy, since OSD 8 is backfilling. I don't think it's related, but I'd like to point out that all of the RGW pools have snapshots: root@ceph1c:/var/log/ceph# rados lssnap -p .us-central-1.rgw.root 3 2014-03-15T02:00:02 2014.03.15 02:00:02 4 2014-03-16T02:00:01 2014.03.16 02:00:02 5 2014-03-17T02:00:01 2014.03.17 02:00:01 6 2014-03-18T02:00:01 2014.03.18 02:00:01 7 2014-03-19T02:00:01 2014.03.19 02:00:01 8 2014-03-20T02:00:01 2014.03.20 02:00:01 9 2014-03-21T02:00:01 2014.03.21 02:00:01 10 2014-03-22T02:00:01 2014.03.22 02:00:01 11 2014-03-23T02:00:02 2014.03.23 02:00:02 12 2014-03-24T02:00:02 2014.03.24 02:00:02 13 2014-03-25T02:00:02 2014.03.25 02:00:02 14 2014-03-26T02:00:01 2014.03.26 02:00:01 12 snaps That's only in my slave zone. The master zone (which is working fine) has no snapshots. For now, I've removed the debugging, started OSD 4, and started radosgw-agent. OSD 8 remains off. There are 43 PGs that live on OSD 4 and 8: root@ceph1c:/var/log/ceph# ceph pg dump | fgrep '[4]' | cut -d"." -f1 | sort | uniq -c 36 11 2 13 1 14 1 18 1 19 1 20 1 21 root@ceph1c:/var/log/ceph# ceph osd dump | grep pool | egrep 'pool (11|13|14|18|19|20|21)' pool 11 '.rgw.buckets' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 8014 owner 0 flags hashpspool pool 13 '.us-central-1.log' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 8007 owner 0 flags hashpspool pool 14 '.us-central-1.intent-log' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 8006 owner 0 flags hashpspool pool 18 '.us-central-1.users.swift' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 8010 owner 0 flags hashpspool pool 19 '.us-central-1.users.uid' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 8011 owner 0 flags hashpspool pool 20 '.rgw.root' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 207 owner 0 flags hashpspool pool 21 '.rgw.control' rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 208 owner 0 flags hashpspool The two OSD logs from 14:14:00 to 14:19:10 can be found at OSD 4: https://cd.centraldesktop.com/p/eAAAAAAADL0vAAAAAABMSio OSD 8: https://cd.centraldesktop.com/p/eAAAAAAADL05AAAAAFAVWRg Any suggestions or ideas are appreciated. |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
