BTW, I have 10 RGW load balanced through Apache. When restarting one of them I get the following messages in log: 2016-06-14 14:44:15.919801 7fd4728dea40 2 all 8 watchers are set, enabling cache 2016-06-14 14:44:15.919879 7fce370f7700 2 garbage collection: start 2016-06-14 14:44:15.919990 7fce368f6700 2 object expiration: start 2016-06-14 14:44:15.920534 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.15 2016-06-14 14:44:15.921257 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.16 2016-06-14 14:44:15.922145 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.17 2016-06-14 14:44:15.923772 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.18 2016-06-14 14:44:15.924557 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.19 2016-06-14 14:44:15.925400 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.20 2016-06-14 14:44:15.926349 7fd4728dea40 0 starting handler: fastcgi 2016-06-14 14:44:15.927125 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.21 2016-06-14 14:44:15.927897 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.22 2016-06-14 14:44:15.928412 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.23 2016-06-14 14:44:15.929042 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.24 2016-06-14 14:44:15.930752 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.25 2016-06-14 14:44:15.931313 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.26 2016-06-14 14:44:15.932482 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.27 2016-06-14 14:44:15.933237 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.28 2016-06-14 14:44:15.934097 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.29 2016-06-14 14:44:15.934660 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.30 2016-06-14 14:44:15.936322 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.31 2016-06-14 14:44:15.936979 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.0 2016-06-14 14:44:15.937559 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.1 2016-06-14 14:44:15.938222 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.2 2016-06-14 14:44:15.939000 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.3 2016-06-14 14:44:15.939622 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.4 2016-06-14 14:44:15.940135 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.5 2016-06-14 14:44:15.940669 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.6 2016-06-14 14:44:15.941227 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.7 2016-06-14 14:44:15.941854 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.8 2016-06-14 14:44:15.942333 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.9 2016-06-14 14:44:15.943036 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.10 2016-06-14 14:44:15.944708 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.11 2016-06-14 14:44:15.946347 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.12 2016-06-14 14:44:15.947001 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.13 2016-06-14 14:44:15.947610 7fce370f7700 0 RGWGC::process() failed to acquire lock on gc.14 2016-06-14 14:44:15.947615 7fce370f7700 2 garbage collection: stop 2016-06-14 14:44:15.947949 7fd4728dea40 -1 rgw realm watcher: Failed to watch realms.87abf44e-cab3-48c4-b012-0a9247519a5b.control with (2) No such file or directory 2016-06-14 14:44:15.948370 7fd4728dea40 -1 rgw realm watcher: Failed to establish a watch on RGWRealm, disabling dynamic reconfiguration. 2016-06-14 17:34 GMT+03:00 Василий Ангапов <angapov@xxxxxxxxx>: > I also get the following: > > $ radosgw-admin period update --commit > 2016-06-14 14:32:28.982847 7fed392baa40 0 ERROR: failed to distribute > cache for .rgw.root:periods.87abf44e-cab3-48c4-b012-0a9247519a5b:staging > 2016-06-14 14:32:38.991846 7fed392baa40 0 ERROR: failed to distribute > cache for .rgw.root:periods.87abf44e-cab3-48c4-b012-0a9247519a5b:staging.latest_epoch > 2016-06-14 14:32:49.002380 7fed392baa40 0 ERROR: failed to distribute > cache for .rgw.root:periods.af0b6743-82ba-4517-bd51-36bdfbe48f9f.3 > 2016-06-14 14:32:59.013307 7fed392baa40 0 ERROR: failed to distribute > cache for .rgw.root:periods.af0b6743-82ba-4517-bd51-36bdfbe48f9f.latest_epoch > 2016-06-14 14:33:09.023554 7fed392baa40 0 ERROR: failed to distribute > cache for .rgw.root:periods.af0b6743-82ba-4517-bd51-36bdfbe48f9f.latest_epoch > 2016-06-14 14:33:19.034593 7fed392baa40 0 ERROR: failed to distribute > cache for .rgw.root:zonegroup_info.bef0aa4e-6670-4c39-8520-ee51140424cc > 2016-06-14 14:33:29.043825 7fed392baa40 0 ERROR: failed to distribute > cache for .rgw.root:zonegroups_names.ed > 2016-06-14 14:33:29.046386 7fed392baa40 0 Realm notify failed with -2 > { > "id": "af0b6743-82ba-4517-bd51-36bdfbe48f9f", > "epoch": 3, > "predecessor_uuid": "f2645d83-b1b4-4045-bf26-2b762c71937b", > "sync_status": [ > "", > "", > > 2016-06-14 17:12 GMT+03:00 Василий Ангапов <angapov@xxxxxxxxx>: >> Hello, >> >> I have Ceph 10.2.1 and when creating user in RGW I get the following error: >> >> $ radosgw-admin user create --uid=test --display-name="test" >> 2016-06-14 14:07:32.332288 7f00a4487a40 0 ERROR: failed to distribute >> cache for ed-1.rgw.meta:.meta:user:test:_dW3fzQ3UX222SWQvr3qeHYR:1 >> 2016-06-14 14:07:42.338251 7f00a4487a40 0 ERROR: failed to distribute >> cache for ed-1.rgw.users.uid:test >> 2016-06-14 14:07:52.362768 7f00a4487a40 0 ERROR: failed to distribute >> cache for ed-1.rgw.users.keys:3J7DOREPC0ZLVFTMIW75 >> { >> "user_id": "test", >> "display_name": "test", >> "email": "", >> "suspended": 0, >> "max_buckets": 1000, >> "auid": 0, >> "subusers": [], >> "keys": [ >> { >> "user": "melesta", >> "access_key": "***", >> "secret_key": "***" >> } >> ], >> "swift_keys": [], >> "caps": [], >> "op_mask": "read, write, delete", >> "default_placement": "", >> "placement_tags": [], >> "bucket_quota": { >> "enabled": false, >> "max_size_kb": -1, >> "max_objects": -1 >> }, >> "user_quota": { >> "enabled": false, >> "max_size_kb": -1, >> "max_objects": -1 >> }, >> "temp_url_keys": [] >> } >> >> What does it mean? Is something wrong? >> >> Thanks! _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com