(I tried to start this discussion on irc, but I wound up with the wrong paste buffer and wound up getting kicked off for a paste flood, sorry, that was on me :( ) We were having some weirdness with our Ceph and did an upgrade up to 10.2.3, which fixed some, but not all of our problems. It looked like our users pool might have been corrupt, so we moved it aside and created a new set: $ sudo ceph osd pool rename .users old.users $ sudo ceph osd pool rename .users.email old.users.email $ sudo ceph osd pool rename .users.swift old.users.swift $ sudo ceph osd pool rename .users.uid old.users.uid $ sudo ceph osd pool create .users 16 16 $ sudo ceph osd pool create .users.email 16 16 $ sudo ceph osd pool create .users.swift 16 16 $ sudo ceph osd pool create .users.uid 16 16 This allowed me to create new users and swift subusers under them, but only the first one is allowing auth, all others are getting 403s when attempting to auth. We googled around a bit and found the fix-zone script: https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone Which ran fine until the last command, which errors out with: + radosgw-admin zone default --rgw-zone=default WARNING: failed to initialize zonegroup the 'default' rgw-zone seems OK: $ sudo radosgw-admin zone get --zone-id=default { "id": "default", "name": "default", "domain_root": ".rgw_", "control_pool": ".rgw.control_", "gc_pool": ".rgw.gc_", "log_pool": ".log_", "intent_log_pool": ".intent-log_", "usage_log_pool": ".usage_", "user_keys_pool": ".users_", "user_email_pool": ".users.email_", "user_swift_pool": ".users.swift_", "user_uid_pool": ".users.uid_", "system_key": { "access_key": "", "secret_key": "" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": ".rgw.buckets.index_", "data_pool": ".rgw.buckets_", "data_extra_pool": ".rgw.buckets.extra_", "index_type": 0 } } ], "metadata_heap": ".rgw.meta", "realm_id": "a113de3d-c506-4112-b419-0d5c94ded7af" } Not really sure where to go from here, any help would be appreciated. -- Michael Parson _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com