Re: Regarding Federated Gateways - Zone Sync Issues

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hello,

Is there an answer to why this is happening, I am facing the same issue, I have the non-system user replicated to the slave zone, but still getting 403, the same thing happening when I am replicating from the master zone of master region to master zone of secondary region.
I am using swift, and have created a non-system user for the same

-Hemant

On Tue, Nov 25, 2014 at 12:37 AM, Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx> wrote:
I'm really not sure.  I'm using the S3 interface rather than the Swift interface.  Once my non-systm user replicated, I was able to access everything in the secondary cluster just fine.

Hopefully somebody else with Swift experience will chime in.



On Sat, Nov 22, 2014 at 12:47 AM, Vinod H I <vinvinod@xxxxxxxxx> wrote:
Thanks for the clarification.
Now I have done exactly as you suggested.
"us-east" is the master zone and "us-west" is the secondary zone.
Each zone has two system users "us-east" and "us-west".
These system users have same access/secret keys in both zones.
I have checked the pools to confirm that the non-system swift user which i created("east-user:swift") in the primary has been replicated to the secondary zone.
The buckets which are created in primary by the swift user are also there in the pools of the secondary zone.
But when i try to authenticate this swift user in secondary zone, it says access denied.

Here are the relevant logs from the secondary zone, when i try to authenticate the swift user.

2014-11-22 14:19:14.239976 7f73ecff9700  2 RGWDataChangesLog::ChangesRenewThread: start
2014-11-22 14:19:14.243454 7f73fe236780 20 get_obj_state: rctx=0x2316ce0 obj=.us.rgw.root:region_info.us state=0x2319048 s->prefetch_data=0
2014-11-22 14:19:14.243454 7f73fe236780 10 cache get: name=.us.rgw.root+region_info.us : miss
2014-11-22 14:19:14.252263 7f73fe236780 10 cache put: name=.us.rgw.root+region_info.us
2014-11-22 14:19:14.252283 7f73fe236780 10 adding .us.rgw.root+region_info.us to cache LRU end
2014-11-22 14:19:14.252310 7f73fe236780 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:14.252336 7f73fe236780 10 cache get: name=.us.rgw.root+region_info.us : type miss (requested=1, cached=6)
2014-11-22 14:19:14.252376 7f73fe236780 20 get_obj_state: rctx=0x2316ce0 obj=.us.rgw.root:region_info.us state=0x2319958 s->prefetch_data=0
2014-11-22 14:19:14.252386 7f73fe236780 10 cache get: name=.us.rgw.root+region_info.us : hit
2014-11-22 14:19:14.252391 7f73fe236780 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:14.252404 7f73fe236780 20 get_obj_state: rctx=0x2316ce0 obj=.us.rgw.root:region_info.us state=0x2319958 s->prefetch_data=0
2014-11-22 14:19:14.252409 7f73fe236780 20 state for obj=.us.rgw.root:region_info.us is not atomic, not appending atomic test
2014-11-22 14:19:14.252412 7f73fe236780 20 rados->read obj-ofs=0 read_ofs=0 read_len=524288
2014-11-22 14:19:14.264611 7f73fe236780 20 rados->read r=0 bl.length=266
2014-11-22 14:19:14.264650 7f73fe236780 10 cache put: name=.us.rgw.root+region_info.us
2014-11-22 14:19:14.264653 7f73fe236780 10 moving .us.rgw.root+region_info.us to cache LRU end
2014-11-22 14:19:14.264766 7f73fe236780 20 get_obj_state: rctx=0x2319860 obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98 s->prefetch_data=0
2014-11-22 14:19:14.264779 7f73fe236780 10 cache get: name=.us-west.rgw.root+zone_info.us-west : miss
2014-11-22 14:19:14.276114 7f73fe236780 10 cache put: name=.us-west.rgw.root+zone_info.us-west
2014-11-22 14:19:14.276131 7f73fe236780 10 adding .us-west.rgw.root+zone_info.us-west to cache LRU end
2014-11-22 14:19:14.276142 7f73fe236780 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:14.276161 7f73fe236780 10 cache get: name=.us-west.rgw.root+zone_info.us-west : type miss (requested=1, cached=6)
2014-11-22 14:19:14.276203 7f73fe236780 20 get_obj_state: rctx=0x2314660 obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98 s->prefetch_data=0
2014-11-22 14:19:14.276212 7f73fe236780 10 cache get: name=.us-west.rgw.root+zone_info.us-west : hit
2014-11-22 14:19:14.276218 7f73fe236780 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:14.276229 7f73fe236780 20 get_obj_state: rctx=0x2314660 obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98 s->prefetch_data=0
2014-11-22 14:19:14.276235 7f73fe236780 20 state for obj=.us-west.rgw.root:zone_info.us-west is not atomic, not appending atomic test
2014-11-22 14:19:14.276238 7f73fe236780 20 rados->read obj-ofs=0 read_ofs=0 read_len=524288
2014-11-22 14:19:14.290757 7f73fe236780 20 rados->read r=0 bl.length=997
2014-11-22 14:19:14.290797 7f73fe236780 10 cache put: name=.us-west.rgw.root+zone_info.us-west
2014-11-22 14:19:14.290803 7f73fe236780 10 moving .us-west.rgw.root+zone_info.us-west to cache LRU end
2014-11-22 14:19:14.290857 7f73fe236780  2 zone us-west is NOT master
2014-11-22 14:19:14.290931 7f73fe236780 20 get_obj_state: rctx=0x2313cc0 obj=.us-west.rgw.root:region_map state=0x2311e08 s->prefetch_data=0
2014-11-22 14:19:14.290949 7f73fe236780 10 cache get: name=.us-west.rgw.root+region_map : miss
2014-11-22 14:19:14.298169 7f73fe236780 10 cache put: name=.us-west.rgw.root+region_map
2014-11-22 14:19:14.298184 7f73fe236780 10 adding .us-west.rgw.root+region_map to cache LRU end
2014-11-22 14:19:14.298195 7f73fe236780 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:14.298212 7f73fe236780 10 cache get: name=.us-west.rgw.root+region_map : type miss (requested=1, cached=6)
2014-11-22 14:19:14.298255 7f73fe236780 20 get_obj_state: rctx=0x2313cc0 obj=.us-west.rgw.root:region_map state=0x2311e08 s->prefetch_data=0
2014-11-22 14:19:14.298267 7f73fe236780 10 cache get: name=.us-west.rgw.root+region_map : hit
2014-11-22 14:19:14.298272 7f73fe236780 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:14.298282 7f73fe236780 20 get_obj_state: rctx=0x2313cc0 obj=.us-west.rgw.root:region_map state=0x2311e08 s->prefetch_data=0
2014-11-22 14:19:14.298286 7f73fe236780 20 state for obj=.us-west.rgw.root:region_map is not atomic, not appending atomic test
2014-11-22 14:19:14.298288 7f73fe236780 20 rados->read obj-ofs=0 read_ofs=0 read_len=524288
2014-11-22 14:19:14.300462 7f73fe236780 20 rados->read r=0 bl.length=334
2014-11-22 14:19:14.300486 7f73fe236780 10 cache put: name=.us-west.rgw.root+region_map
2014-11-22 14:19:14.300490 7f73fe236780 10 moving .us-west.rgw.root+region_map to cache LRU end
2014-11-22 14:19:14.545543 7f73fe236780 20 generating connection object for zone us-east
2014-11-22 14:19:14.548178 7f73fe236780  0 framework: fastcgi
2014-11-22 14:19:14.548767 7f73fe236780  0 starting handler: fastcgi
2014-11-22 14:19:14.549768 7f73ceffd700 20 UserSyncThread: start
2014-11-22 14:19:14.550665 7f73cf7fe700 20 BucketsSyncThread: start
2014-11-22 14:19:14.553127 7f73cdffb700 10 allocated request req=0x7f73e000d010
2014-11-22 14:19:14.553458 7f73cffff700  2 garbage collection: start
2014-11-22 14:19:14.576569 7f73ceffd700 20 RGWRados::pool_iterate: got east-user.buckets
2014-11-22 14:19:14.731146 7f73ceffd700 20 RGWRados::pool_iterate: got us-west
2014-11-22 14:19:14.771842 7f73ceffd700 20 RGWRados::pool_iterate: got us-east
2014-11-22 14:19:14.803904 7f73ceffd700 20 RGWRados::pool_iterate: got east-user
2014-11-22 14:19:14.834887 7f73ceffd700 20 RGWUserStatsCache: sync user=us-west
2014-11-22 14:19:14.839166 7f73ceffd700  0 ERROR: can't read user header: ret=-2
2014-11-22 14:19:14.839182 7f73ceffd700  0 ERROR: sync_user() failed, user=us-west ret=-2
2014-11-22 14:19:14.839186 7f73ceffd700 20 RGWUserStatsCache: sync user=us-east
2014-11-22 14:19:14.846530 7f73ceffd700  0 ERROR: can't read user header: ret=-2
2014-11-22 14:19:14.846540 7f73ceffd700  0 ERROR: sync_user() failed, user=us-east ret=-2
2014-11-22 14:19:14.846543 7f73ceffd700 20 RGWUserStatsCache: sync user=east-user
2014-11-22 14:19:14.852550 7f73ceffd700 20 user is idle, not doing a full sync (user=east-user)
2014-11-22 14:19:15.994741 7f73cffff700  2 garbage collection: stop
2014-11-22 14:19:21.917609 7f73cdffb700 20 enqueued request req=0x7f73e000d010
2014-11-22 14:19:21.917657 7f73cdffb700 20 RGWWQ:
2014-11-22 14:19:21.917661 7f73cdffb700 20 req: 0x7f73e000d010
2014-11-22 14:19:21.917679 7f73cdffb700 10 allocated request req=0x7f73e0013dc0
2014-11-22 14:19:21.918043 7f73b07c0700 20 dequeued request req=0x7f73e000d010
2014-11-22 14:19:21.918067 7f73b07c0700 20 RGWWQ: empty
2014-11-22 14:19:21.918220 7f73b07c0700 20 DOCUMENT_ROOT=/var/www
2014-11-22 14:19:21.918228 7f73b07c0700 20 FCGI_ROLE=RESPONDER
2014-11-22 14:19:21.918228 7f73b07c0700 20 GATEWAY_INTERFACE=CGI/1.1
2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_ACCEPT_ENCODING=identity
2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_AUTHORIZATION=
2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_HOST=us-west-1.lt.com
2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_X_AUTH_KEY=MHA4vFaDy5XsJqpF5NuZLcBMCoJcuot44ASDuReY
2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_X_AUTH_USER=east-user:swift
2014-11-22 14:19:21.918228 7f73b07c0700 20 PATH=/usr/local/bin:/usr/bin:/bin
2014-11-22 14:19:21.918228 7f73b07c0700 20 QUERY_STRING=
2014-11-22 14:19:21.918228 7f73b07c0700 20 REMOTE_ADDR=192.168.7.141
2014-11-22 14:19:21.918228 7f73b07c0700 20 REMOTE_PORT=50857
2014-11-22 14:19:21.918228 7f73b07c0700 20 REQUEST_METHOD=GET
2014-11-22 14:19:21.918228 7f73b07c0700 20 REQUEST_URI=/auth/
2014-11-22 14:19:21.918228 7f73b07c0700 20 SCRIPT_FILENAME=/var/www/s3gw.fcgi
2014-11-22 14:19:21.918228 7f73b07c0700 20 SCRIPT_NAME=/auth/
2014-11-22 14:19:21.918228 7f73b07c0700 20 SCRIPT_URI=http://us-west-1.lt.com/auth/
2014-11-22 14:19:21.918229 7f73b07c0700 20 SCRIPT_URL=/auth/
2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_ADDR=192.168.7.117
2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_ADMIN=vinvinod@xxxxxxxxx
2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_NAME=us-west-1.lt.com
2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_PORT=80
2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_PROTOCOL=HTTP/1.1
2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_SIGNATURE=
2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_SOFTWARE=Apache/2.2.22 (Ubuntu)
2014-11-22 14:19:21.918229 7f73b07c0700  1 ====== starting new request req=0x7f73e000d010 =====
2014-11-22 14:19:21.918229 7f73b07c0700  2 req 1:0.000000::GET /auth/::initializing
2014-11-22 14:19:21.918229 7f73b07c0700 10 host=us-west-1.lt.com rgw_dns_name=us-west-1.lt.com
2014-11-22 14:19:21.918288 7f73b07c0700  2 req 1:0.000053:swift-auth:GET /auth/::getting op
2014-11-22 14:19:21.918300 7f73b07c0700  2 req 1:0.000071:swift-auth:GET /auth/:swift_auth_get:authorizing
2014-11-22 14:19:21.918307 7f73b07c0700  2 req 1:0.000078:swift-auth:GET /auth/:swift_auth_get:reading permissions
2014-11-22 14:19:21.918313 7f73b07c0700  2 req 1:0.000084:swift-auth:GET /auth/:swift_auth_get:init op
2014-11-22 14:19:21.918319 7f73b07c0700  2 req 1:0.000090:swift-auth:GET /auth/:swift_auth_get:verifying op mask
2014-11-22 14:19:21.918325 7f73b07c0700 20 required_mask= 0 user.op_mask=7
2014-11-22 14:19:21.918330 7f73b07c0700  2 req 1:0.000100:swift-auth:GET /auth/:swift_auth_get:verifying op permissions
2014-11-22 14:19:21.918336 7f73b07c0700  2 req 1:0.000107:swift-auth:GET /auth/:swift_auth_get:verifying op params
2014-11-22 14:19:21.918341 7f73b07c0700  2 req 1:0.000112:swift-auth:GET /auth/:swift_auth_get:executing
2014-11-22 14:19:21.918470 7f73b07c0700 20 get_obj_state: rctx=0x7f73dc002030 obj=.us-west.users.swift:east-user:swift state=0x7f73dc0066d8 s->prefetch_data=0
2014-11-22 14:19:21.918494 7f73b07c0700 10 cache get: name=.us-west.users.swift+east-user:swift : miss
2014-11-22 14:19:21.931892 7f73b07c0700 10 cache put: name=.us-west.users.swift+east-user:swift
2014-11-22 14:19:21.931892 7f73b07c0700 10 adding .us-west.users.swift+east-user:swift to cache LRU end
2014-11-22 14:19:21.931892 7f73b07c0700 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:21.931892 7f73b07c0700 10 cache get: name=.us-west.users.swift+east-user:swift : type miss (requested=1, cached=6)
2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state: rctx=0x7f73dc007300 obj=.us-west.users.swift:east-user:swift state=0x7f73dc006558 s->prefetch_data=0
2014-11-22 14:19:21.931893 7f73b07c0700 10 cache get: name=.us-west.users.swift+east-user:swift : hit
2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state: rctx=0x7f73dc007300 obj=.us-west.users.swift:east-user:swift state=0x7f73dc006558 s->prefetch_data=0
2014-11-22 14:19:21.931893 7f73b07c0700 20 state for obj=.us-west.users.swift:east-user:swift is not atomic, not appending atomic test
2014-11-22 14:19:21.931893 7f73b07c0700 20 rados->read obj-ofs=0 read_ofs=0 read_len=524288
2014-11-22 14:19:21.932003 7f73b07c0700 20 rados->read r=0 bl.length=13
2014-11-22 14:19:21.932021 7f73b07c0700 10 cache put: name=.us-west.users.swift+east-user:swift
2014-11-22 14:19:21.932023 7f73b07c0700 10 moving .us-west.users.swift+east-user:swift to cache LRU end
2014-11-22 14:19:21.932054 7f73b07c0700 20 get_obj_state: rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user state=0x7f73dc006498 s->prefetch_data=0
2014-11-22 14:19:21.932062 7f73b07c0700 10 cache get: name=.us-west.users.uid+east-user : miss
2014-11-22 14:19:21.933559 7f73b07c0700 10 cache put: name=.us-west.users.uid+east-user
2014-11-22 14:19:21.933567 7f73b07c0700 10 adding .us-west.users.uid+east-user to cache LRU end
2014-11-22 14:19:21.933572 7f73b07c0700 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:21.933580 7f73b07c0700 10 cache get: name=.us-west.users.uid+east-user : type miss (requested=1, cached=6)
2014-11-22 14:19:21.933601 7f73b07c0700 20 get_obj_state: rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user state=0x7f73dc006498 s->prefetch_data=0
2014-11-22 14:19:21.933607 7f73b07c0700 10 cache get: name=.us-west.users.uid+east-user : hit
2014-11-22 14:19:21.933611 7f73b07c0700 20 get_obj_state: s->obj_tag was set empty
2014-11-22 14:19:21.933617 7f73b07c0700 20 get_obj_state: rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user state=0x7f73dc006498 s->prefetch_data=0
2014-11-22 14:19:21.933620 7f73b07c0700 20 state for obj=.us-west.users.uid:east-user is not atomic, not appending atomic test
2014-11-22 14:19:21.933622 7f73b07c0700 20 rados->read obj-ofs=0 read_ofs=0 read_len=524288
2014-11-22 14:19:21.934709 7f73b07c0700 20 rados->read r=0 bl.length=310
2014-11-22 14:19:21.934725 7f73b07c0700 10 cache put: name=.us-west.users.uid+east-user
2014-11-22 14:19:21.934727 7f73b07c0700 10 moving .us-west.users.uid+east-user to cache LRU end
2014-11-22 14:19:21.934790 7f73b07c0700  2 req 1:0.016560:swift-auth:GET /auth/:swift_auth_get:http status=403
2014-11-22 14:19:21.934794 7f73b07c0700  1 ====== req done req=0x7f73e000d010 http_status=403 ======
2014-11-22 14:19:21.934800 7f73b07c0700 20 process_request() returned -1

Why am I not able to authenticate?

On Fri, Nov 21, 2014 at 1:04 AM, Craig Lewis <clewis@xxxxxxxxxxxxxxxxxx> wrote:
You need to create two system users, in both zones.  They should have the same name, access key, and secret in both zones.  By convention, these system users are named the same as the zones.

You shouldn't use those system users for anything other than replication.  You should create a non-system user to interact with the cluster.  Just like you don't run as root all the time, you don't want to be a radosgw system user all the time.  You only need to create this user in the primary zone.

Once replication is working, it should copy the non-system user to the secondary cluster, as well as any buckets and objects this user creates.


On Wed, Nov 19, 2014 at 1:16 AM, Vinod H I <vinvinod@xxxxxxxxx> wrote:
Hi, 
I am using firefly version 0.80.7.
I am testing disaster recovery mechanism for rados gateways.
I have followed the federated gateway setup as mentioned in the docs.
There is one region with two zones on the same cluster.
After sync(using radosgw-agent, with "--sync-scope=full"), container created by the swift user(with "--system" flag) on the master zone gateway is not visible for the swift user(with "--system" flag) on the slave zone.
There are no error during the syncing process.
I tried by creating a new slave zone user with same uid and access and secret keys as that of master. It did not work!
Any idea on how to be able to read the synced containers from the slave zone?
Is there any requirement that the two zones must be on separate clusters?
--
Vinod H I


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





--
Vinod H I



_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux