Re: Regarding Federated Gateways - Zone Sync Issues

Mark Kirkwood <mark.kirkwood@xxxxxxxxxxxxxxx> · Wed, 07 Jan 2015 16:22:18 +1300

On 06/01/15 06:45, hemant burman wrote:
One more thing Yehuda,

In radosgw log in Slave Zone:

2015-01-05 17:22:42.188108 7fe4b66d2780 20 enqueued request req=0xbc1f50

2015-01-05 17:22:42.188125 7fe4b66d2780 20 RGWWQ:

2015-01-05 17:22:42.188126 7fe4b66d2780 20 req: 0xbc1f50

2015-01-05 17:22:42.188129 7fe4b66d2780 10 allocated request req=0xc1b4f0

2015-01-05 17:22:42.190310 7fe4617b2700 20 dequeued request req=0xbc1f50

2015-01-05 17:22:42.190951 7fe4617b2700 20 RGWWQ: empty

2015-01-05 17:22:42.191466 7fe4617b2700  1 ====== starting new request
req=0xbc1f50 =====

2015-01-05 17:22:42.192023 7fe4617b2700  2 req 4374:0.000558::GET
/auth::initializing

2015-01-05 17:22:42.192046 7fe4617b2700 20 FCGI_ROLE=RESPONDER

2015-01-05 17:22:42.192047 7fe4617b2700 20 SCRIPT_URL=/auth

2015-01-05 17:22:42.192047 7fe4617b2700 20
SCRIPT_URI=http://ceph-all-1:81/auth

2015-01-05 17:22:42.192048 7fe4617b2700 20 HTTP_AUTHORIZATION=

2015-01-05 17:22:42.192048 7fe4617b2700 20 HTTP_USER_AGENT=curl/7.22.0
(x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4
<http://1.2.3.4> libidn/1.23 librtmp/2.3

2015-01-05 17:22:42.192050 7fe4617b2700 20 HTTP_HOST=ceph-all-1:81

2015-01-05 17:22:42.192050 7fe4617b2700 20 HTTP_ACCEPT=*/*

2015-01-05 17:22:42.192051 7fe4617b2700 20 HTTP_X_AUTH_USER=johndoe3\:swift

2015-01-05 17:22:42.192051 7fe4617b2700 20
HTTP_X_AUTH_KEY=pehvS2YFl8QcaI1ehvBcSANkQyrQUOQqbib8V2wK

2015-01-05 17:22:42.192052 7fe4617b2700 20 PATH=/usr/local/bin:/usr/bin:/bin

2015-01-05 17:22:42.192052 7fe4617b2700 20 SERVER_SIGNATURE=

2015-01-05 17:22:42.192052 7fe4617b2700 20 SERVER_SOFTWARE=Apache

2015-01-05 17:22:42.192053 7fe4617b2700 20 SERVER_NAME=ceph-all-1

2015-01-05 17:22:42.192053 7fe4617b2700 20 SERVER_ADDR=192.168.56.108

2015-01-05 17:22:42.192054 7fe4617b2700 20 SERVER_PORT=81

2015-01-05 17:22:42.192054 7fe4617b2700 20 REMOTE_ADDR=192.168.56.107

2015-01-05 17:22:42.192054 7fe4617b2700 20 DOCUMENT_ROOT=/var/www

2015-01-05 17:22:42.192055 7fe4617b2700 20
SERVER_ADMIN=admin@xxxxxxxxxxx <mailto:admin@xxxxxxxxxxx>

2015-01-05 17:22:42.192055 7fe4617b2700 20
SCRIPT_FILENAME=/var/www/s3gw-us-1-west-1.fcgi

2015-01-05 17:22:42.192055 7fe4617b2700 20 REMOTE_PORT=46084

2015-01-05 17:22:42.192056 7fe4617b2700 20 GATEWAY_INTERFACE=CGI/1.1

2015-01-05 17:22:42.192056 7fe4617b2700 20 SERVER_PROTOCOL=HTTP/1.1

2015-01-05 17:22:42.192056 7fe4617b2700 20 REQUEST_METHOD=GET

2015-01-05 17:22:42.192057 7fe4617b2700 20 QUERY_STRING=page=auth&params=

2015-01-05 17:22:42.192057 7fe4617b2700 20 REQUEST_URI=/auth

2015-01-05 17:22:42.192058 7fe4617b2700 20 SCRIPT_NAME=/auth

2015-01-05 17:22:42.192058 7fe4617b2700  2 req
4374:0.000593:swift-auth:GET /auth::getting op

2015-01-05 17:22:42.192060 7fe4617b2700  2 req
4374:0.000595:swift-auth:GET /auth:swift_auth_get:authorizing

2015-01-05 17:22:42.192061 7fe4617b2700  2 req
4374:0.000596:swift-auth:GET /auth:swift_auth_get:reading permissions

2015-01-05 17:22:42.192062 7fe4617b2700  2 req
4374:0.000597:swift-auth:GET /auth:swift_auth_get:verifying op mask

2015-01-05 17:22:42.192063 7fe4617b2700 20 required_mask= 0 user.op_mask=7

2015-01-05 17:22:42.192064 7fe4617b2700  2 req
4374:0.000599:swift-auth:GET /auth:swift_auth_get:verifying op permissions

2015-01-05 17:22:42.192065 7fe4617b2700  2 req
4374:0.000600:swift-auth:GET /auth:swift_auth_get:verifying op params

2015-01-05 17:22:42.192066 7fe4617b2700  2 req
4374:0.000601:swift-auth:GET /auth:swift_auth_get:executing

2015-01-05 17:22:42.192082 7fe4617b2700 20 get_obj_state:
rctx=0x7fe494009210 obj=.us-1-west-1.users.swift:johndoe3\:swift
state=0x7fe49401e968 s->prefetch_data=0

2015-01-05 17:22:42.192090 7fe4617b2700 10 moving
.us-1-west-1.users.swift+johndoe3\:swift to cache LRU end

2015-01-05 17:22:42.192092 7fe4617b2700 10 cache get:
name=.us-1-west-1.users.swift+johndoe3\:swift : type miss (requested=6,
cached=0)

2015-01-05 17:22:42.197835 7fe4617b2700 10 cache put:
name=.us-1-west-1.users.swift+johndoe3\:swift

2015-01-05 17:22:42.197842 7fe4617b2700 10 moving
.us-1-west-1.users.swift+johndoe3\:swift to cache LRU end

2015-01-05 17:22:42.197871 7fe4617b2700  5 nothing to log for operation

2015-01-05 17:22:42.197873 7fe4617b2700  2 req
4374:0.006408:swift-auth:GET /auth:swift_auth_get:http status=403

*2015-01-05 17:22:42.198725 7fe4617b2700  1 ====== req done req=0xbc1f50
http_status=403 ======*

In Master Zone:

I can see a call going to build _token, but its not happening for the
slave zone, seems like its failing somewhere in rgw_swift_auth.cc, but
not sure which section could be in the bold section below
get_random_bytes or may be somewhere before that,:

static int encode_token(CephContext *cct, string& swift_user, string&
key, bufferlist& bl)
	{
	uint64_t nonce;

	*int ret = get_random_bytes((char *)&nonce, sizeof(nonce));*
	*if (ret < 0)*
	*return ret;*

	utime_t expiration = ceph_clock_now(cct);
	expiration += cct->_conf->rgw_swift_token_expiration;

	ret = build_token(swift_user, key, nonce, expiration, bl);

On Mon, Jan 5, 2015 at 11:10 PM, hemant burman <hemant.burman@xxxxxxxxx
<mailto:hemant.burman@xxxxxxxxx>> wrote:

    Hello Yehuda,

    It is exactly the same:

    2015-01-05 17:39:06.303832 7fd4458ed780 20 rados->read obj-ofs=0
    read_ofs=0 read_len=524288

    { "user_id": "johndoe3",

       "display_name": "John Doe_3",

       "email": "john_3@xxxxxxxxxxx <mailto:john_3@xxxxxxxxxxx>",

       "suspended": 0,

       "max_buckets": 1000,

       "auid": 0,

       "subusers": [

             { "id": "johndoe3:swift",

               "permissions": "full-control"}],

       "keys": [

             { "user": "johndoe3",

               "access_key": "EWN39B7F5QY392CKCSNP",

               "secret_key": "075ehemwUTRlOJxJ07gaq6Jh9CZc7bWhbNDgkfiP"}],

       "swift_keys": [

             { "user": "johndoe3:swift",

               "secret_key": "pehvS2YFl8QcaI1ehvBcSANkQyrQUOQqbib8V2wK"}],

       "caps": [],

       "op_mask": "read, write, delete",

       "default_placement": "",

       "placement_tags": []}

    2015-01-05 17:39:06.310263 7fd4458ed780 20 rados->read r=0 bl.length=469

    2015-01-05 17:39:06.310275 7fd4458ed780 10 cache put:
    name=.us-1-east-1.users.uid+johndoe3

    2015-01-05 17:39:06.310278 7fd4458ed780 10 moving
    .us-1-east-1.users.uid+johndoe3 to cache LRU end

    root@ceph-all:/var/log/radosgw# radosgw-admin user info
    --uid=johndoe3 --name client.radosgw.us-1-east-1

    2015-01-05 17:39:17.608313 7fa38ba5c780 20 rados->read obj-ofs=0
    read_ofs=0 read_len=524288

    { "user_id": "johndoe3",

       "display_name": "John Doe_3",

       "email": "john_3@xxxxxxxxxxx <mailto:john_3@xxxxxxxxxxx>",

       "suspended": 0,

       "max_buckets": 1000,

       "auid": 0,

       "subusers": [

             { "id": "johndoe3:swift",

               "permissions": "full-control"}],

       "keys": [

             { "user": "johndoe3",

               "access_key": "EWN39B7F5QY392CKCSNP",

               "secret_key": "075ehemwUTRlOJxJ07gaq6Jh9CZc7bWhbNDgkfiP"}],

       "swift_keys": [

             { "user": "johndoe3:swift",

               "secret_key": "pehvS2YFl8QcaI1ehvBcSANkQyrQUOQqbib8V2wK"}],

       "caps": [],

       "op_mask": "read, write, delete",

       "default_placement": "",

       "placement_tags": []}

    2015-01-05 17:39:17.615560 7fa38ba5c780 20 rados->read r=0 bl.length=460

    2015-01-05 17:39:17.615575 7fa38ba5c780 10 cache put:
    name=.us-1-west-1.users.uid+johndoe3

    2015-01-05 17:39:17.615579 7fa38ba5c780 10 moving
    .us-1-west-1.users.uid+johndoe3 to cache LRU end

    root@ceph-all-1:/var/log/radosgw# radosgw-admin user info
    --uid=johndoe3 --name client.radosgw.us-1-west-1

    On Mon, Jan 5, 2015 at 10:06 PM, Yehuda Sadeh <yehuda@xxxxxxxxxx
    <mailto:yehuda@xxxxxxxxxx>> wrote:

        On Sat, Nov 22, 2014 at 12:47 AM, Vinod H I <vinvinod@xxxxxxxxx
        <mailto:vinvinod@xxxxxxxxx>> wrote:
         > Thanks for the clarification.
         > Now I have done exactly as you suggested.
         > "us-east" is the master zone and "us-west" is the secondary zone.
         > Each zone has two system users "us-east" and "us-west".
         > These system users have same access/secret keys in both zones.
         > I have checked the pools to confirm that the non-system swift
        user which i
         > created("east-user:swift") in the primary has been replicated
        to the
         > secondary zone.
         > The buckets which are created in primary by the swift user
        are also there in
         > the pools of the secondary zone.
         > But when i try to authenticate this swift user in secondary
        zone, it says
         > access denied.
         >
         > Here are the relevant logs from the secondary zone, when i try to
         > authenticate the swift user.
         >
         > 2014-11-22 14:19:14.239976 7f73ecff9700  2
         > RGWDataChangesLog::ChangesRenewThread: start
         > 2014-11-22 14:19:14.243454 7f73fe236780 20 get_obj_state:
        rctx=0x2316ce0
         > obj=.us.rgw.root:region_info.us <http://region_info.us>
        state=0x2319048 s->prefetch_data=0
         > 2014-11-22 14:19:14.243454 7f73fe236780 10 cache get:
         > name=.us.rgw.root+region_info.us <http://region_info.us> : miss
         > 2014-11-22 14:19:14.252263 7f73fe236780 10 cache put:
         > name=.us.rgw.root+region_info.us <http://region_info.us>
         > 2014-11-22 14:19:14.252283 7f73fe236780 10 adding
         > .us.rgw.root+region_info.us <http://region_info.us> to cache
        LRU end
         > 2014-11-22 14:19:14.252310 7f73fe236780 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:14.252336 7f73fe236780 10 cache get:
         > name=.us.rgw.root+region_info.us <http://region_info.us> :
        type miss (requested=1, cached=6)
         > 2014-11-22 14:19:14.252376 7f73fe236780 20 get_obj_state:
        rctx=0x2316ce0
         > obj=.us.rgw.root:region_info.us <http://region_info.us>
        state=0x2319958 s->prefetch_data=0
         > 2014-11-22 14:19:14.252386 7f73fe236780 10 cache get:
         > name=.us.rgw.root+region_info.us <http://region_info.us> : hit
         > 2014-11-22 14:19:14.252391 7f73fe236780 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:14.252404 7f73fe236780 20 get_obj_state:
        rctx=0x2316ce0
         > obj=.us.rgw.root:region_info.us <http://region_info.us>
        state=0x2319958 s->prefetch_data=0
         > 2014-11-22 14:19:14.252409 7f73fe236780 20 state for
         > obj=.us.rgw.root:region_info.us <http://region_info.us> is
        not atomic, not appending atomic test
         > 2014-11-22 14:19:14.252412 7f73fe236780 20 rados->read
        obj-ofs=0 read_ofs=0
         > read_len=524288
         > 2014-11-22 14:19:14.264611 7f73fe236780 20 rados->read r=0
        bl.length=266
         > 2014-11-22 14:19:14.264650 7f73fe236780 10 cache put:
         > name=.us.rgw.root+region_info.us <http://region_info.us>
         > 2014-11-22 14:19:14.264653 7f73fe236780 10 moving
         > .us.rgw.root+region_info.us <http://region_info.us> to cache
        LRU end
         > 2014-11-22 14:19:14.264766 7f73fe236780 20 get_obj_state:
        rctx=0x2319860
         > obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98
        s->prefetch_data=0
         > 2014-11-22 14:19:14.264779 7f73fe236780 10 cache get:
         > name=.us-west.rgw.root+zone_info.us-west : miss
         > 2014-11-22 14:19:14.276114 7f73fe236780 10 cache put:
         > name=.us-west.rgw.root+zone_info.us-west
         > 2014-11-22 14:19:14.276131 7f73fe236780 10 adding
         > .us-west.rgw.root+zone_info.us-west to cache LRU end
         > 2014-11-22 14:19:14.276142 7f73fe236780 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:14.276161 7f73fe236780 10 cache get:
         > name=.us-west.rgw.root+zone_info.us-west : type miss
        (requested=1, cached=6)
         > 2014-11-22 14:19:14.276203 7f73fe236780 20 get_obj_state:
        rctx=0x2314660
         > obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98
        s->prefetch_data=0
         > 2014-11-22 14:19:14.276212 7f73fe236780 10 cache get:
         > name=.us-west.rgw.root+zone_info.us-west : hit
         > 2014-11-22 14:19:14.276218 7f73fe236780 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:14.276229 7f73fe236780 20 get_obj_state:
        rctx=0x2314660
         > obj=.us-west.rgw.root:zone_info.us-west state=0x2313b98
        s->prefetch_data=0
         > 2014-11-22 14:19:14.276235 7f73fe236780 20 state for
         > obj=.us-west.rgw.root:zone_info.us-west is not atomic, not
        appending atomic
         > test
         > 2014-11-22 14:19:14.276238 7f73fe236780 20 rados->read
        obj-ofs=0 read_ofs=0
         > read_len=524288
         > 2014-11-22 14:19:14.290757 7f73fe236780 20 rados->read r=0
        bl.length=997
         > 2014-11-22 14:19:14.290797 7f73fe236780 10 cache put:
         > name=.us-west.rgw.root+zone_info.us-west
         > 2014-11-22 14:19:14.290803 7f73fe236780 10 moving
         > .us-west.rgw.root+zone_info.us-west to cache LRU end
         > 2014-11-22 14:19:14.290857 7f73fe236780  2 zone us-west is
        NOT master
         > 2014-11-22 14:19:14.290931 7f73fe236780 20 get_obj_state:
        rctx=0x2313cc0
         > obj=.us-west.rgw.root:region_map state=0x2311e08
        s->prefetch_data=0
         > 2014-11-22 14:19:14.290949 7f73fe236780 10 cache get:
         > name=.us-west.rgw.root+region_map : miss
         > 2014-11-22 14:19:14.298169 7f73fe236780 10 cache put:
         > name=.us-west.rgw.root+region_map
         > 2014-11-22 14:19:14.298184 7f73fe236780 10 adding
         > .us-west.rgw.root+region_map to cache LRU end
         > 2014-11-22 14:19:14.298195 7f73fe236780 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:14.298212 7f73fe236780 10 cache get:
         > name=.us-west.rgw.root+region_map : type miss (requested=1,
        cached=6)
         > 2014-11-22 14:19:14.298255 7f73fe236780 20 get_obj_state:
        rctx=0x2313cc0
         > obj=.us-west.rgw.root:region_map state=0x2311e08
        s->prefetch_data=0
         > 2014-11-22 14:19:14.298267 7f73fe236780 10 cache get:
         > name=.us-west.rgw.root+region_map : hit
         > 2014-11-22 14:19:14.298272 7f73fe236780 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:14.298282 7f73fe236780 20 get_obj_state:
        rctx=0x2313cc0
         > obj=.us-west.rgw.root:region_map state=0x2311e08
        s->prefetch_data=0
         > 2014-11-22 14:19:14.298286 7f73fe236780 20 state for
         > obj=.us-west.rgw.root:region_map is not atomic, not appending
        atomic test
         > 2014-11-22 14:19:14.298288 7f73fe236780 20 rados->read
        obj-ofs=0 read_ofs=0
         > read_len=524288
         > 2014-11-22 14:19:14.300462 7f73fe236780 20 rados->read r=0
        bl.length=334
         > 2014-11-22 14:19:14.300486 7f73fe236780 10 cache put:
         > name=.us-west.rgw.root+region_map
         > 2014-11-22 14:19:14.300490 7f73fe236780 10 moving
         > .us-west.rgw.root+region_map to cache LRU end
         > 2014-11-22 14:19:14.545543 7f73fe236780 20 generating
        connection object for
         > zone us-east
         > 2014-11-22 14:19:14.548178 7f73fe236780  0 framework: fastcgi
         > 2014-11-22 14:19:14.548767 7f73fe236780  0 starting handler:
        fastcgi
         > 2014-11-22 14:19:14.549768 7f73ceffd700 20 UserSyncThread: start
         > 2014-11-22 14:19:14.550665 7f73cf7fe700 20 BucketsSyncThread:
        start
         > 2014-11-22 14:19:14.553127 7f73cdffb700 10 allocated request
         > req=0x7f73e000d010
         > 2014-11-22 14:19:14.553458 7f73cffff700  2 garbage
        collection: start
         > 2014-11-22 14:19:14.576569 7f73ceffd700 20
        RGWRados::pool_iterate: got
         > east-user.buckets
         > 2014-11-22 14:19:14.731146 7f73ceffd700 20
        RGWRados::pool_iterate: got
         > us-west
         > 2014-11-22 14:19:14.771842 7f73ceffd700 20
        RGWRados::pool_iterate: got
         > us-east
         > 2014-11-22 14:19:14.803904 7f73ceffd700 20
        RGWRados::pool_iterate: got
         > east-user
         > 2014-11-22 14:19:14.834887 7f73ceffd700 20 RGWUserStatsCache:
        sync
         > user=us-west
         > 2014-11-22 14:19:14.839166 7f73ceffd700  0 ERROR: can't read
        user header:
         > ret=-2
         > 2014-11-22 14:19:14.839182 7f73ceffd700  0 ERROR: sync_user()
        failed,
         > user=us-west ret=-2
         > 2014-11-22 14:19:14.839186 7f73ceffd700 20 RGWUserStatsCache:
        sync
         > user=us-east
         > 2014-11-22 14:19:14.846530 7f73ceffd700  0 ERROR: can't read
        user header:
         > ret=-2
         > 2014-11-22 14:19:14.846540 7f73ceffd700  0 ERROR: sync_user()
        failed,
         > user=us-east ret=-2
         > 2014-11-22 14:19:14.846543 7f73ceffd700 20 RGWUserStatsCache:
        sync
         > user=east-user
         > 2014-11-22 14:19:14.852550 7f73ceffd700 20 user is idle, not
        doing a full
         > sync (user=east-user)
         > 2014-11-22 14:19:15.994741 7f73cffff700  2 garbage
        collection: stop
         > 2014-11-22 14:19:21.917609 7f73cdffb700 20 enqueued request
         > req=0x7f73e000d010
         > 2014-11-22 14:19:21.917657 7f73cdffb700 20 RGWWQ:
         > 2014-11-22 14:19:21.917661 7f73cdffb700 20 req: 0x7f73e000d010
         > 2014-11-22 14:19:21.917679 7f73cdffb700 10 allocated request
         > req=0x7f73e0013dc0
         > 2014-11-22 14:19:21.918043 7f73b07c0700 20 dequeued request
         > req=0x7f73e000d010
         > 2014-11-22 14:19:21.918067 7f73b07c0700 20 RGWWQ: empty
         > 2014-11-22 14:19:21.918220 7f73b07c0700 20 DOCUMENT_ROOT=/var/www
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20 FCGI_ROLE=RESPONDER
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
        GATEWAY_INTERFACE=CGI/1.1
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
        HTTP_ACCEPT_ENCODING=identity
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20 HTTP_AUTHORIZATION=
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
        HTTP_HOST=us-west-1.lt.com <http://us-west-1.lt.com>
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
         > HTTP_X_AUTH_KEY=MHA4vFaDy5XsJqpF5NuZLcBMCoJcuot44ASDuReY
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
        HTTP_X_AUTH_USER=east-user:swift
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
        PATH=/usr/local/bin:/usr/bin:/bin
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20 QUERY_STRING=
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
        REMOTE_ADDR=192.168.7.141
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20 REMOTE_PORT=50857
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20 REQUEST_METHOD=GET
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20 REQUEST_URI=/auth/
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
         > SCRIPT_FILENAME=/var/www/s3gw.fcgi
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20 SCRIPT_NAME=/auth/
         > 2014-11-22 14:19:21.918228 7f73b07c0700 20
         > SCRIPT_URI=http://us-west-1.lt.com/auth/
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20 SCRIPT_URL=/auth/
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20
        SERVER_ADDR=192.168.7.117
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20
        SERVER_ADMIN=vinvinod@xxxxxxxxx <mailto:vinvinod@xxxxxxxxx>
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20
        SERVER_NAME=us-west-1.lt.com <http://us-west-1.lt.com>
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_PORT=80
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20
        SERVER_PROTOCOL=HTTP/1.1
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20 SERVER_SIGNATURE=
         > 2014-11-22 14:19:21.918229 7f73b07c0700 20
        SERVER_SOFTWARE=Apache/2.2.22
         > (Ubuntu)
         > 2014-11-22 14:19:21.918229 7f73b07c0700  1 ====== starting
        new request
         > req=0x7f73e000d010 =====
         > 2014-11-22 14:19:21.918229 7f73b07c0700  2 req 1:0.000000::GET
         > /auth/::initializing
         > 2014-11-22 14:19:21.918229 7f73b07c0700 10
        host=us-west-1.lt.com <http://us-west-1.lt.com>
         > rgw_dns_name=us-west-1.lt.com <http://us-west-1.lt.com>
         > 2014-11-22 14:19:21.918288 7f73b07c0700  2 req
        1:0.000053:swift-auth:GET
         > /auth/::getting op
         > 2014-11-22 14:19:21.918300 7f73b07c0700  2 req
        1:0.000071:swift-auth:GET
         > /auth/:swift_auth_get:authorizing
         > 2014-11-22 14:19:21.918307 7f73b07c0700  2 req
        1:0.000078:swift-auth:GET
         > /auth/:swift_auth_get:reading permissions
         > 2014-11-22 14:19:21.918313 7f73b07c0700  2 req
        1:0.000084:swift-auth:GET
         > /auth/:swift_auth_get:init op
         > 2014-11-22 14:19:21.918319 7f73b07c0700  2 req
        1:0.000090:swift-auth:GET
         > /auth/:swift_auth_get:verifying op mask
         > 2014-11-22 14:19:21.918325 7f73b07c0700 20 required_mask= 0
        user.op_mask=7
         > 2014-11-22 14:19:21.918330 7f73b07c0700  2 req
        1:0.000100:swift-auth:GET
         > /auth/:swift_auth_get:verifying op permissions
         > 2014-11-22 14:19:21.918336 7f73b07c0700  2 req
        1:0.000107:swift-auth:GET
         > /auth/:swift_auth_get:verifying op params
         > 2014-11-22 14:19:21.918341 7f73b07c0700  2 req
        1:0.000112:swift-auth:GET
         > /auth/:swift_auth_get:executing
         > 2014-11-22 14:19:21.918470 7f73b07c0700 20 get_obj_state:
         > rctx=0x7f73dc002030 obj=.us-west.users.swift:east-user:swift
         > state=0x7f73dc0066d8 s->prefetch_data=0
         > 2014-11-22 14:19:21.918494 7f73b07c0700 10 cache get:
         > name=.us-west.users.swift+east-user:swift : miss
         > 2014-11-22 14:19:21.931892 7f73b07c0700 10 cache put:
         > name=.us-west.users.swift+east-user:swift
         > 2014-11-22 14:19:21.931892 7f73b07c0700 10 adding
         > .us-west.users.swift+east-user:swift to cache LRU end
         > 2014-11-22 14:19:21.931892 7f73b07c0700 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:21.931892 7f73b07c0700 10 cache get:
         > name=.us-west.users.swift+east-user:swift : type miss
        (requested=1,
         > cached=6)
         > 2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state:
         > rctx=0x7f73dc007300 obj=.us-west.users.swift:east-user:swift
         > state=0x7f73dc006558 s->prefetch_data=0
         > 2014-11-22 14:19:21.931893 7f73b07c0700 10 cache get:
         > name=.us-west.users.swift+east-user:swift : hit
         > 2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:21.931893 7f73b07c0700 20 get_obj_state:
         > rctx=0x7f73dc007300 obj=.us-west.users.swift:east-user:swift
         > state=0x7f73dc006558 s->prefetch_data=0
         > 2014-11-22 14:19:21.931893 7f73b07c0700 20 state for
         > obj=.us-west.users.swift:east-user:swift is not atomic, not
        appending atomic
         > test
         > 2014-11-22 14:19:21.931893 7f73b07c0700 20 rados->read
        obj-ofs=0 read_ofs=0
         > read_len=524288
         > 2014-11-22 14:19:21.932003 7f73b07c0700 20 rados->read r=0
        bl.length=13
         > 2014-11-22 14:19:21.932021 7f73b07c0700 10 cache put:
         > name=.us-west.users.swift+east-user:swift
         > 2014-11-22 14:19:21.932023 7f73b07c0700 10 moving
         > .us-west.users.swift+east-user:swift to cache LRU end
         > 2014-11-22 14:19:21.932054 7f73b07c0700 20 get_obj_state:
         > rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user
        state=0x7f73dc006498
         > s->prefetch_data=0
         > 2014-11-22 14:19:21.932062 7f73b07c0700 10 cache get:
         > name=.us-west.users.uid+east-user : miss
         > 2014-11-22 14:19:21.933559 7f73b07c0700 10 cache put:
         > name=.us-west.users.uid+east-user
         > 2014-11-22 14:19:21.933567 7f73b07c0700 10 adding
         > .us-west.users.uid+east-user to cache LRU end
         > 2014-11-22 14:19:21.933572 7f73b07c0700 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:21.933580 7f73b07c0700 10 cache get:
         > name=.us-west.users.uid+east-user : type miss (requested=1,
        cached=6)
         > 2014-11-22 14:19:21.933601 7f73b07c0700 20 get_obj_state:
         > rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user
        state=0x7f73dc006498
         > s->prefetch_data=0
         > 2014-11-22 14:19:21.933607 7f73b07c0700 10 cache get:
         > name=.us-west.users.uid+east-user : hit
         > 2014-11-22 14:19:21.933611 7f73b07c0700 20 get_obj_state:
        s->obj_tag was set
         > empty
         > 2014-11-22 14:19:21.933617 7f73b07c0700 20 get_obj_state:
         > rctx=0x7f73dc006b30 obj=.us-west.users.uid:east-user
        state=0x7f73dc006498
         > s->prefetch_data=0
         > 2014-11-22 14:19:21.933620 7f73b07c0700 20 state for
         > obj=.us-west.users.uid:east-user is not atomic, not appending
        atomic test
         > 2014-11-22 14:19:21.933622 7f73b07c0700 20 rados->read
        obj-ofs=0 read_ofs=0
         > read_len=524288
         > 2014-11-22 14:19:21.934709 7f73b07c0700 20 rados->read r=0
        bl.length=310
         > 2014-11-22 14:19:21.934725 7f73b07c0700 10 cache put:
         > name=.us-west.users.uid+east-user
         > 2014-11-22 14:19:21.934727 7f73b07c0700 10 moving
         > .us-west.users.uid+east-user to cache LRU end
         > 2014-11-22 14:19:21.934790 7f73b07c0700  2 req
        1:0.016560:swift-auth:GET
         > /auth/:swift_auth_get:http status=403
         > 2014-11-22 14:19:21.934794 7f73b07c0700  1 ====== req done
         > req=0x7f73e000d010 http_status=403 ======
         > 2014-11-22 14:19:21.934800 7f73b07c0700 20 process_request()
        returned -1
         >
         > Why am I not able to authenticate?
         >

        What does radosgw-admin show about the user in the secondary zone?

        $ radosgw-admin user info --rgw-zone=<zone> --uid=<uid>

        Try comparing to the primary zone.

FWIW I can reproduce this too (ceph 0.90-663-ge1384af). The *user* 
replicates ok (complete with its swift keys and secret). I can 
authenticate to both zones ok using S3 api (boto version 2.29), but only 
to the master using swift (swift client versions 2.3.1 and 2.0.3). In 
the case of the slave zone I'm seeing the same error stack as the above.

I'm running Ubuntu 14.10 for ceph and rgw with Apache (version 2.4.10) 
the standard repos. I'll try replacing the fastcgi module to see if that 
is a factor.

Regards

Mark
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com