We're using the Hammer version of RadosGW, with Keystone for authN/Z.
When a user started sending a lot of S3 requests (using rclone), the
load on our Keystone service has skyrocketed.
This surprised me because all those requests are from the same user, and
RadosGW has caching for Keystone tokens. But looking at the code, this
caching only seems to be used by rgw_swift.cc, not by rgw_rest_s3.cc.
That would explain why no caching is going on here.
Can anyone confirm?
And if so, is there a fundamental problem that makes it hard to use
caching when validating S3 tokens against a Keystone backend?
(Otherwise I guess I should write a feature request and/or start coding
this up myself...)
Here are the facts for background:
$ sudo ceph --admin-daemon /var/run/ceph/ceph-radosgw.gateway.asok config show | grep keystone
"rgw_keystone_url": "https:\/\/...",
"rgw_keystone_admin_token": "...",
"rgw_keystone_admin_user": "",
"rgw_keystone_admin_password": "",
"rgw_keystone_admin_tenant": "",
"rgw_keystone_accepted_roles": "_member_, ResellerAdmin",
"rgw_keystone_token_cache_size": "10000",
"rgw_keystone_revocation_interval": "900",
"rgw_s3_auth_use_keystone": "true",
$ sudo ceph --admin-daemon /var/run/ceph/ceph-radosgw.gateway.asok perf dump | grep token_cache
"keystone_token_cache_hit": 0,
"keystone_token_cache_miss": 0
When I turn on debugging (config set debug_rgw 20/20), I get many
messages like these:
2017-02-09 21:50:06.606216 7f6ac5d83700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606
2017-02-09 21:50:06.635940 7f6aac550700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:06.747616 7f6aadd53700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:06.818267 7f6ac2d7d700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606
2017-02-09 21:50:06.853492 7f6ab3d5f700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:06.895471 7f6ac5582700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606
2017-02-09 21:50:06.951734 7f6abf576700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606
2017-02-09 21:50:07.016555 7f6ab7566700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606
2017-02-09 21:50:07.038997 7f6ab355e700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:07.160196 7f6ac1d7b700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606
2017-02-09 21:50:07.189930 7f6aaf556700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:07.233593 7f6aabd4f700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607
2017-02-09 21:50:07.263116 7f6abcd71700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607
2017-02-09 21:50:07.263915 7f6ab8d69700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607
2017-02-09 21:50:07.263990 7f6aae554700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:07.280523 7f6ab2d5d700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607
2017-02-09 21:50:07.290892 7f6aa954a700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:07.311201 7f6ab6d65700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:07.317667 7f6aad552700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:07.380957 7f6ab6564700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607
2017-02-09 21:50:07.421227 7f6abd572700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607
2017-02-09 21:50:07.446867 7f6ab0d59700 20 s3 keystone: trying keystone auth
2017-02-09 21:50:07.459225 7f6aa9d4b700 20 s3 keystone: trying keystone auth
and, as I said, our Keystone service is pretty much DoSed right now...
--
Simon.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
