We're using the Hammer version of RadosGW, with Keystone for authN/Z. When a user started sending a lot of S3 requests (using rclone), the load on our Keystone service has skyrocketed. This surprised me because all those requests are from the same user, and RadosGW has caching for Keystone tokens. But looking at the code, this caching only seems to be used by rgw_swift.cc, not by rgw_rest_s3.cc. That would explain why no caching is going on here. Can anyone confirm? And if so, is there a fundamental problem that makes it hard to use caching when validating S3 tokens against a Keystone backend? (Otherwise I guess I should write a feature request and/or start coding this up myself...) Here are the facts for background: $ sudo ceph --admin-daemon /var/run/ceph/ceph-radosgw.gateway.asok config show | grep keystone "rgw_keystone_url": "https:\/\/...", "rgw_keystone_admin_token": "...", "rgw_keystone_admin_user": "", "rgw_keystone_admin_password": "", "rgw_keystone_admin_tenant": "", "rgw_keystone_accepted_roles": "_member_, ResellerAdmin", "rgw_keystone_token_cache_size": "10000", "rgw_keystone_revocation_interval": "900", "rgw_s3_auth_use_keystone": "true", $ sudo ceph --admin-daemon /var/run/ceph/ceph-radosgw.gateway.asok perf dump | grep token_cache "keystone_token_cache_hit": 0, "keystone_token_cache_miss": 0 When I turn on debugging (config set debug_rgw 20/20), I get many messages like these: 2017-02-09 21:50:06.606216 7f6ac5d83700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606 2017-02-09 21:50:06.635940 7f6aac550700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:06.747616 7f6aadd53700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:06.818267 7f6ac2d7d700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606 2017-02-09 21:50:06.853492 7f6ab3d5f700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:06.895471 7f6ac5582700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606 2017-02-09 21:50:06.951734 7f6abf576700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606 2017-02-09 21:50:07.016555 7f6ab7566700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606 2017-02-09 21:50:07.038997 7f6ab355e700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:07.160196 7f6ac1d7b700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680606 2017-02-09 21:50:07.189930 7f6aaf556700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:07.233593 7f6aabd4f700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607 2017-02-09 21:50:07.263116 7f6abcd71700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607 2017-02-09 21:50:07.263915 7f6ab8d69700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607 2017-02-09 21:50:07.263990 7f6aae554700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:07.280523 7f6ab2d5d700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607 2017-02-09 21:50:07.290892 7f6aa954a700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:07.311201 7f6ab6d65700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:07.317667 7f6aad552700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:07.380957 7f6ab6564700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607 2017-02-09 21:50:07.421227 7f6abd572700 5 s3 keystone: validated token: <BUCKET>:<ACCOUNT> expires: 1486680607 2017-02-09 21:50:07.446867 7f6ab0d59700 20 s3 keystone: trying keystone auth 2017-02-09 21:50:07.459225 7f6aa9d4b700 20 s3 keystone: trying keystone auth and, as I said, our Keystone service is pretty much DoSed right now... -- Simon. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com