Good news! I figured out what was going on. I found in my ceph.conf where
the debug logging location was:
[client.radosgw.gateway_name]
debug ms = 1
debug rgw = 20
host = gateway_name
keyring = /etc/ceph/ceph.client.radosgw.keyring
log file = /var/log/radosgw/client.radosgw.gateway_name.log
#<------ log file is here
rgw dns name = key-cloak-sc140.osnexus.net
rgw frontends = beast endpoint=10.0.26.140:7480
rgw print continue = false
rgw s3 auth use sts = true
rgw socket path =
/var/run/ceph/ceph-client.radosgw.gateway_name.asok
rgw sts key = abcdefghijklmnop
rgw zone = default
I tailed that log to see what kind of exceptions were being thrown and
found that the token verification failed because it expired. I must have
been taking too long between testing iterations after generating new access
tokens. I was able to create my-bucket on my ceph object storage after
regenerating the access token, plugging it in, and then immediately running
the sample script. Looks like my issues were self-inflicted. For those who
find this thread, either extend the expiration time for the access tokens,
plug in and run the sample script in the allotted expiration time for the
access token, or modify the sample script to load in a new access token
each time.
On Mon, Mar 21, 2022 at 1:45 PM Seth Cagampang <seth.cagampang@xxxxxxxxxxx>
wrote:
> I think that the app_id condition was a typo. After I run the python
> script to create the role I get the following role:
> {
> "Roles": [
> {
> "Path": "/",
> "RoleName": "S3Access",
> "RoleId": "2097f1fc-8a56-454c-8f00-23ded3c3c3b4",
> "Arn": "arn:aws:iam:::role/S3Access",
> "CreateDate": "2022-03-21T18:35:34.282000+00:00",
> "AssumeRolePolicyDocument": {
> "Version": "2012-10-17",
> "Statement": [
> {
> "Effect": "Allow",
> "Principal": {
> "Federated": [
>
> "arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo"
> ]
> },
> "Action": [
> "sts:AssumeRoleWithWebIdentity"
> ],
> "Condition": {
> "StringEquals": {
> "localhost:8080/auth/realms/demo:app_id":
> "account"
> }
> }
> }
> ]
> },
> "MaxSessionDuration": 3600
> }
> ]
> }
>
> I am able to verify the 'aud' and 'client_id' attribute using the keycloak
> introspection URL:
>
> {
> "exp": 1647889827,
> "iat": 1647889527,
> "jti": "5b754ce5-6601-416f-aeb5-2163bd3f8315",
> "iss": "http://localhost:8080/auth/realms/demo";,
> "aud": "account",
> "sub": "19bd5627-2952-4aca-bc67-a2724b7d61b5",
> "typ": "Bearer",
> "azp": "myclient",
> "preferred_username": "service-account-myclient",
> "email_verified": false,
> "acr": "1",
> "allowed-origins": [
> "https://10.0.26.140:7480";
> ],
> "realm_access": {
> "roles": [
> "offline_access",
> "default-roles-demo",
> "uma_authorization"
> ]
> },
> "resource_access": {
> "myclient": {
> "roles": [
> "uma_protection"
> ]
> },
> "account": {
> "roles": [
> "manage-account",
> "manage-account-links",
> "view-profile"
> ]
> }
> },
> "scope": "openid email profile",
> "clientId": "myclient",
> "clientHost": "10.0.26.140",
> "clientAddress": "10.0.26.140",
> "client_id": "myclient",
> "username": "service-account-myclient",
> "active": true
> }
>
> VIA simones suggestion it looks like my 'rgw sts key = abcdefghijklmnop'
> and 'rgw s3 auth use sts = true' are not being applied. I added the debug
> options and sts options to the /etc/ceph/ceph.conf file and verified that
> all nodes in the cluster have the settings applied. Then, I restarted the
> 'radosgw' service using 'systemctl restart radosgw.service'. Finally, I
> check the rgw config using 'radosgw-admin --show-config':
>
> root@terminal:~# radosgw-admin --show-config | grep -i sts
> mds_forward_all_requests_to_auth = false
> mds_max_completed_requests = 100000
> rbd_readahead_trigger_requests = 10
> rgw_enable_apis = s3, s3website, swift, swift_auth, admin, sts, iam,
> notifications
> rgw_max_concurrent_requests = 1024
> rgw_s3_auth_order = sts, external, local
> rgw_s3_auth_use_sts = false
> rgw_sts_client_id =
> rgw_sts_client_secret =
> rgw_sts_entry = sts
> rgw_sts_key = sts
> rgw_sts_max_session_duration = 43200
> rgw_sts_min_session_duration = 900
> rgw_sts_token_introspection_url =
>
> root@terminal:~# radosgw-admin --show-config | grep -i debug_ms
> debug_ms = 0/0
>
> root@terminal:~# radosgw-admin --show-config | grep -i debug_rgw
> debug_rgw = 1/5
>
> As you can see it looks like the settings in the config file did not get
> applied from the perspective of the radosgw-admin CLI tool. Am I doing
> something wrong to apply these settings? It seems I won't be able to get
> the debug logs until I can apply some of these settings. After running the
> example boto3 script, I am not seeing any sort of rgw logs in
> '/var/log/ceph/' :
>
> root@terminal# ls -la /var/log/ceph/
> total 19304
> drwxrws--T 2 ceph ceph 4096 Mar 21 12:58 .
> drwxrwxr-x 19 root syslog 4096 Mar 21 13:02 ..
> -rw------- 1 ceph ceph 4071721 Mar 21 13:34 ceph.audit.log
> -rw------- 1 ceph ceph 776987 Mar 21 13:34 ceph.log
> -rw-r--r-- 1 ceph ceph 1823776 Mar 21 13:28 ceph-mgr.key-cloak-sc140.log
> -rw-r--r-- 1 ceph ceph 3484236 Mar 21 13:34 ceph-mon.key-cloak-sc140.log
> -rw-r--r-- 1 ceph ceph 2359942 Mar 21 13:28 ceph-osd.0.log
> -rw-r--r-- 1 ceph ceph 2306777 Mar 21 13:28 ceph-osd.1.log
> -rw-r--r-- 1 ceph ceph 2312102 Mar 21 13:28 ceph-osd.2.log
> -rw-r--r-- 1 ceph ceph 2365239 Mar 21 13:28 ceph-osd.3.log
> -rw-rw-rw- 1 root ceph 184626 Mar 21 12:58 ceph-volume.log
> -rw-r--r-- 1 root ceph 11306 Mar 21 12:58 ceph-volume-systemd.log
>
> I was trying to tail the log files while running the POC script, but I did
> not notice any clear error messages related to the
> AssumeRoleWithWebIdentity call. Does this mean that my radosgw is not set
> up properly? I used this guide <
> https://access.redhat.com/solutions/2085183#:~:text=The%20logs%20will%20be%20inside,on%20the%20Rados%20Gateway%20node.>
> to try to set up the debug logging:
>
> ># sudo systemctl list-units | grep -i rgw
> ># sudo systemctl restart ceph-radosgw@<$service_name>.service
>
> I am finding that there is no service ceph-radosgw@<$service_name>.service
> for my radosgw and the 'list-units + grep' command returns empty. This
> makes me think that maybe the rgw service is not set up properly.
>
> Please advise,
>
> Seth
>
>
> On Mon, Mar 21, 2022 at 12:53 AM <simone.beccato@xxxxxxxxxxxxxx> wrote:
>
>> Hi,
>>
>> I'm working on it too, use this settings into ceph.conf to setup debug
>> logs
>> and check what you get:
>>
>> [client.radosgw.gateway_name]
>> debug ms = 1
>> debug rgw = 20
>>
>>
>> Check also if settings are applied correctly from config file into RGW
>> with
>> this command:
>>
>> radosgw-admin --show-config
>>
>> As suggested also by Pritha, check if in the field "aud" of you token are
>> present the value "account".
>>
>> Best
>> Simone
>>
>>
>>
>> -----Messaggio originale-----
>> Da: Pritha Srivastava <prsrivas@xxxxxxxxxx>
>> Inviato: sabato 19 marzo 2022 05:24
>> A: Seth Cagampang <seth.cagampang@xxxxxxxxxxx>
>> Cc: ceph-users <ceph-users@xxxxxxx>
>> Oggetto: Re: Ceph RADOSGW with Keycloak ODIC
>>
>> Hi,
>>
>> When you list the roles, the Condition element of the trust policy in the
>> role doesn't seem quite right:
>>
>> "Condition": {
>> > "StringEquals": {
>> >
>> > "localhost:8080/auth/realms/demo:myclient
>> <http://10.0.26.1:8080/auth/realms/demo:myclient>": "account"
>> > }
>>
>> But what you have mentioned in the policy_document just above is correct:
>>
>>
>> "Condition":{"StringEquals":{"localhost:8080/auth/realms/demo:app_id":"accou
>> nt"}}
>>
>> Is the value of 'aud' field in the access token that you generated, set to
>> "account"?
>>
>> Another thing to check would be to see that the clientid (myclient) that
>> you
>> have set in clientIdList as part of create_openid_connect_provider() call,
>> matches with the value of either clientId or client_id field in the access
>> token.
>>
>> Or you can also check rgw logs and see what error is being logged for
>> AssumeRoleWithWebIdentity.
>>
>> Thanks,
>> Pritha
>>
>> On Sat, Mar 19, 2022 at 12:21 AM Seth Cagampang <
>> seth.cagampang@xxxxxxxxxxx>
>> wrote:
>>
>> > Hello,
>> >
>> >
>> >
>> > It seems like Pritha is the Ceph RGW expert in this forum. I am
>> > currently trying to integrate CephRGW object storage with KeyCloak as
>> > the OIDC provider. I am running ceph version 16.2.7 Pacific stable.
>> >
>> >
>> >
>> > At this point, I am just trying to get a POC working with the python
>> > scripts provided in the example in these docs <
>> > https://docs.ceph.com/en/latest/radosgw/STS/#sts-configuration> . Here
>> > are some step by step instructions on how I set up the ceph cluster
>> > and KeyCloak server:
>> >
>> >
>> >
>> > *Set up keycloak server*:
>> >
>> > 1. Create new Realm 'demo'
>> >
>> > 2. Create 'testuser' and add credentials. Verify that I am able to
>> > login to the realm using the new credentials.
>> >
>> > 3. Create a client 'myclient' and set Access Type as 'confidential' to
>> > generate client secret
>> >
>> > 4. Add a keycloak-oidc provider using the client credentials.
>> >
>> > 5. On the client set 'Authorization Enabled' to ON and 'Service
>> > Accounts Enabled' to ON.
>> >
>> >
>> >
>> > We should now be able to get the access tokens from the OIDC provider.
>> > To do this I used the sample curl calls from these docs <
>> > https://docs.ceph.com/en/latest/radosgw/keycloak/#setting-up-keycloak>
>> > which I put into scripts:
>> >
>> > access_token.sh
>> >
>> > #!/bin/bash
>> >
>> > KC_REALM=demo
>> >
>> > KC_CLIENT=myclient
>> >
>> > KC_CLIENT_SECRET=620b31fa-****-****-****-************
>> >
>> > KC_SERVER=localhost:8080 <http://10.0.26.1:8080/>
>> >
>> > KC_CONTEXT=auth
>> >
>> >
>> >
>> > # Request Tokens for credentials
>> >
>> > KC_RESPONSE=$( \
>> >
>> > curl -k -v -X POST \
>> >
>> > -H "Content-Type: application/x-www-form-urlencoded" \
>> >
>> > -d "scope=openid" \
>> >
>> > -d "grant_type=client_credentials" \
>> >
>> > -d "client_id=$KC_CLIENT" \
>> >
>> > -d "client_secret=$KC_CLIENT_SECRET" \
>> >
>> > "http://
>> > $KC_SERVER/$KC_CONTEXT/realms/$KC_REALM/protocol/openid-connect/token"
>> > \
>> >
>> > | jq .
>> >
>> > )
>> >
>> >
>> >
>> > KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token)
>> >
>> > echo $KC_RESPONSE | jq .
>> >
>> > echo $KC_ACCESS_TOKEN
>> >
>> >
>> >
>> > Using this script I am able to get the access token for later usage
>> > and it has been verified that we are able to get the access token from
>> > the key cloak OIDC.
>> >
>> >
>> >
>> > *Set up Ceph Cluster w/ RGW*:
>> >
>> > 1. Create Ceph Cluster with OSD's and journals. Create an S3 object
>> > storage pool and then create an RGW on the cluster manager node.
>> >
>> > 2. Enable sts in the gateway config in /etc/ceph/ceph.conf as seen in
>> > the example from the docs <
>> > https://docs.ceph.com/en/latest/radosgw/keycloak/#setting-up-keycloak>
>> :
>> >
>> > > [client.radosgw.gateway_name]
>> >
>> > > rgw sts key = abcdefghijklmnop
>> >
>> > > rgw s3 auth use sts = true
>> >
>> > 3. Create test users to be used in the test application python script.
>> >
>> > > radosgw-admin --uid TESTER --display-name "testuser" --access_key
>> > > TESTER
>> > --secret test123 user create
>> > > radosgw-admin caps add --uid="TESTER" --caps="oidc-provider=*"
>> > > radosgw-admin caps add --uid="TESTER" --caps="roles=*"
>> > >
>> > > radosgw-admin --uid TESTER1 --display-name "testuser1"
>> > > --access_key
>> > TESTER1 --secret test321 user create
>> > > radosgw-admin caps add --uid="TESTER1" --caps="roles=*"
>> >
>> > 4. We need to generate thumbprints of the OIDC provider. I used the
>> > docs here
>> > <https://docs.ceph.com/en/latest/radosgw/STS/#sts-configuration> to
>> write
>> a script to generate the thumbprints:
>> >
>> > # Get the 'x5c' from this response to turn into an IDP-cert
>> >
>> > KEY1_RESPONSE=$(curl -k -v \
>> >
>> > -X GET \
>> >
>> > -H "Content-Type: application/x-www-form-urlencoded" \
>> >
>> >
>> > "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs
>> > "
>> > \
>> >
>> > | jq -r .keys[0].x5c)
>> >
>> >
>> >
>> > KEY2_RESPONSE=$(curl -k -v \
>> >
>> > -X GET \
>> >
>> > -H "Content-Type: application/x-www-form-urlencoded" \
>> >
>> >
>> > "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs
>> > "
>> > \
>> >
>> > | jq -r .keys[1].x5c)
>> >
>> >
>> >
>> > echo
>> >
>> > echo "Assembling Certificates...."
>> >
>> >
>> >
>> > # Assemble Cert1
>> >
>> > echo '-----BEGIN CERTIFICATE-----' > certificate1.crt
>> >
>> > echo $(echo $KEY1_RESPONSE) | sed
>> > 's/^.//;s/.$//;s/^.//;s/.$//;s/^.//;s/.$//' >> certificate1.crt
>> >
>> > echo '-----END CERTIFICATE-----' >> certificate1.crt
>> >
>> > echo $(cat certificate1.crt)
>> >
>> >
>> >
>> > # Assemble Cert2
>> >
>> > echo '-----BEGIN CERTIFICATE-----' > certificate2.crt
>> >
>> > echo $(echo $KEY2_RESPONSE) | sed
>> > 's/^.//;s/.$//;s/^.//;s/.$//;s/^.//;s/.$//' >> certificate2.crt
>> >
>> > echo '-----END CERTIFICATE-----' >> certificate2.crt
>> >
>> > echo $(cat certificate2.crt)
>> >
>> >
>> >
>> > echo
>> >
>> > echo "Generating thumbprints...."
>> >
>> > # Create Thumbprint for both certs
>> >
>> > PRETHUMBPRINT1=$(openssl x509 -in certificate1.crt -fingerprint
>> > -noout)
>> >
>> > PRETHUMBPRINT2=$(openssl x509 -in certificate2.crt -fingerprint
>> > -noout)
>> >
>> >
>> >
>> > PRETHUMBPRINT1=$(echo $PRETHUMBPRINT1 | awk '{ print substr($0, 18)
>> > }')
>> >
>> > PRETHUMBPRINT2=$(echo $PRETHUMBPRINT2 | awk '{ print substr($0, 18)
>> > }')
>> >
>> >
>> >
>> > echo "${PRETHUMBPRINT1//:}"
>> >
>> > echo "${PRETHUMBPRINT2//:}"
>> >
>> > I copied and pasted these thumbprints into the example application
>> > python script to perform the create_open_id_connect_provider() for the
>> > 'iam_client'.
>> >
>> > 5. Next I filled out the missing information in the example
>> > application
>> > script:
>> >
>> > #!/usr/bin/python3
>> >
>> > import boto3
>> >
>> >
>> >
>> > iam_client = boto3.client('iam',
>> >
>> > aws_access_key_id="TESTER",
>> >
>> > aws_secret_access_key="test123",
>> >
>> > endpoint_url="http://10.x.x.x:7480";, #<----Ceph RGW endpoint -
>> > using http for proof of concept
>> >
>> > region_name=''
>> >
>> > )
>> >
>> >
>> >
>> > oidc_response = iam_client.create_open_id_connect_provider(
>> >
>> > Url="http://localhost:8080/auth/realms/demo";,
>> >
>> > ClientIDList=[
>> >
>> > "myclient"
>> >
>> > ],
>> >
>> > ThumbprintList=[
>> >
>> > "E43DBA95FC202A9773F3F542F12CF2A831FC7A6F",
>> >
>> > "CE0E06206F6F5670E2BC725BD1557D177B3708BF"
>> >
>> > ]
>> >
>> > )
>> >
>> >
>> >
>> > policy_document =
>> >
>> >
>>
>> '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Fede
>>
>> rated":["arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo"]},"Act
>>
>> ion":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"localho
>> st:8080/auth/realms/demo:app_id":"account"}}}]}'''
>> >
>> > role_response = iam_client.create_role(
>> >
>> > AssumeRolePolicyDocument=policy_document,
>> >
>> > Path='/',
>> >
>> > RoleName='S3Access',
>> >
>> > )
>> >
>> >
>> >
>> > role_policy =
>> >
>> >
>>
>> '''{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Re
>> source":"arn:aws:s3:::*"}}'''
>> >
>> > response = iam_client.put_role_policy(
>> >
>> > RoleName='S3Access',
>> >
>> > PolicyName='Policy1',
>> >
>> > PolicyDocument=role_policy
>> >
>> > )
>> >
>> > #TESTER1
>> >
>> > sts_client = boto3.client('sts',
>> >
>> > aws_access_key_id="TESTER1",
>> >
>> > aws_secret_access_key="test321",
>> >
>> > endpoint_url="http://10.x.x.x:7480";,
>> >
>> > region_name='',
>> >
>> > )
>> >
>> >
>> >
>> > response = sts_client.assume_role_with_web_identity(
>> >
>> > RoleArn=get_response['Role']['Arn'],
>> >
>> > RoleSessionName='Bob',
>> >
>> > DurationSeconds=3600,
>> >
>> > WebIdentityToken="inserted-access-token" #<---- access token from
>> > step
>> > 5 in keycloak setup
>> >
>> > )
>> >
>> >
>> >
>> > s3client = boto3.client('s3',
>> >
>> > aws_access_key_id = response['Credentials']['AccessKeyId'],
>> >
>> > aws_secret_access_key =
>> > response['Credentials']['SecretAccessKey'],
>> >
>> > aws_session_token = response['Credentials']['SessionToken'],
>> >
>> > endpoint_url="http://10.x.x.x:7480";,
>> >
>> > region_name='',
>> >
>> > )
>> >
>> >
>> >
>> > bucket_name = 'my-bucket'
>> >
>> > s3bucket = s3client.create_bucket(Bucket=bucket_name)
>> >
>> > resp = s3client.list_buckets()
>> >
>> >
>> >
>> > print(resp)
>> >
>> >
>> >
>> > Currently, This script cannot run to completion. I find that it throws
>> > a fatal error when trying to run
>> 'sts_client.assume_role_with_web_identity()'
>> > with the exception: "An error occurred (Unknown) when calling the
>> > AssumeRoleWithWebIdentity operation: Unknown" .
>> >
>> >
>> >
>> > I am able to verify that the OIDC provider and role has been created
>> > on the
>> > RGW: >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam
>> > list-open-id-connect-providers --region=""
>> > >{
>> > > "OpenIDConnectProviderList": [
>> > > {
>> > > "Arn":
>> > "arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo"
>> > > }
>> > > ]
>> > >}
>> > >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam
>> > get-open-id-connect-provider
>> >
>> >
>>
>> --open-id-connect-provider-arn="arn:aws:iam:::oidc-provider/localhost:8080/a
>> uth/realms/demo"
>> > --region=""
>> > >{
>> > > "Url": "http://localhost:8080/auth/realms/demo";,
>> > > "ClientIDList": [
>> > > "myclient"
>> > > ],
>> > > "ThumbprintList": [
>> > > "E43DBA95FC202A9773F3F542F12CF2A831FC7A6F",
>> > > "CE0E06206F6F5670E2BC725BD1557D177B3708BF"
>> > > ],
>> > > "CreateDate": "2022-03-15T17:40:17.572000+00:00"
>> > >}
>> >
>> > >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam list-roles
>> > --region=""
>> >
>> > >{
>> > > "Roles": [
>> > > {
>> > > "Path": "/",
>> > > "RoleName": "S3Access",
>> > > "RoleId": "7329b54e-40a2-4476-ae68-52a72b43376c",
>> > > "Arn": "arn:aws:iam:::role/S3Access",
>> > > "CreateDate": "2022-03-14T21:30:11.750000+00:00",
>> > > "AssumeRolePolicyDocument": {
>> > > "Version": "2012-10-17",
>> > > "Statement": [
>> > > {
>> > > "Effect": "Allow",
>> > > "Principal": {
>> > > "Federated": [
>> > >
>> > >"arn:aws:iam:::oidc-provider/localhost
>> > :8080/auth/realms/demo <http://10.0.26.1:8080/auth/realms/demo>"
>> > > ]
>> > > },
>> > > "Action": [
>> > > "sts:AssumeRoleWithWebIdentity"
>> > > ],
>> > > "Condition": {
>> > > "StringEquals": {
>> > >
>> > > "localhost:8080/auth/realms/demo:myclient
>> > <http://10.0.26.1:8080/auth/realms/demo:myclient>": "account"
>> > > }
>> > > }
>> > > }
>> > > ]
>> > > },
>> > > "MaxSessionDuration": 3600
>> > > }
>> > > ]
>> > >}
>> >
>> >
>> >
>> > I must be missing something here. Any advice you might have for me
>> > would be greatly appreciated.
>> >
>> >
>> >
>> > Thank you
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
>> > email to ceph-users-leave@xxxxxxx
>> >
>> >
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
>> email
>> to ceph-users-leave@xxxxxxx
>>
>>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
