I think that the app_id condition was a typo. After I run the python script
to create the role I get the following role:
{
"Roles": [
{
"Path": "/",
"RoleName": "S3Access",
"RoleId": "2097f1fc-8a56-454c-8f00-23ded3c3c3b4",
"Arn": "arn:aws:iam:::role/S3Access",
"CreateDate": "2022-03-21T18:35:34.282000+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": [
"arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo"
]
},
"Action": [
"sts:AssumeRoleWithWebIdentity"
],
"Condition": {
"StringEquals": {
"localhost:8080/auth/realms/demo:app_id":
"account"
}
}
}
]
},
"MaxSessionDuration": 3600
}
]
}
I am able to verify the 'aud' and 'client_id' attribute using the keycloak
introspection URL:
{
"exp": 1647889827,
"iat": 1647889527,
"jti": "5b754ce5-6601-416f-aeb5-2163bd3f8315",
"iss": "http://localhost:8080/auth/realms/demo";,
"aud": "account",
"sub": "19bd5627-2952-4aca-bc67-a2724b7d61b5",
"typ": "Bearer",
"azp": "myclient",
"preferred_username": "service-account-myclient",
"email_verified": false,
"acr": "1",
"allowed-origins": [
"https://10.0.26.140:7480";
],
"realm_access": {
"roles": [
"offline_access",
"default-roles-demo",
"uma_authorization"
]
},
"resource_access": {
"myclient": {
"roles": [
"uma_protection"
]
},
"account": {
"roles": [
"manage-account",
"manage-account-links",
"view-profile"
]
}
},
"scope": "openid email profile",
"clientId": "myclient",
"clientHost": "10.0.26.140",
"clientAddress": "10.0.26.140",
"client_id": "myclient",
"username": "service-account-myclient",
"active": true
}
VIA simones suggestion it looks like my 'rgw sts key = abcdefghijklmnop'
and 'rgw s3 auth use sts = true' are not being applied. I added the debug
options and sts options to the /etc/ceph/ceph.conf file and verified that
all nodes in the cluster have the settings applied. Then, I restarted the
'radosgw' service using 'systemctl restart radosgw.service'. Finally, I
check the rgw config using 'radosgw-admin --show-config':
root@terminal:~# radosgw-admin --show-config | grep -i sts
mds_forward_all_requests_to_auth = false
mds_max_completed_requests = 100000
rbd_readahead_trigger_requests = 10
rgw_enable_apis = s3, s3website, swift, swift_auth, admin, sts, iam,
notifications
rgw_max_concurrent_requests = 1024
rgw_s3_auth_order = sts, external, local
rgw_s3_auth_use_sts = false
rgw_sts_client_id =
rgw_sts_client_secret =
rgw_sts_entry = sts
rgw_sts_key = sts
rgw_sts_max_session_duration = 43200
rgw_sts_min_session_duration = 900
rgw_sts_token_introspection_url =
root@terminal:~# radosgw-admin --show-config | grep -i debug_ms
debug_ms = 0/0
root@terminal:~# radosgw-admin --show-config | grep -i debug_rgw
debug_rgw = 1/5
As you can see it looks like the settings in the config file did not get
applied from the perspective of the radosgw-admin CLI tool. Am I doing
something wrong to apply these settings? It seems I won't be able to get
the debug logs until I can apply some of these settings. After running the
example boto3 script, I am not seeing any sort of rgw logs in
'/var/log/ceph/' :
root@terminal# ls -la /var/log/ceph/
total 19304
drwxrws--T 2 ceph ceph 4096 Mar 21 12:58 .
drwxrwxr-x 19 root syslog 4096 Mar 21 13:02 ..
-rw------- 1 ceph ceph 4071721 Mar 21 13:34 ceph.audit.log
-rw------- 1 ceph ceph 776987 Mar 21 13:34 ceph.log
-rw-r--r-- 1 ceph ceph 1823776 Mar 21 13:28 ceph-mgr.key-cloak-sc140.log
-rw-r--r-- 1 ceph ceph 3484236 Mar 21 13:34 ceph-mon.key-cloak-sc140.log
-rw-r--r-- 1 ceph ceph 2359942 Mar 21 13:28 ceph-osd.0.log
-rw-r--r-- 1 ceph ceph 2306777 Mar 21 13:28 ceph-osd.1.log
-rw-r--r-- 1 ceph ceph 2312102 Mar 21 13:28 ceph-osd.2.log
-rw-r--r-- 1 ceph ceph 2365239 Mar 21 13:28 ceph-osd.3.log
-rw-rw-rw- 1 root ceph 184626 Mar 21 12:58 ceph-volume.log
-rw-r--r-- 1 root ceph 11306 Mar 21 12:58 ceph-volume-systemd.log
I was trying to tail the log files while running the POC script, but I did
not notice any clear error messages related to the
AssumeRoleWithWebIdentity call. Does this mean that my radosgw is not set
up properly? I used this guide <
https://access.redhat.com/solutions/2085183#:~:text=The%20logs%20will%20be%20inside,on%20the%20Rados%20Gateway%20node.>
to try to set up the debug logging:
># sudo systemctl list-units | grep -i rgw
># sudo systemctl restart ceph-radosgw@<$service_name>.service
I am finding that there is no service ceph-radosgw@<$service_name>.service
for my radosgw and the 'list-units + grep' command returns empty. This
makes me think that maybe the rgw service is not set up properly.
Please advise,
Seth
On Mon, Mar 21, 2022 at 12:53 AM <simone.beccato@xxxxxxxxxxxxxx> wrote:
> Hi,
>
> I'm working on it too, use this settings into ceph.conf to setup debug logs
> and check what you get:
>
> [client.radosgw.gateway_name]
> debug ms = 1
> debug rgw = 20
>
>
> Check also if settings are applied correctly from config file into RGW with
> this command:
>
> radosgw-admin --show-config
>
> As suggested also by Pritha, check if in the field "aud" of you token are
> present the value "account".
>
> Best
> Simone
>
>
>
> -----Messaggio originale-----
> Da: Pritha Srivastava <prsrivas@xxxxxxxxxx>
> Inviato: sabato 19 marzo 2022 05:24
> A: Seth Cagampang <seth.cagampang@xxxxxxxxxxx>
> Cc: ceph-users <ceph-users@xxxxxxx>
> Oggetto: Re: Ceph RADOSGW with Keycloak ODIC
>
> Hi,
>
> When you list the roles, the Condition element of the trust policy in the
> role doesn't seem quite right:
>
> "Condition": {
> > "StringEquals": {
> >
> > "localhost:8080/auth/realms/demo:myclient
> <http://10.0.26.1:8080/auth/realms/demo:myclient>": "account"
> > }
>
> But what you have mentioned in the policy_document just above is correct:
>
>
> "Condition":{"StringEquals":{"localhost:8080/auth/realms/demo:app_id":"accou
> nt"}}
>
> Is the value of 'aud' field in the access token that you generated, set to
> "account"?
>
> Another thing to check would be to see that the clientid (myclient) that
> you
> have set in clientIdList as part of create_openid_connect_provider() call,
> matches with the value of either clientId or client_id field in the access
> token.
>
> Or you can also check rgw logs and see what error is being logged for
> AssumeRoleWithWebIdentity.
>
> Thanks,
> Pritha
>
> On Sat, Mar 19, 2022 at 12:21 AM Seth Cagampang <
> seth.cagampang@xxxxxxxxxxx>
> wrote:
>
> > Hello,
> >
> >
> >
> > It seems like Pritha is the Ceph RGW expert in this forum. I am
> > currently trying to integrate CephRGW object storage with KeyCloak as
> > the OIDC provider. I am running ceph version 16.2.7 Pacific stable.
> >
> >
> >
> > At this point, I am just trying to get a POC working with the python
> > scripts provided in the example in these docs <
> > https://docs.ceph.com/en/latest/radosgw/STS/#sts-configuration> . Here
> > are some step by step instructions on how I set up the ceph cluster
> > and KeyCloak server:
> >
> >
> >
> > *Set up keycloak server*:
> >
> > 1. Create new Realm 'demo'
> >
> > 2. Create 'testuser' and add credentials. Verify that I am able to
> > login to the realm using the new credentials.
> >
> > 3. Create a client 'myclient' and set Access Type as 'confidential' to
> > generate client secret
> >
> > 4. Add a keycloak-oidc provider using the client credentials.
> >
> > 5. On the client set 'Authorization Enabled' to ON and 'Service
> > Accounts Enabled' to ON.
> >
> >
> >
> > We should now be able to get the access tokens from the OIDC provider.
> > To do this I used the sample curl calls from these docs <
> > https://docs.ceph.com/en/latest/radosgw/keycloak/#setting-up-keycloak>
> > which I put into scripts:
> >
> > access_token.sh
> >
> > #!/bin/bash
> >
> > KC_REALM=demo
> >
> > KC_CLIENT=myclient
> >
> > KC_CLIENT_SECRET=620b31fa-****-****-****-************
> >
> > KC_SERVER=localhost:8080 <http://10.0.26.1:8080/>
> >
> > KC_CONTEXT=auth
> >
> >
> >
> > # Request Tokens for credentials
> >
> > KC_RESPONSE=$( \
> >
> > curl -k -v -X POST \
> >
> > -H "Content-Type: application/x-www-form-urlencoded" \
> >
> > -d "scope=openid" \
> >
> > -d "grant_type=client_credentials" \
> >
> > -d "client_id=$KC_CLIENT" \
> >
> > -d "client_secret=$KC_CLIENT_SECRET" \
> >
> > "http://
> > $KC_SERVER/$KC_CONTEXT/realms/$KC_REALM/protocol/openid-connect/token"
> > \
> >
> > | jq .
> >
> > )
> >
> >
> >
> > KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token)
> >
> > echo $KC_RESPONSE | jq .
> >
> > echo $KC_ACCESS_TOKEN
> >
> >
> >
> > Using this script I am able to get the access token for later usage
> > and it has been verified that we are able to get the access token from
> > the key cloak OIDC.
> >
> >
> >
> > *Set up Ceph Cluster w/ RGW*:
> >
> > 1. Create Ceph Cluster with OSD's and journals. Create an S3 object
> > storage pool and then create an RGW on the cluster manager node.
> >
> > 2. Enable sts in the gateway config in /etc/ceph/ceph.conf as seen in
> > the example from the docs <
> > https://docs.ceph.com/en/latest/radosgw/keycloak/#setting-up-keycloak> :
> >
> > > [client.radosgw.gateway_name]
> >
> > > rgw sts key = abcdefghijklmnop
> >
> > > rgw s3 auth use sts = true
> >
> > 3. Create test users to be used in the test application python script.
> >
> > > radosgw-admin --uid TESTER --display-name "testuser" --access_key
> > > TESTER
> > --secret test123 user create
> > > radosgw-admin caps add --uid="TESTER" --caps="oidc-provider=*"
> > > radosgw-admin caps add --uid="TESTER" --caps="roles=*"
> > >
> > > radosgw-admin --uid TESTER1 --display-name "testuser1"
> > > --access_key
> > TESTER1 --secret test321 user create
> > > radosgw-admin caps add --uid="TESTER1" --caps="roles=*"
> >
> > 4. We need to generate thumbprints of the OIDC provider. I used the
> > docs here
> > <https://docs.ceph.com/en/latest/radosgw/STS/#sts-configuration> to
> write
> a script to generate the thumbprints:
> >
> > # Get the 'x5c' from this response to turn into an IDP-cert
> >
> > KEY1_RESPONSE=$(curl -k -v \
> >
> > -X GET \
> >
> > -H "Content-Type: application/x-www-form-urlencoded" \
> >
> >
> > "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs
> > "
> > \
> >
> > | jq -r .keys[0].x5c)
> >
> >
> >
> > KEY2_RESPONSE=$(curl -k -v \
> >
> > -X GET \
> >
> > -H "Content-Type: application/x-www-form-urlencoded" \
> >
> >
> > "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs
> > "
> > \
> >
> > | jq -r .keys[1].x5c)
> >
> >
> >
> > echo
> >
> > echo "Assembling Certificates...."
> >
> >
> >
> > # Assemble Cert1
> >
> > echo '-----BEGIN CERTIFICATE-----' > certificate1.crt
> >
> > echo $(echo $KEY1_RESPONSE) | sed
> > 's/^.//;s/.$//;s/^.//;s/.$//;s/^.//;s/.$//' >> certificate1.crt
> >
> > echo '-----END CERTIFICATE-----' >> certificate1.crt
> >
> > echo $(cat certificate1.crt)
> >
> >
> >
> > # Assemble Cert2
> >
> > echo '-----BEGIN CERTIFICATE-----' > certificate2.crt
> >
> > echo $(echo $KEY2_RESPONSE) | sed
> > 's/^.//;s/.$//;s/^.//;s/.$//;s/^.//;s/.$//' >> certificate2.crt
> >
> > echo '-----END CERTIFICATE-----' >> certificate2.crt
> >
> > echo $(cat certificate2.crt)
> >
> >
> >
> > echo
> >
> > echo "Generating thumbprints...."
> >
> > # Create Thumbprint for both certs
> >
> > PRETHUMBPRINT1=$(openssl x509 -in certificate1.crt -fingerprint
> > -noout)
> >
> > PRETHUMBPRINT2=$(openssl x509 -in certificate2.crt -fingerprint
> > -noout)
> >
> >
> >
> > PRETHUMBPRINT1=$(echo $PRETHUMBPRINT1 | awk '{ print substr($0, 18)
> > }')
> >
> > PRETHUMBPRINT2=$(echo $PRETHUMBPRINT2 | awk '{ print substr($0, 18)
> > }')
> >
> >
> >
> > echo "${PRETHUMBPRINT1//:}"
> >
> > echo "${PRETHUMBPRINT2//:}"
> >
> > I copied and pasted these thumbprints into the example application
> > python script to perform the create_open_id_connect_provider() for the
> > 'iam_client'.
> >
> > 5. Next I filled out the missing information in the example
> > application
> > script:
> >
> > #!/usr/bin/python3
> >
> > import boto3
> >
> >
> >
> > iam_client = boto3.client('iam',
> >
> > aws_access_key_id="TESTER",
> >
> > aws_secret_access_key="test123",
> >
> > endpoint_url="http://10.x.x.x:7480";, #<----Ceph RGW endpoint -
> > using http for proof of concept
> >
> > region_name=''
> >
> > )
> >
> >
> >
> > oidc_response = iam_client.create_open_id_connect_provider(
> >
> > Url="http://localhost:8080/auth/realms/demo";,
> >
> > ClientIDList=[
> >
> > "myclient"
> >
> > ],
> >
> > ThumbprintList=[
> >
> > "E43DBA95FC202A9773F3F542F12CF2A831FC7A6F",
> >
> > "CE0E06206F6F5670E2BC725BD1557D177B3708BF"
> >
> > ]
> >
> > )
> >
> >
> >
> > policy_document =
> >
> >
>
> '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Fede
>
> rated":["arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo"]},"Act
>
> ion":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"localho
> st:8080/auth/realms/demo:app_id":"account"}}}]}'''
> >
> > role_response = iam_client.create_role(
> >
> > AssumeRolePolicyDocument=policy_document,
> >
> > Path='/',
> >
> > RoleName='S3Access',
> >
> > )
> >
> >
> >
> > role_policy =
> >
> >
>
> '''{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Re
> source":"arn:aws:s3:::*"}}'''
> >
> > response = iam_client.put_role_policy(
> >
> > RoleName='S3Access',
> >
> > PolicyName='Policy1',
> >
> > PolicyDocument=role_policy
> >
> > )
> >
> > #TESTER1
> >
> > sts_client = boto3.client('sts',
> >
> > aws_access_key_id="TESTER1",
> >
> > aws_secret_access_key="test321",
> >
> > endpoint_url="http://10.x.x.x:7480";,
> >
> > region_name='',
> >
> > )
> >
> >
> >
> > response = sts_client.assume_role_with_web_identity(
> >
> > RoleArn=get_response['Role']['Arn'],
> >
> > RoleSessionName='Bob',
> >
> > DurationSeconds=3600,
> >
> > WebIdentityToken="inserted-access-token" #<---- access token from
> > step
> > 5 in keycloak setup
> >
> > )
> >
> >
> >
> > s3client = boto3.client('s3',
> >
> > aws_access_key_id = response['Credentials']['AccessKeyId'],
> >
> > aws_secret_access_key =
> > response['Credentials']['SecretAccessKey'],
> >
> > aws_session_token = response['Credentials']['SessionToken'],
> >
> > endpoint_url="http://10.x.x.x:7480";,
> >
> > region_name='',
> >
> > )
> >
> >
> >
> > bucket_name = 'my-bucket'
> >
> > s3bucket = s3client.create_bucket(Bucket=bucket_name)
> >
> > resp = s3client.list_buckets()
> >
> >
> >
> > print(resp)
> >
> >
> >
> > Currently, This script cannot run to completion. I find that it throws
> > a fatal error when trying to run
> 'sts_client.assume_role_with_web_identity()'
> > with the exception: "An error occurred (Unknown) when calling the
> > AssumeRoleWithWebIdentity operation: Unknown" .
> >
> >
> >
> > I am able to verify that the OIDC provider and role has been created
> > on the
> > RGW: >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam
> > list-open-id-connect-providers --region=""
> > >{
> > > "OpenIDConnectProviderList": [
> > > {
> > > "Arn":
> > "arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo"
> > > }
> > > ]
> > >}
> > >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam
> > get-open-id-connect-provider
> >
> >
>
> --open-id-connect-provider-arn="arn:aws:iam:::oidc-provider/localhost:8080/a
> uth/realms/demo"
> > --region=""
> > >{
> > > "Url": "http://localhost:8080/auth/realms/demo";,
> > > "ClientIDList": [
> > > "myclient"
> > > ],
> > > "ThumbprintList": [
> > > "E43DBA95FC202A9773F3F542F12CF2A831FC7A6F",
> > > "CE0E06206F6F5670E2BC725BD1557D177B3708BF"
> > > ],
> > > "CreateDate": "2022-03-15T17:40:17.572000+00:00"
> > >}
> >
> > >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam list-roles
> > --region=""
> >
> > >{
> > > "Roles": [
> > > {
> > > "Path": "/",
> > > "RoleName": "S3Access",
> > > "RoleId": "7329b54e-40a2-4476-ae68-52a72b43376c",
> > > "Arn": "arn:aws:iam:::role/S3Access",
> > > "CreateDate": "2022-03-14T21:30:11.750000+00:00",
> > > "AssumeRolePolicyDocument": {
> > > "Version": "2012-10-17",
> > > "Statement": [
> > > {
> > > "Effect": "Allow",
> > > "Principal": {
> > > "Federated": [
> > >
> > >"arn:aws:iam:::oidc-provider/localhost
> > :8080/auth/realms/demo <http://10.0.26.1:8080/auth/realms/demo>"
> > > ]
> > > },
> > > "Action": [
> > > "sts:AssumeRoleWithWebIdentity"
> > > ],
> > > "Condition": {
> > > "StringEquals": {
> > >
> > > "localhost:8080/auth/realms/demo:myclient
> > <http://10.0.26.1:8080/auth/realms/demo:myclient>": "account"
> > > }
> > > }
> > > }
> > > ]
> > > },
> > > "MaxSessionDuration": 3600
> > > }
> > > ]
> > >}
> >
> >
> >
> > I must be missing something here. Any advice you might have for me
> > would be greatly appreciated.
> >
> >
> >
> > Thank you
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
> > email to ceph-users-leave@xxxxxxx
> >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email
> to ceph-users-leave@xxxxxxx
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
