Hello, It seems like Pritha is the Ceph RGW expert in this forum. I am currently trying to integrate CephRGW object storage with KeyCloak as the OIDC provider. I am running ceph version 16.2.7 Pacific stable. At this point, I am just trying to get a POC working with the python scripts provided in the example in these docs < https://docs.ceph.com/en/latest/radosgw/STS/#sts-configuration> . Here are some step by step instructions on how I set up the ceph cluster and KeyCloak server: *Set up keycloak server*: 1. Create new Realm 'demo' 2. Create 'testuser' and add credentials. Verify that I am able to login to the realm using the new credentials. 3. Create a client 'myclient' and set Access Type as 'confidential' to generate client secret 4. Add a keycloak-oidc provider using the client credentials. 5. On the client set 'Authorization Enabled' to ON and 'Service Accounts Enabled' to ON. We should now be able to get the access tokens from the OIDC provider. To do this I used the sample curl calls from these docs < https://docs.ceph.com/en/latest/radosgw/keycloak/#setting-up-keycloak> which I put into scripts: access_token.sh #!/bin/bash KC_REALM=demo KC_CLIENT=myclient KC_CLIENT_SECRET=620b31fa-****-****-****-************ KC_SERVER=localhost:8080 <http://10.0.26.1:8080/> KC_CONTEXT=auth # Request Tokens for credentials KC_RESPONSE=$( \ curl -k -v -X POST \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "scope=openid" \ -d "grant_type=client_credentials" \ -d "client_id=$KC_CLIENT" \ -d "client_secret=$KC_CLIENT_SECRET" \ "http://$KC_SERVER/$KC_CONTEXT/realms/$KC_REALM/protocol/openid-connect/token"; \ | jq . ) KC_ACCESS_TOKEN=$(echo $KC_RESPONSE| jq -r .access_token) echo $KC_RESPONSE | jq . echo $KC_ACCESS_TOKEN Using this script I am able to get the access token for later usage and it has been verified that we are able to get the access token from the key cloak OIDC. *Set up Ceph Cluster w/ RGW*: 1. Create Ceph Cluster with OSD's and journals. Create an S3 object storage pool and then create an RGW on the cluster manager node. 2. Enable sts in the gateway config in /etc/ceph/ceph.conf as seen in the example from the docs < https://docs.ceph.com/en/latest/radosgw/keycloak/#setting-up-keycloak> : > [client.radosgw.gateway_name] > rgw sts key = abcdefghijklmnop > rgw s3 auth use sts = true 3. Create test users to be used in the test application python script. > radosgw-admin --uid TESTER --display-name "testuser" --access_key TESTER --secret test123 user create > radosgw-admin caps add --uid="TESTER" --caps="oidc-provider=*" > radosgw-admin caps add --uid="TESTER" --caps="roles=*" > > radosgw-admin --uid TESTER1 --display-name "testuser1" --access_key TESTER1 --secret test321 user create > radosgw-admin caps add --uid="TESTER1" --caps="roles=*" 4. We need to generate thumbprints of the OIDC provider. I used the docs here <https://docs.ceph.com/en/latest/radosgw/STS/#sts-configuration> to write a script to generate the thumbprints: # Get the 'x5c' from this response to turn into an IDP-cert KEY1_RESPONSE=$(curl -k -v \ -X GET \ -H "Content-Type: application/x-www-form-urlencoded" \ "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs"; \ | jq -r .keys[0].x5c) KEY2_RESPONSE=$(curl -k -v \ -X GET \ -H "Content-Type: application/x-www-form-urlencoded" \ "http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs"; \ | jq -r .keys[1].x5c) echo echo "Assembling Certificates...." # Assemble Cert1 echo '-----BEGIN CERTIFICATE-----' > certificate1.crt echo $(echo $KEY1_RESPONSE) | sed 's/^.//;s/.$//;s/^.//;s/.$//;s/^.//;s/.$//' >> certificate1.crt echo '-----END CERTIFICATE-----' >> certificate1.crt echo $(cat certificate1.crt) # Assemble Cert2 echo '-----BEGIN CERTIFICATE-----' > certificate2.crt echo $(echo $KEY2_RESPONSE) | sed 's/^.//;s/.$//;s/^.//;s/.$//;s/^.//;s/.$//' >> certificate2.crt echo '-----END CERTIFICATE-----' >> certificate2.crt echo $(cat certificate2.crt) echo echo "Generating thumbprints...." # Create Thumbprint for both certs PRETHUMBPRINT1=$(openssl x509 -in certificate1.crt -fingerprint -noout) PRETHUMBPRINT2=$(openssl x509 -in certificate2.crt -fingerprint -noout) PRETHUMBPRINT1=$(echo $PRETHUMBPRINT1 | awk '{ print substr($0, 18) }') PRETHUMBPRINT2=$(echo $PRETHUMBPRINT2 | awk '{ print substr($0, 18) }') echo "${PRETHUMBPRINT1//:}" echo "${PRETHUMBPRINT2//:}" I copied and pasted these thumbprints into the example application python script to perform the create_open_id_connect_provider() for the 'iam_client'. 5. Next I filled out the missing information in the example application script: #!/usr/bin/python3 import boto3 iam_client = boto3.client('iam', aws_access_key_id="TESTER", aws_secret_access_key="test123", endpoint_url="http://10.x.x.x:7480";, #<----Ceph RGW endpoint - using http for proof of concept region_name='' ) oidc_response = iam_client.create_open_id_connect_provider( Url="http://localhost:8080/auth/realms/demo";, ClientIDList=[ "myclient" ], ThumbprintList=[ "E43DBA95FC202A9773F3F542F12CF2A831FC7A6F", "CE0E06206F6F5670E2BC725BD1557D177B3708BF" ] ) policy_document = '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Federated":["arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo"]},"Action":["sts:AssumeRoleWithWebIdentity"],"Condition":{"StringEquals":{"localhost:8080/auth/realms/demo:app_id":"account"}}}]}''' role_response = iam_client.create_role( AssumeRolePolicyDocument=policy_document, Path='/', RoleName='S3Access', ) role_policy = '''{"Version":"2012-10-17","Statement":{"Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}}''' response = iam_client.put_role_policy( RoleName='S3Access', PolicyName='Policy1', PolicyDocument=role_policy ) #TESTER1 sts_client = boto3.client('sts', aws_access_key_id="TESTER1", aws_secret_access_key="test321", endpoint_url="http://10.x.x.x:7480";, region_name='', ) response = sts_client.assume_role_with_web_identity( RoleArn=get_response['Role']['Arn'], RoleSessionName='Bob', DurationSeconds=3600, WebIdentityToken="inserted-access-token" #<---- access token from step 5 in keycloak setup ) s3client = boto3.client('s3', aws_access_key_id = response['Credentials']['AccessKeyId'], aws_secret_access_key = response['Credentials']['SecretAccessKey'], aws_session_token = response['Credentials']['SessionToken'], endpoint_url="http://10.x.x.x:7480";, region_name='', ) bucket_name = 'my-bucket' s3bucket = s3client.create_bucket(Bucket=bucket_name) resp = s3client.list_buckets() print(resp) Currently, This script cannot run to completion. I find that it throws a fatal error when trying to run 'sts_client.assume_role_with_web_identity()' with the exception: "An error occurred (Unknown) when calling the AssumeRoleWithWebIdentity operation: Unknown" . I am able to verify that the OIDC provider and role has been created on the RGW: >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam list-open-id-connect-providers --region="" >{ > "OpenIDConnectProviderList": [ > { > "Arn": "arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo" > } > ] >} >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam get-open-id-connect-provider --open-id-connect-provider-arn="arn:aws:iam:::oidc-provider/localhost:8080/auth/realms/demo" --region="" >{ > "Url": "http://localhost:8080/auth/realms/demo";, > "ClientIDList": [ > "myclient" > ], > "ThumbprintList": [ > "E43DBA95FC202A9773F3F542F12CF2A831FC7A6F", > "CE0E06206F6F5670E2BC725BD1557D177B3708BF" > ], > "CreateDate": "2022-03-15T17:40:17.572000+00:00" >} >root@terminal# aws --endpoint=http://10.x.x.x:7480/ iam list-roles --region="" >{ > "Roles": [ > { > "Path": "/", > "RoleName": "S3Access", > "RoleId": "7329b54e-40a2-4476-ae68-52a72b43376c", > "Arn": "arn:aws:iam:::role/S3Access", > "CreateDate": "2022-03-14T21:30:11.750000+00:00", > "AssumeRolePolicyDocument": { > "Version": "2012-10-17", > "Statement": [ > { > "Effect": "Allow", > "Principal": { > "Federated": [ > "arn:aws:iam:::oidc-provider/localhost :8080/auth/realms/demo <http://10.0.26.1:8080/auth/realms/demo>" > ] > }, > "Action": [ > "sts:AssumeRoleWithWebIdentity" > ], > "Condition": { > "StringEquals": { > "localhost:8080/auth/realms/demo:myclient <http://10.0.26.1:8080/auth/realms/demo:myclient>": "account" > } > } > } > ] > }, > "MaxSessionDuration": 3600 > } > ] >} I must be missing something here. Any advice you might have for me would be greatly appreciated. Thank you _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |