Hello Casey, Thank you so much for the response. I'm applying these right now and let you know the results. Regards, Jayanth On Wed, Nov 8, 2023 at 8:15 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote: > i've opened https://tracker.ceph.com/issues/63485 to allow > admin/system users to override policy parsing errors like this. i'm > not sure yet where this parsing regression was introduced. in reef, > https://github.com/ceph/ceph/pull/49395 added better error messages > here, along with a rgw_policy_reject_invalid_principals option to be > strict about principal names > > > to remove a bucket policy that fails to parse with "Error reading IAM > Policy", you can follow these steps: > > 1. find the bucket's instance id using the 'bucket stats' command > > $ radosgw-admin bucket stats --bucket {bucketname} | grep id > > 2. use the rados tool to remove the bucket policy attribute > (user.rgw.iam-policy) from the bucket instance metadata object > > $ rados -p default.rgw.meta -N root rmxattr > .bucket.meta.{bucketname}:{bucketid} user.rgw.iam-policy > > 3. radosgws may be caching the existing bucket metadata and xattrs, so > you'd either need to restart them or clear their metadata caches > > $ ceph daemon client.rgw.xyz cache zap > > On Wed, Nov 8, 2023 at 9:06 AM Jayanth Reddy <jayanthreddy5666@xxxxxxxxx> > wrote: > > > > Hello Wesley, > > Thank you for the response. I tried the same but ended up with 403. > > > > Regards, > > Jayanth > > > > On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx> > wrote: > >> > >> Jaynath: > >> > >> Just to be clear with the "--admin" user's key's you have attempted to > delete the bucket policy using the following method: > https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy.html > >> > >> This is what worked for me (on a 16.2.14 cluster). I didn't attempt to > interact with the affected bucket in any way other than "aws s3api > delete-bucket-policy" > >> > >> Respectfully, > >> > >> Wes Dillingham > >> wes@xxxxxxxxxxxxxxxxx > >> LinkedIn > >> > >> > >> On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy < > jayanthreddy5666@xxxxxxxxx> wrote: > >>> > >>> Hello Casey, > >>> > >>> We're totally stuck at this point and none of the options seem to > work. Please let us know if there is something in metadata or index to > remove those applied bucket policies. We downgraded to v17.2.6 and > encountering the same. > >>> > >>> Regards, > >>> Jayanth > >>> > >>> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy < > jayanthreddy5666@xxxxxxxxx> wrote: > >>>> > >>>> Hello Casey, > >>>> > >>>> And on further inspection, we identified that there were bucket > policies set from the initial days; we were in v16.2.12. > >>>> We upgraded the cluster to v17.2.7 two days ago and it seems obvious > that the IAM error logs are generated the next minute rgw daemon upgraded > from v16.2.12 to v17.2.7. Looks like there is some issue with parsing. > >>>> > >>>> I'm thinking to downgrade back to v17.2.6 and earlier, please let me > know if this is a good option for now. > >>>> > >>>> Thanks, > >>>> Jayanth > >>>> ________________________________ > >>>> From: Jayanth Reddy <jayanthreddy5666@xxxxxxxxx> > >>>> Sent: Tuesday, November 7, 2023 11:59:38 PM > >>>> To: Casey Bodley <cbodley@xxxxxxxxxx> > >>>> Cc: Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx>; ceph-users < > ceph-users@xxxxxxx>; Adam Emerson <aemerson@xxxxxxxxxx> > >>>> Subject: Re: Re: owner locked out of bucket via bucket > policy > >>>> > >>>> Hello Casey, > >>>> > >>>> Thank you for the quick response. I see > `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please > let me know. > >>>> > >>>> Regards > >>>> Jayanth > >>>> > >>>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley@xxxxxxxxxx> > wrote: > >>>> > >>>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy > >>>> <jayanthreddy5666@xxxxxxxxx> wrote: > >>>> > > >>>> > Hello Wesley and Casey, > >>>> > > >>>> > We've ended up with the same issue and here it appears that even > the user with "--admin" isn't able to do anything. We're now unable to > figure out if it is due to bucket policies, ACLs or IAM of some sort. I'm > seeing these IAM errors in the logs > >>>> > > >>>> > ``` > >>>> > > >>>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851 > 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to > Handler error. > >>>> > > >>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 > 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due > to Handler error. > >>>> > >>>> it's failing to parse the bucket policy document, but the error > >>>> message doesn't say what's wrong with it > >>>> > >>>> disabling rgw_policy_reject_invalid_principals might help if it's > >>>> failing on the Principal > >>>> > >>>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 > 0.000000000s s3:list_bucket init_permissions on > :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13 > >>>> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req > 13293029267332025583 0.000000000s op->ERRORHANDLER: err_no=-13 > new_err_no=-13 > >>>> > > >>>> > ``` > >>>> > > >>>> > Please help what's wrong here. We're in Ceph v17.2.7. > >>>> > > >>>> > Regards, > >>>> > Jayanth > >>>> > > >>>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham < > wes@xxxxxxxxxxxxxxxxx> wrote: > >>>> >> > >>>> >> Thank you, this has worked to remove the policy. > >>>> >> > >>>> >> Respectfully, > >>>> >> > >>>> >> *Wes Dillingham* > >>>> >> wes@xxxxxxxxxxxxxxxxx > >>>> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham> > >>>> >> > >>>> >> > >>>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley@xxxxxxxxxx> > wrote: > >>>> >> > >>>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham < > wes@xxxxxxxxxxxxxxxxx> > >>>> >> > wrote: > >>>> >> > > > >>>> >> > > Thank you, I am not sure (inherited cluster). I presume such > an admin > >>>> >> > user created after-the-fact would work? > >>>> >> > > >>>> >> > yes > >>>> >> > > >>>> >> > > Is there a good way to discover an admin user other than > iterate over > >>>> >> > all users and retrieve user information? (I presume > radosgw-admin user info > >>>> >> > --uid=<user>" would illustrate such administrative access? > >>>> >> > > >>>> >> > not sure there's an easy way to search existing users, but you > could > >>>> >> > create a temporary admin user for this repair > >>>> >> > > >>>> >> > > > >>>> >> > > Respectfully, > >>>> >> > > > >>>> >> > > Wes Dillingham > >>>> >> > > wes@xxxxxxxxxxxxxxxxx > >>>> >> > > LinkedIn > >>>> >> > > > >>>> >> > > > >>>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley < > cbodley@xxxxxxxxxx> wrote: > >>>> >> > >> > >>>> >> > >> if you have an administrative user (created with --admin), > you should > >>>> >> > >> be able to use its credentials with awscli to delete or > overwrite this > >>>> >> > >> bucket policy > >>>> >> > >> > >>>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham < > >>>> >> > wes@xxxxxxxxxxxxxxxxx> wrote: > >>>> >> > >> > > >>>> >> > >> > I have a bucket which got injected with bucket policy which > locks the > >>>> >> > >> > bucket even to the bucket owner. The bucket now cannot be > accessed > >>>> >> > (even > >>>> >> > >> > get its info or delete bucket policy does not work) I have > looked in > >>>> >> > the > >>>> >> > >> > radosgw-admin command for a way to delete a bucket policy > but do not > >>>> >> > see > >>>> >> > >> > anything. I presume I will need to somehow remove the > bucket policy > >>>> >> > from > >>>> >> > >> > however it is stored in the bucket metadata / omap etc. If > anyone can > >>>> >> > point > >>>> >> > >> > me in the right direction on that I would appreciate it. > Thanks > >>>> >> > >> > > >>>> >> > >> > Respectfully, > >>>> >> > >> > > >>>> >> > >> > *Wes Dillingham* > >>>> >> > >> > wes@xxxxxxxxxxxxxxxxx > >>>> >> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > >>>> >> > >> > _______________________________________________ > >>>> >> > >> > ceph-users mailing list -- ceph-users@xxxxxxx > >>>> >> > >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>>> >> > >> > > >>>> >> > >> > >>>> >> > > >>>> >> > > >>>> >> _______________________________________________ > >>>> >> ceph-users mailing list -- ceph-users@xxxxxxx > >>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>>> > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |