Hello Wesley, Thank you for the response. I tried the same but ended up with 403. Regards, Jayanth On Wed, Nov 8, 2023 at 7:34 PM Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx> wrote: > Jaynath: > > Just to be clear with the "--admin" user's key's you have attempted to > delete the bucket policy using the following method: > https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket-policy.html > > This is what worked for me (on a 16.2.14 cluster). I didn't attempt to > interact with the affected bucket in any way other than "aws s3api > delete-bucket-policy" > > Respectfully, > > *Wes Dillingham* > wes@xxxxxxxxxxxxxxxxx > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > > > On Wed, Nov 8, 2023 at 8:30 AM Jayanth Reddy <jayanthreddy5666@xxxxxxxxx> > wrote: > >> Hello Casey, >> >> We're totally stuck at this point and none of the options seem to work. >> Please let us know if there is something in metadata or index to remove >> those applied bucket policies. We downgraded to v17.2.6 and encountering >> the same. >> >> Regards, >> Jayanth >> >> On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5666@xxxxxxxxx> >> wrote: >> >>> Hello Casey, >>> >>> And on further inspection, we identified that there were bucket policies >>> set from the initial days; we were in v16.2.12. >>> We upgraded the cluster to v17.2.7 two days ago and it seems obvious >>> that the IAM error logs are generated the next minute rgw daemon upgraded >>> from v16.2.12 to v17.2.7. Looks like there is some issue with parsing. >>> >>> I'm thinking to downgrade back to v17.2.6 and earlier, please let me >>> know if this is a good option for now. >>> >>> Thanks, >>> Jayanth >>> ------------------------------ >>> *From:* Jayanth Reddy <jayanthreddy5666@xxxxxxxxx> >>> *Sent:* Tuesday, November 7, 2023 11:59:38 PM >>> *To:* Casey Bodley <cbodley@xxxxxxxxxx> >>> *Cc:* Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx>; ceph-users < >>> ceph-users@xxxxxxx>; Adam Emerson <aemerson@xxxxxxxxxx> >>> *Subject:* Re: Re: owner locked out of bucket via bucket >>> policy >>> >>> Hello Casey, >>> >>> Thank you for the quick response. I see >>> `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please >>> let me know. >>> >>> Regards >>> Jayanth >>> >>> On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote: >>> >>> On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy >>> <jayanthreddy5666@xxxxxxxxx> wrote: >>> > >>> > Hello Wesley and Casey, >>> > >>> > We've ended up with the same issue and here it appears that even the >>> user with "--admin" isn't able to do anything. We're now unable to figure >>> out if it is due to bucket policies, ACLs or IAM of some sort. I'm seeing >>> these IAM errors in the logs >>> > >>> > ``` >>> > >>> > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851 >>> 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to >>> Handler error. >>> > >>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 >>> 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due >>> to Handler error. >>> >>> it's failing to parse the bucket policy document, but the error >>> message doesn't say what's wrong with it >>> >>> disabling rgw_policy_reject_invalid_principals might help if it's >>> failing on the Principal >>> >>> > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 >>> 0.000000000s s3:list_bucket init_permissions on >>> :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13 >>> > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req >>> 13293029267332025583 0.000000000s op->ERRORHANDLER: err_no=-13 >>> new_err_no=-13 >>> > >>> > ``` >>> > >>> > Please help what's wrong here. We're in Ceph v17.2.7. >>> > >>> > Regards, >>> > Jayanth >>> > >>> > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham < >>> wes@xxxxxxxxxxxxxxxxx> wrote: >>> >> >>> >> Thank you, this has worked to remove the policy. >>> >> >>> >> Respectfully, >>> >> >>> >> *Wes Dillingham* >>> >> wes@xxxxxxxxxxxxxxxxx >>> >> LinkedIn <http://www.linkedin.com/in/wesleydillingham> >>> >> >>> >> >>> >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley@xxxxxxxxxx> >>> wrote: >>> >> >>> >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham < >>> wes@xxxxxxxxxxxxxxxxx> >>> >> > wrote: >>> >> > > >>> >> > > Thank you, I am not sure (inherited cluster). I presume such an >>> admin >>> >> > user created after-the-fact would work? >>> >> > >>> >> > yes >>> >> > >>> >> > > Is there a good way to discover an admin user other than iterate >>> over >>> >> > all users and retrieve user information? (I presume radosgw-admin >>> user info >>> >> > --uid=<user>" would illustrate such administrative access? >>> >> > >>> >> > not sure there's an easy way to search existing users, but you could >>> >> > create a temporary admin user for this repair >>> >> > >>> >> > > >>> >> > > Respectfully, >>> >> > > >>> >> > > Wes Dillingham >>> >> > > wes@xxxxxxxxxxxxxxxxx >>> >> > > LinkedIn >>> >> > > >>> >> > > >>> >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley@xxxxxxxxxx> >>> wrote: >>> >> > >> >>> >> > >> if you have an administrative user (created with --admin), you >>> should >>> >> > >> be able to use its credentials with awscli to delete or >>> overwrite this >>> >> > >> bucket policy >>> >> > >> >>> >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham < >>> >> > wes@xxxxxxxxxxxxxxxxx> wrote: >>> >> > >> > >>> >> > >> > I have a bucket which got injected with bucket policy which >>> locks the >>> >> > >> > bucket even to the bucket owner. The bucket now cannot be >>> accessed >>> >> > (even >>> >> > >> > get its info or delete bucket policy does not work) I have >>> looked in >>> >> > the >>> >> > >> > radosgw-admin command for a way to delete a bucket policy but >>> do not >>> >> > see >>> >> > >> > anything. I presume I will need to somehow remove the bucket >>> policy >>> >> > from >>> >> > >> > however it is stored in the bucket metadata / omap etc. If >>> anyone can >>> >> > point >>> >> > >> > me in the right direction on that I would appreciate it. Thanks >>> >> > >> > >>> >> > >> > Respectfully, >>> >> > >> > >>> >> > >> > *Wes Dillingham* >>> >> > >> > wes@xxxxxxxxxxxxxxxxx >>> >> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham> >>> >> > >> > _______________________________________________ >>> >> > >> > ceph-users mailing list -- ceph-users@xxxxxxx >>> >> > >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >> > >> > >>> >> > >> >>> >> > >>> >> > >>> >> _______________________________________________ >>> >> ceph-users mailing list -- ceph-users@xxxxxxx >>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> >>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |