Hello Casey, We're totally stuck at this point and none of the options seem to work. Please let us know if there is something in metadata or index to remove those applied bucket policies. We downgraded to v17.2.6 and encountering the same. Regards, Jayanth On Wed, Nov 8, 2023 at 7:14 AM Jayanth Reddy <jayanthreddy5666@xxxxxxxxx> wrote: > Hello Casey, > > And on further inspection, we identified that there were bucket policies > set from the initial days; we were in v16.2.12. > We upgraded the cluster to v17.2.7 two days ago and it seems obvious that > the IAM error logs are generated the next minute rgw daemon upgraded from > v16.2.12 to v17.2.7. Looks like there is some issue with parsing. > > I'm thinking to downgrade back to v17.2.6 and earlier, please let me know > if this is a good option for now. > > Thanks, > Jayanth > ------------------------------ > *From:* Jayanth Reddy <jayanthreddy5666@xxxxxxxxx> > *Sent:* Tuesday, November 7, 2023 11:59:38 PM > *To:* Casey Bodley <cbodley@xxxxxxxxxx> > *Cc:* Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx>; ceph-users < > ceph-users@xxxxxxx>; Adam Emerson <aemerson@xxxxxxxxxx> > *Subject:* Re: Re: owner locked out of bucket via bucket > policy > > Hello Casey, > > Thank you for the quick response. I see > `rgw_policy_reject_invalid_principals` is not present in v17.2.7. Please > let me know. > > Regards > Jayanth > > On Tue, Nov 7, 2023 at 11:50 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote: > > On Tue, Nov 7, 2023 at 12:41 PM Jayanth Reddy > <jayanthreddy5666@xxxxxxxxx> wrote: > > > > Hello Wesley and Casey, > > > > We've ended up with the same issue and here it appears that even the > user with "--admin" isn't able to do anything. We're now unable to figure > out if it is due to bucket policies, ACLs or IAM of some sort. I'm seeing > these IAM errors in the logs > > > > ``` > > > > Nov 7 00:02:00 ceph-05 radosgw[4054570]: req 8786689665323103851 > 0.003999968s s3:get_obj Error reading IAM Policy: Terminate parsing due to > Handler error. > > > > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 > 0.000000000s s3:list_bucket Error reading IAM Policy: Terminate parsing due > to Handler error. > > it's failing to parse the bucket policy document, but the error > message doesn't say what's wrong with it > > disabling rgw_policy_reject_invalid_principals might help if it's > failing on the Principal > > > Nov 7 22:51:40 ceph-05 radosgw[4054570]: req 13293029267332025583 > 0.000000000s s3:list_bucket init_permissions on > :window-dev[1d0fa0b4-04eb-48f9-889b-a60de865ccd8.24143.10]) failed, ret=-13 > > Nov 7 22:51:40 ceph-feed-05 radosgw[4054570]: req 13293029267332025583 > 0.000000000s op->ERRORHANDLER: err_no=-13 new_err_no=-13 > > > > ``` > > > > Please help what's wrong here. We're in Ceph v17.2.7. > > > > Regards, > > Jayanth > > > > On Thu, Oct 26, 2023 at 7:14 PM Wesley Dillingham <wes@xxxxxxxxxxxxxxxxx> > wrote: > >> > >> Thank you, this has worked to remove the policy. > >> > >> Respectfully, > >> > >> *Wes Dillingham* > >> wes@xxxxxxxxxxxxxxxxx > >> LinkedIn <http://www.linkedin.com/in/wesleydillingham> > >> > >> > >> On Wed, Oct 25, 2023 at 5:10 PM Casey Bodley <cbodley@xxxxxxxxxx> > wrote: > >> > >> > On Wed, Oct 25, 2023 at 4:59 PM Wesley Dillingham < > wes@xxxxxxxxxxxxxxxxx> > >> > wrote: > >> > > > >> > > Thank you, I am not sure (inherited cluster). I presume such an > admin > >> > user created after-the-fact would work? > >> > > >> > yes > >> > > >> > > Is there a good way to discover an admin user other than iterate > over > >> > all users and retrieve user information? (I presume radosgw-admin > user info > >> > --uid=<user>" would illustrate such administrative access? > >> > > >> > not sure there's an easy way to search existing users, but you could > >> > create a temporary admin user for this repair > >> > > >> > > > >> > > Respectfully, > >> > > > >> > > Wes Dillingham > >> > > wes@xxxxxxxxxxxxxxxxx > >> > > LinkedIn > >> > > > >> > > > >> > > On Wed, Oct 25, 2023 at 4:41 PM Casey Bodley <cbodley@xxxxxxxxxx> > wrote: > >> > >> > >> > >> if you have an administrative user (created with --admin), you > should > >> > >> be able to use its credentials with awscli to delete or overwrite > this > >> > >> bucket policy > >> > >> > >> > >> On Wed, Oct 25, 2023 at 4:11 PM Wesley Dillingham < > >> > wes@xxxxxxxxxxxxxxxxx> wrote: > >> > >> > > >> > >> > I have a bucket which got injected with bucket policy which > locks the > >> > >> > bucket even to the bucket owner. The bucket now cannot be > accessed > >> > (even > >> > >> > get its info or delete bucket policy does not work) I have > looked in > >> > the > >> > >> > radosgw-admin command for a way to delete a bucket policy but do > not > >> > see > >> > >> > anything. I presume I will need to somehow remove the bucket > policy > >> > from > >> > >> > however it is stored in the bucket metadata / omap etc. If > anyone can > >> > point > >> > >> > me in the right direction on that I would appreciate it. Thanks > >> > >> > > >> > >> > Respectfully, > >> > >> > > >> > >> > *Wes Dillingham* > >> > >> > wes@xxxxxxxxxxxxxxxxx > >> > >> > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > >> > >> > _______________________________________________ > >> > >> > ceph-users mailing list -- ceph-users@xxxxxxx > >> > >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx > >> > >> > > >> > >> > >> > > >> > > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |