Hi Matt, Thanks for the reply. I did following as you suggested: bash-4.4$ ceph auth get-or-create client.demouser mon 'allow r' osd 'allow rw pool=.nfs namespace=nfs4rgw, allow rw tag cephfs data=myfs' mds 'allow rw path=/bucketexport' [client.demouser] key = AQCZJ8BjDbqZKBAAQVQbGZ4EYATtENbMv6a/sA== bash-4.4$ ceph nfs export create rgw --cluster-id nfs4rgw --pseudo-path /bucketexport --bucket testbk --user-id demouser After this, Ganesha server crashes the same way. I wonder where is the problem to set it up. Ben > 2023年1月12日 21:03，Matt Benjamin <mbenjami@xxxxxxxxxx> 写道： > > Hi Ben, > > The issue seems to be that you don't have a ceph keyring available to the nfs-ganesha server. The upstream doc talks about this. The nfs-ganesha runtime environment needs to be essentially identical to one (a pod, I guess) that would run radosgw. > > Matt > > On Thu, Jan 12, 2023 at 7:27 AM Ruidong Gao <ruidong.gao@xxxxxxxxx <mailto:ruidong.gao@xxxxxxxxx>> wrote: >> Hi, >> >> This is running Quincy 17.2.5 deployed by rook on k8s. RGW nfs export will crash Ganesha server pod. CephFS export works just fine. Here are steps of it: >> 1, create export: >> bash-4.4$ ceph nfs export create rgw --cluster-id nfs4rgw --pseudo-path /bucketexport --bucket testbk >> { >> "bind": "/bucketexport", >> "path": "testbk", >> "cluster": "nfs4rgw", >> "mode": "RW", >> "squash": "none" >> } >> >> 2, check pods status afterwards: >> rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h3m >> rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 1/2 Error 2 4h6m >> >> 3, check failing pod’s logs: >> >> 11/01/2023 08:11:53 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 >> 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_start_grace :STATE :EVENT :grace reload client info completed from backend >> 11/01/2023 08:11:54 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_try_lift_grace :STATE :EVENT :check grace:reclaim complete(0) clid count(0) >> 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE >> 11/01/2023 08:11:57 : epoch 63be6f49 : rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 : nfs-ganesha-1[main] export_defaults_commit :CONFIG :INFO :Export Defaults now (options=03303002/00080000 , , , , , , , , expire= 0) >> 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory >> 2023-01-11T08:11:57.853+0000 7f59dac7c200 -1 AuthRegistry(0x56476817a480) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx >> 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-admin/keyring: (2) No such file or directory >> 2023-01-11T08:11:57.855+0000 7f59dac7c200 -1 AuthRegistry(0x7ffe4d092c90) no keyring found at /var/lib/ceph/radosgw/ceph-admin/keyring, disabling cephx >> 2023-01-11T08:11:57.856+0000 7f5987537700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] >> 2023-01-11T08:11:57.856+0000 7f5986535700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] >> 2023-01-11T08:12:00.861+0000 7f5986d36700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] >> 2023-01-11T08:12:00.861+0000 7f59dac7c200 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication >> failed to fetch mon config (--no-mon-config to skip) >> >> 4, delete the export: >> ceph nfs export delete nfs4rgw /bucketexport >> >> Ganesha servers go back normal: >> rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0 4h30m >> rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 2/2 Running 10 4h33m >> >> Any ideas to make it work? >> >> Thanks >> Ben >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx <mailto:ceph-users-leave@xxxxxxx> > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |