Hi Meghana, > Il giorno 18/giu/2015, alle ore 16:06, Meghana Madhusudhan <mmadhusu@xxxxxxxxxx> ha scritto: > > Hi Allesandro, > > I need the following output from you, > > 1. After you execute ganesha.enable on command. > ps aux | grep ganesha root 6699 94.0 1.1 953488 193272 ? Ssl Jun17 1137:36 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -p /var/run/ganesha.nfsd.pid root 22263 0.0 0.0 112644 960 pts/0 S+ 16:14 0:00 grep --color=auto ganesha (both nodes of the cluster show the same, apart from the PIDs, of course) > showmount -e localhost Export list for node1: /atlas-backup-01 (everyone) The same output is shown on the other node. > cat /etc/ganesha/ganesha.conf # Ganesha options file %include "/etc/ganesha/ganesha-opts.conf" %include "/etc/ganesha/exports/export.atlas-backup-01.conf" The first include just point to a different file with the following lines, to change the rquota port NFS_Core_Param { #Use a non-privileged port for RQuota Rquota_Port = 4501; } > cat "/etc/ganesha/exports/export.VOLNAME.conf" or "/usr/..../export.VOLNAME.conf” # WARNING : Using Gluster CLI will overwrite manual # changes made to this file. To avoid it, edit the # file, copy it over to all the NFS-Ganesha nodes # and run ganesha-ha.sh --refresh-config. EXPORT{ Export_Id= 2 ; Path = "/atlas-backup-01"; FSAL { name = GLUSTER; hostname="localhost"; volume="atlas-backup-01"; } Access_type = RW; Disable_ACL = true; Squash="No_root_squash"; Pseudo="/atlas-backup-01"; Protocols = "3", "4" ; Transports = "UDP","TCP"; SecType = "sys"; } > tail /var/log/ganesha.log or wherever the ganesha log is. 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 17/06/2015 20:02:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE 18/06/2015 16:13:50 : epoch 5581b5da : node2 : ganesha.nfsd-26983[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/' > > 2. gluster vol set VOLNAME ganesha.enable off > > Output of the same as above. # gluster vol set atlas-backup-01 ganesha.enable off volume set: failed: Dynamic export addition/deletion failed. Please see log file for details # ps aux | grep ganesha root 32033 0.0 0.0 112640 960 pts/0 S+ 16:20 0:00 grep --color=auto ganesha [This is due to the fact that the export file was deleted by the config still references it, so ganesha fails to resume] # cat /etc/ganesha/ganesha.conf # Ganesha options file %include "/etc/ganesha/ganesha-opts.conf" %include "/etc/ganesha/exports/export.atlas-backup-01.conf" # cat /etc/ganesha/exports/export.atlas-backup-01.conf cat: /etc/ganesha/exports/export.atlas-backup-01.conf: No such file or directory # tail /var/log/ganesha.log 18/06/2015 16:19:20 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/' 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (2:2) 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor. 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 18/06/2015 16:20:22 : epoch 5582d3a6 : node2 : ganesha.nfsd-31542[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31550[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31557[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31562[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31567[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31908[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31932[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31949[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31956[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31971[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2 I have not enabled the FULL_DEBUG but if you need it I can do it. > > Thanks for your perseverance :) I have to thank you for the help! :-) Cheers, Alessandro > > Meghana > > ----- Original Message ----- > From: "Alessandro De Salvo" <Alessandro.DeSalvo@xxxxxxxxxxxxx> > To: "Meghana Madhusudhan" <mmadhusu@xxxxxxxxxx> > Cc: gluster-users@xxxxxxxxxxx, nfs-ganesha-devel@xxxxxxxxxxxxxxxxxxxxx, "Soumya Koduri" <skoduri@xxxxxxxxxx> > Sent: Thursday, June 18, 2015 7:24:55 PM > Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster > > Hi Meghana, > >> Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan <mmadhusu@xxxxxxxxxx> ha scritto: >> >> >> >> >> On 06/17/2015 10:57 PM, Alessandro De Salvo wrote: >>> Hi, >>> when disabling exports from gluster 3.7.1, by using gluster vol set <volume> ganesha.enable off, I always get the following error: >>> >>> Error: Dynamic export addition/deletion failed. Please see log file for details >>> >>> This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see: >> you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by >> Gluster-CLI to unexport the volume "gluster volume set <volname> >> ganesha.enable off" which rightly deletes the export file too while >> un-exporting the volume. >> >>> >>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 >>> Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus) >>> >>> So, there is a timeout and it fails completely. >> Check if nfs-ganesha is still running. There was a bug in unexporting >> the volume. Its been fixed recently in V2.3-dev, yet to be back-ported >> to V2.2-stable branch. >> https://review.gerrithub.io/#/c/236129/ >> >> Thanks, >> Soumya >> >>> In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration. >> >> Yes. I see that the "sed -i /$VOL.conf/d $CONF" is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal >> will fail in three cases, >> 1. nfs-ganesha is not running. > > no, it was running > >> 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID. > > I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems. > >> 3. The bug that Soumya pointed out. > > This might well be the real cause! > >> >> If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed, >> I can help you with it. > > Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works. > >> >> Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in "/etc/ganesha/exports" dir. > > And this works perfectly, I see them correctly. > >> If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine. > > But this is not, at least in my case. > The command I’m using are just the following: > > gluster vol set <volume> ganesha.enable on > gluster vol set <volume> ganesha.enable off > > It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume. > The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport. > Many thanks for the help, > > Alessandro > > >> >> Meghana >> >> >>> >>> What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works: >>> >>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2 >>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2 >>> >>> So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why? >>> I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3. >>> Thanks, >>> >>> Alessandro >>> >>> PS: sorry for reporting so many things in a few days :-) >>> >>> >>> >>> ------------------------------------------------------------------------------ >>> >>> >>> >>> _______________________________________________ >>> Nfs-ganesha-devel mailing list >>> Nfs-ganesha-devel@xxxxxxxxxxxxxxxxxxxxx >>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >>> >> >> ------------------------------------------------------------------------------ >> _______________________________________________ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@xxxxxxxxxxxxxxxxxxxxx >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >> > >
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users