I have been having this for some time, it pops up out of the blue. Next time this occurs I will enable the logging. Thanks, Marc -----Original Message----- From: Daniel Gryniewicz [mailto:dang@xxxxxxxxxx] Sent: 12 December 2018 16:49 To: Marc Roos; ceph-users Subject: Re: How to troubleshoot rsync to cephfs via nfs-ganesha stalling Okay, this all looks fine, and it's extremely unlikely that a text file will have holes in it (I thought holes, because rsync handles holes, but wget would just copy zeros instead). Is this reproducible? If so, can you turn up Ganesha logging and post a log file somewhere? Daniel On 12/12/2018 04:56 AM, Marc Roos wrote: > > Hi Daniel, thanks for looking at this. > > These are the mount options > type nfs4 > (rw,nodev,relatime,vers=4,intr,local_lock=none,retrans=2,proto=tcp,rsi > ze > =8192,wsize=8192,hard,namlen=255,sec=sys) > > I have overwritten the original files, so I cannot examine if they had > holes. To be honest I don't even know how to query the file, to > identify holes. > > These are the contents of the files, just plain text. > [@os0 CentOS7-x86_64]# cat CentOS_BuildTag 20181125-1500 [@os0 > CentOS7-x86_64]# cat .discinfo 1543162572.807980 > 7.6 > x86_64 > > > > -----Original Message----- > From: Daniel Gryniewicz [mailto:dang@xxxxxxxxxx] > Sent: 10 December 2018 15:54 > To: ceph-users@xxxxxxxxxxxxxx > Subject: Re: How to troubleshoot rsync to cephfs via > nfs-ganesha stalling > > This isn't something I've seen before. rsync generally works fine, even > over cephfs. More inline. > > On 12/09/2018 09:42 AM, Marc Roos wrote: >> >> >> This rsync command fails and makes the local nfs unavailable (Have to >> stop nfs-ganesha, kill all rsync processes on the client and then >> start >> nfs-ganesha) >> >> rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*" >> --exclude "isos" >> anonymous@xxxxxxxxxxxxxxxxxxxxxxxxxxx::centos/7/os/x86_64/ >> /localpath/CentOS7-x86_64/ >> >> When I do individual rsyncs on the subfolders >> >> -rw-r--r-- 1 nobody 500 14 Nov 25 17:01 CentOS_BuildTag >> -rw-r--r-- 1 nobody 500 29 Nov 25 17:16 .discinfo >> drwxr-xr-x 3 nobody 500 8.3M Nov 25 17:20 EFI >> -rw-rw-r-- 1 nobody 500 227 Aug 30 2017 EULA >> -rw-rw-r-- 1 nobody 500 18K Dec 9 2015 GPL drwxr-xr-x 3 nobody 500 >> 572M Nov 25 17:21 images drwxr-xr-x 2 nobody 500 57M Dec 9 14:11 >> isolinux drwxr-xr-x 2 nobody 500 433M Nov 25 17:20 LiveOS drwxrwxr-x 2 > >> nobody 500 9.5G Nov 25 16:58 Packages drwxrwxr-x 2 nobody 500 29M Dec > >> 9 13:53 repodata >> -rw-rw-r-- 1 nobody 500 1.7K Dec 9 2015 RPM-GPG-KEY-CentOS-7 >> -rw-rw-r-- 1 nobody 500 1.7K Dec 9 2015 RPM-GPG-KEY-CentOS-Testing-7 >> -rw-r--r-- 1 nobody 500 354 Nov 25 17:21 .treeinfo >> >> These rsyncs are all going fine. >> >> rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*" >> --exclude "isos" >> anonymous@xxxxxxxxxxxxxxxxxxxxxxxxxxx::centos/7/os/x86_64/Packages/ >> /localpath/CentOS7-x86_64/Packages/ >> rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*" >> --exclude "isos" >> anonymous@xxxxxxxxxxxxxxxxxxxxxxxxxxx::centos/7/os/x86_64/repodata/ >> /localpath/CentOS7-x86_64/repodata/ >> rsync -rlptDvSHP --delete --exclude config.repo --exclude "local*" >> --exclude "isos" >> anonymous@xxxxxxxxxxxxxxxxxxxxxxxxxxx::centos/7/os/x86_64/LiveOS/ >> /localpath/CentOS7-x86_64/LiveOS/ >> >> Except when I try to rsync the file CentOS_BuildTag then everything >> stalls. Leaving such files >> -rw------- 1 500 500 0 Dec 9 14:26 .CentOS_BuildTag.2igwc5 >> -rw------- 1 500 500 0 Dec 9 14:28 .CentOS_BuildTag.tkiwc5 > > So something is failing on the write, it seems. These are the temporary > files made by rsync, and they're empty, so the initial write seems to > have failed. > >> I can resolf this by doing a wget and moving the file to the location >> wget >> > 'http://mirror.ams1.nl.leaseweb.net/centos/7/os/x86_64/CentOS_BuildTag' >> mv CentOS_BuildTag /localpath/CentOS7-x86_64/ >> >> I had also problems with .discinfo and when I ls this directory on >> cephfs mount it takes a long time to produce output. >> >> When I do the full rsync to the cephfs mount it completes without >> errors, when I then later do the sync on the nfs mount it completes >> also (nothing being copied) > > This confirms that it's not metadata related, as this second successful > rsync is purely metadata. > >> Anybody know what I should do to resolv this? Is this a typical >> ganesha issue or is this cephfs corruption, that make ganesha stall? > > Writes in Ganesha are pretty much passthrough, modulo some metadata > tracking. This means that a write hang is likely to be somewhere > between Ganesha and CephFS. However, this is a single, small file, so I > don't see how it could hang, especially when wget can copy the file > correctly. Maybe there's something about the structure of the file? > Does it have holes in it, for example? > > Also, can you send the mount options for the NFS mount? > > Daniel > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |