On Wed, Mar 22, 2023 at 8:27 AM Frank Schilder <frans@xxxxxx> wrote:
> Hi Gregory,
>
> thanks for your reply. First a quick update. Here is how I get ln to work
> after it failed, there seems no timeout:
>
> $ ln envs/satwindspy/include/ffi.h
> mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
> ln: failed to create hard link
> 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h': Read-only file system
> $ ls -l envs/satwindspy/include mambaforge/pkgs/libffi-3.3-h58526e2_2
> envs/satwindspy/include:
> total 7664
> -rw-rw-r--. 1 rit rit 959 Mar 5 2021 ares_build.h
> [...]
> $ ln envs/satwindspy/include/ffi.h
> mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h
>
> After an ls -l on both directories ln works.
>
> To the question: How can I pull out a log from the nfs server? There is
> nothing in /var/log/messages.
So you’re using the kernel server and re-exporting, right?
I’m not very familiar with its implementation; I wonder if it’s doing
something strange via the kernel vfs.
AFAIK this isn’t really supportable for general use because nfs won’t
respect the CephFS file consistency protocol. But maybe it’s trying a bit
and that’s causing trouble?
-Greg
>
> I can't reproduce it with simple commands on the NFS client. It seems to
> occur only when a large number of files/dirs is created. I can make the
> archive available to you if this helps.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Gregory Farnum <gfarnum@xxxxxxxxxx>
> Sent: Wednesday, March 22, 2023 4:14 PM
> To: Frank Schilder
> Cc: ceph-users@xxxxxxx
> Subject: Re: Re: ln: failed to create hard link 'file name':
> Read-only file system
>
> Do you have logs of what the nfs server is doing?
> Managed to reproduce it in terms of direct CephFS ops?
>
>
> On Wed, Mar 22, 2023 at 8:05 AM Frank Schilder <frans@xxxxxx<mailto:
> frans@xxxxxx>> wrote:
> I have to correct myself. It also fails on an export with "sync" mode.
> Here is an strace on the client (strace ln envs/satwindspy/include/ffi.h
> mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h):
>
> [...]
> stat("mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h",
> 0x7ffdc5c32820) = -1 ENOENT (No such file or directory)
> lstat("envs/satwindspy/include/ffi.h", {st_mode=S_IFREG|0664,
> st_size=13934, ...}) = 0
> linkat(AT_FDCWD, "envs/satwindspy/include/ffi.h", AT_FDCWD,
> "mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h", 0) = -1 EROFS
> (Read-only file system)
> [...]
> write(2, "ln: ", 4ln: ) = 4
> write(2, "failed to create hard link 'mamb"..., 80failed to create hard
> link 'mambaforge/pkgs/libffi-3.3-h58526e2_2/include/ffi.h') = 80
> [...]
> write(2, ": Read-only file system", 23: Read-only file system) = 23
> write(2, "\n", 1
> ) = 1
> lseek(0, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
> close(0) = 0
> close(1) = 0
> close(2) = 0
> exit_group(1) = ?
> +++ exited with 1 +++
>
> Has anyone advice?
>
> Thanks!
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Frank Schilder <frans@xxxxxx<mailto:frans@xxxxxx>>
> Sent: Wednesday, March 22, 2023 2:44 PM
> To: ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
> Subject: ln: failed to create hard link 'file name':
> Read-only file system
>
> Hi all,
>
> on an NFS re-export of a ceph-fs (kernel client) I observe a very strange
> error. I'm un-taring a larger package (1.2G) and after some time I get
> these errors:
>
> ln: failed to create hard link 'file name': Read-only file system
>
> The strange thing is that this seems only temporary. When I used "ln src
> dst" for manual testing, the command failed as above. However, after that I
> tried "ln -v src dst" and this command created the hard link with exactly
> the same path arguments. During the period when the error occurs, I can't
> see any FS in read-only mode, neither on the NFS client nor the NFS server.
> Funny thing is that file creation and write still works, its only the
> hard-link creation that fails.
>
> For details, the set-up is:
>
> file-server: mount ceph-fs at /shares/path, export /shares/path as nfs4 to
> other server
> other server: mount /shares/path as NFS
>
> More precisely, on the file-server:
>
> fstab: MON-IPs:/shares/folder /shares/nfs/folder ceph
> defaults,noshare,name=NAME,secretfile=sec.file,mds_namespace=FS-NAME,_netdev
> 0 0
> exports: /shares/nfs/folder
> -no_root_squash,rw,async,mountpoint,no_subtree_check DEST-IP
>
> On the host at DEST-IP:
>
> fstab: FILE-SERVER-IP:/shares/nfs/folder /mnt/folder nfs defaults,_netdev
> 0 0
>
> Both, the file server and the client server are virtual machines. The file
> server is on Centos 8 stream (4.18.0-338.el8.x86_64) and the client machine
> is on AlmaLinux 8 (4.18.0-425.13.1.el8_7.x86_64).
>
> When I change the NFS export from "async" to "sync" everything works.
> However, that's a rather bad workaround and not a solution. Although this
> looks like an NFS issue, I'm afraid it is a problem with hard links and
> ceph-fs. It looks like a race with scheduling and executing operations on
> the ceph-fs kernel mount.
>
> Has anyone seen something like that?
>
> Thanks and best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
> To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:
> ceph-users-leave@xxxxxxx>
>
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
