On Thu, 2019-05-16 at 20:31 +0800, Xinying Song wrote:
> Hi, cephers:
> I'm using ceph Luminous. Recently, one of my colleagues put his
> gcc-tools , a whole binary folder with corresponding lib-files, into
> cephfs. However, when running arm-gcc command to compile files,
> arm-gcc always complains about "cc1plus" not found. If the whole
> binary folder is located under a local filesystem, everything is ok.
>
> Here is some pieces of log with strace when run arm-gcc under cephfs:
>
> open("/tmp/ccIlAAE5.s", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
> close(3) = 0
> stat64("/mnt/cephfs/CentOS-6.6/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../libexec/gcc/arm-linux-gnueabihf/4.9.2/cc1plus",
> {st_mode=S_IFREG|0755, st_size=16224744, ...}) = 0
> stat64("/mnt/cephfs/CentOS-6.6/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../libexec/gcc/arm-linux-gnueabihf/cc1plus",
> 0xfffe9014) = -1 ENOENT (No such file or directory)
> stat64("/mnt/cephfs/CentOS-6.6/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../libexec/gcc/cc1plus",
> 0xfffe9014) = -1 ENOENT (No such file or directory)
> stat64("/mnt/cephfs/CentOS-6.6/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.9.2/../../../../arm-linux-gnueabihf/bin/arm-linux-gnueabihf/4.9.2/cc1plus",
> 0xfffe9014) = -1 ENOENT (No such file or directory)
> stat64("/mnt/cephfs/CentOS-6.6/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.9.2/../../../../arm-linux-gnueabihf/bin/arm-linux-gnueabihf/cc1plus",
> 0xfffe9014) = -1 ENOENT (No such file or directory)
> stat64("/mnt/cephfs/CentOS-6.6/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.9.2/../../../../arm-linux-gnueabihf/bin/cc1plus",
> 0xfffe9014) = -1 ENOENT (No such file or directory)
> vfork(arm-linux-gnueabihf-g++: error trying to exec 'cc1plus': execvp:
> No such file or directory
> ) = 8885
> --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=8885,
> si_status=255, si_utime=0, si_stime=0} ---
> waitpid(8885, [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], 0) = 8885
> stat64("/tmp/ccIlAAE5.s", {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
> unlink("/tmp/ccIlAAE5.s") = 0
> exit_group(1) = ?
>
> It shows arm-gcc has indeed found cc1plus, but somehow it continues to
> search other directories.
>
> Below gives log for running arm-gcc under local filesystem:
>
> open("/tmp/ccU6Mkbg.s", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
> close(3) = 0
> stat64("/mnt/local/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../libexec/gcc/arm-linux-gnueabihf/4.9.2/cc1plus",
> {st_mode=S_IFREG|0755, st_size=16224744, ...}) = 0
> access("/mnt/local/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux/bin/../libexec/gcc/arm-linux-gnueabihf/4.9.2/cc1plus",
> X_OK) = 0
> vfork() = 8866
> waitpid(8866, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 8866
> --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=8866,
> si_status=0, si_utime=16, si_stime=5} ---
>
> The main difference is that when using a local filesystem, arm-gcc
> will not continue to look up in other directories after it has found
> cc1plus. I'm really confused by this behavior. MDS's log just showed
> it received some lookup requests and responded each request correctly.
>
> The /mnt/local/CentOS-6.6 is copied from /mnt/cephfs/CentOS-6.6 with
> `cp -rp` command, so symbol-link or file-mode should keep the same. I
> tried gcc in this gcc-tool directory, and it worked well under cephfs.
> Probably there is a bug in arm-gcc, but I can't imagine what code will
> lead to such strange behavior.
>
> Could anyone give some tips on this? Thanks!
One possibility is that even though the stat64 call succeeded, some of
the attributes in the returned struct stat made the program consider it
unsuitable for some reason.
I'd redo those straces with -v, longer -s value, etc., and see whether
you can discern any major differences between the corresponding stat64
calls.
--
Jeff Layton <jlayton@xxxxxxxxxx>
