Re: Nfs-ganesha 2.6 packages in ceph repo

[David C <dcsysengineer@xxxxxxxxx>]

Hi Oliver

Thanks for the detailed reponse! I've downgraded my libcephfs2 to 12.2.4 and still get a similar error:

load_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalceph.so Error:/lib64/libcephfs.so.2: undefined symbol: _Z14common_preinitRK18CephInitParameters18code_environment_ti
load_fsal :NFS STARTUP :MAJ :Failed to load module (/usr/lib64/ganesha/libfsalceph.so) because: Can not access a needed shared library

I'm on CentOS 7.4, using the following package versions:

# rpm -qa | grep ganesha
nfs-ganesha-2.6.1-0.1.el7.x86_64
nfs-ganesha-vfs-2.6.1-0.1.el7.x86_64
nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64

# rpm -qa | grep ceph
libcephfs2-12.2.4-0.el7.x86_64
nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64

I don't have the ceph user space components installed, assuming they're not nesscary apart from libcephfs2? Any idea why it's giving me this error?

Thanks,

On Fri, May 11, 2018 at 2:17 AM, Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx> wrote:

Hi David,

for what it's worth, we are running with nfs-ganesha 2.6.1 from Ceph repos on CentOS 7.4 with the following set of versions:
libcephfs2-12.2.4-0.el7.x86_64
nfs-ganesha-2.6.1-0.1.el7.x86_64
nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64
Of course, we plan to upgrade to 12.2.5 soon-ish...

Am 11.05.2018 um 00:05 schrieb David C:
> Hi All
>
> I'm testing out the nfs-ganesha-2.6.1-0.1.el7.x86_64.rpm package from http://download.ceph.com/nfs-ganesha/rpm-V2.6-stable/luminous/x86_64/
>
> It's failing to load /usr/lib64/ganesha/libfsalceph.so
>
> With libcephfs-12.2.1 installed I get the following error in my ganesha log:
>
>     load_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalceph.so Error:
>     /usr/lib64/ganesha/libfsalceph.so: undefined symbol: ceph_set_deleg_timeout
>     load_fsal :NFS STARTUP :MAJ :Failed to load module (/usr/lib64/ganesha/libfsalceph.so) because
>     : Can not access a needed shared library

That looks like an ABI incompatibility, probably the nfs-ganesha packages should block this libcephfs2-version (and older ones).

>
>
> With libcephfs-12.2.5 installed I get:
>
>     load_fsal :NFS STARTUP :CRIT :Could not dlopen module:/usr/lib64/ganesha/libfsalceph.so Error:
>     /lib64/libcephfs.so.2: undefined symbol: _ZNK5FSMap10parse_roleEN5boost17basic_string_viewIcSt11char_traitsIcEEEP10mds_role_tRSo
>     load_fsal :NFS STARTUP :MAJ :Failed to load module (/usr/lib64/ganesha/libfsalceph.so) because
>     : Can not access a needed shared library

That looks ugly and makes me fear for our planned 12.2.5-upgrade.
Interestingly, we do not have that symbol on 12.2.4:
# nm -D /lib64/libcephfs.so.2 | grep FSMap
                 U _ZNK5FSMap10parse_roleERKSsP10mds_role_tRSo
                 U _ZNK5FSMap13print_summaryEPN4ceph9FormatterEPSo
and NFS-Ganesha works fine.

Looking at:
https://github.com/ceph/ceph/blob/v12.2.4/src/mds/FSMap.h
versus
https://github.com/ceph/ceph/blob/v12.2.5/src/mds/FSMap.h
it seems this commit:
https://github.com/ceph/ceph/commit/7d8b3c1082b6b870710989773f3cd98a472b9a3d
changed libcephfs2 ABI.

I've no idea how that's usually handled and whether ABI breakage should occur within point releases (I would not have expected that...).
At least, this means either:
- ABI needs to be reverted to the old state.
- A new NFS Ganesha build is needed. Probably, if this is a common thing, builds should be automated and be synchronized to ceph releases,
  and old versions should be kept around.

I'll hold back our update to 12.2.5 until this is resolved, so many thanks from my side!

Let's see who jumps in to resolve it...

Cheers,
        Oliver
>
>
> My cluster is running 12.2.1
>
> All package versions:
>
> nfs-ganesha-2.6.1-0.1.el7.x86_64
> nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64
> libcephfs2-12.2.5-0.el7.x86_64
>
> Can anyone point me in the right direction?
>
> Thanks,
> David
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com