Re: Ceph mon cannot join to cluster during upgrade

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



There are a couple of threads wrt oom killers but I'm not sure if they apply here. Do you have any custom configs in Mimic which may not be applicable in Nautilus? The cluster is/was otherwise healthy (I assume it was)? Maybe you could try to add new MON temporarily to see if that successfully starts and joins the cluster?


Zitat von Iban Cabrillo <cabrillo@xxxxxxxxxxxxxx>:

Hi Eugen,
There is only ceph-mgr and ceph-mon on this node (working fine for years with versions <14)

Jun 29 16:08:42 cephmon03 systemd: ceph-mon@cephmon03.service failed.
Jun 29 16:16:36 cephmon03 kernel: ceph-mon[7498]: segfault at 8 ip 00007fa4c2e75ed7 sp 00007ffee88e3730 error 4 in libceph-common.so.0[7fa4c2b97000+9b8000]
Jun 29 16:19:04 cephmon03 systemd: Reloading.
Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mgr@.service:15] Unknown lvalue 'LockPersonality' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mgr@.service:18] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mgr@.service:21] Unknown lvalue 'ProtectControlGroups' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mgr@.service:23] Unknown lvalue 'ProtectKernelModules' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mgr@.service:24] Unknown lvalue 'ProtectKernelTunables' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mon@.service:19] Unknown lvalue 'LockPersonality' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mon@.service:21] Unknown lvalue 'MemoryDenyWriteExecute' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mon@.service:25] Unknown lvalue 'ProtectControlGroups' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mon@.service:27] Unknown lvalue 'ProtectKernelModules' in section 'Service' Jun 29 16:19:05 cephmon03 systemd: [/usr/lib/systemd/system/ceph-mon@.service:28] Unknown lvalue 'ProtectKernelTunables' in section 'Service'

I thought this was related to this bug https://tracker.ceph.com/issues/50997, the the trick didn't work for me. #MemoryDenyWriteExecute=true MemoryDenyWriteExecute=false

running manualy:
/usr/bin/ceph-mon -f --cluster ceph --id cephmon03 --setuser ceph --setgroup ceph

I see the ceph-mon procc start to consume whole swap

src/central_freelist.cc:333] tcmalloc: allocation failed 8192 *** Caught signal (Segmentation fault) ** in thread 7fb892cfd1c0 thread_name:ceph-mon ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (()+0xf630) [0x7fb886d2d630] 2: (std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)+0x59) [0x55ea4086f709] 3: (std::string::_M_mutate(unsigned long, unsigned long, unsigned long)+0x6b) [0x55ea40870dcb] 4: (std::string::assign(char const*, unsigned long)+0x55) [0x55ea40870fb5] 5: (()+0x2aa5c) [0x7fb88949ea5c]
......
 Mons are under Centos7.5 machine

Regards, I



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux