Re: my cluster is down after upgrade to 10.1.2

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Am 2016-04-14 16:05, schrieb Lomayani S. Laizer:
Hello,

I upgraded from 10.1.0 to 10.1.2 with ceph-deploy and my cluster is
down now. getting below errors

ceph -s

2016-04-14 17:04:58.909894 7f14686e4700  0 -- :/2590574876 >>
10.10.200.4:6789/0 [1] pipe(0x7f146405adf0 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7f146405c0b0).fault
2016-04-14 17:05:01.909949 7f14685e3700  0 -- :/2590574876 >>
10.10.200.3:6789/0 [2] pipe(0x7f1458000c80 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7f1458001f90).fault
2016-04-14 17:05:04.910416 7f14686e4700  0 -- :/2590574876 >>
10.10.200.4:6789/0 [1] pipe(0x7f1458005120 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7f14580063e0).fault
2016-04-14 17:05:07.910697 7f14685e3700  0 -- :/2590574876 >>
10.10.200.2:6789/0 [3] pipe(0x7f1458000c80 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7f1458002410).fault

--

Lomayani



Links:
------
[1] http://10.10.200.4:6789/0
[2] http://10.10.200.3:6789/0
[3] http://10.10.200.2:6789/0

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Hi Lomayani and other cephers,

i have the same issue - thankfully i am playing around with our test cluster.

this is what we get:

terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7fe2370a24c0 thread_name:ceph-mon
 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x4f3712) [0x55b2ed4b7712]
 2: (()+0x10340) [0x7fe2363b1340]
 3: (gsignal()+0x39) [0x7fe234639cc9]
 4: (abort()+0x148) [0x7fe23463d0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fe234f44535]
 6: (()+0x5e6d6) [0x7fe234f426d6]
 7: (()+0x5e703) [0x7fe234f42703]
 8: (()+0x5e922) [0x7fe234f42922]
 9: (()+0x618f15) [0x55b2ed5dcf15]
10: (FSMap::decode(ceph::buffer::list::iterator&)+0x101f) [0x55b2ed4faebf]
 11: (MDSMonitor::update_from_paxos(bool*)+0x178) [0x55b2ed321738]
 12: (PaxosService::refresh(bool*)+0x19a) [0x55b2ed2958da]
 13: (Monitor::refresh_from_paxos(bool*)+0x143) [0x55b2ed232643]
 14: (Monitor::init_paxos()+0x85) [0x55b2ed232a55]
 15: (Monitor::preinit()+0x925) [0x55b2ed242505]
 16: (main()+0x236d) [0x55b2ed1d10ed]
 17: (__libc_start_main()+0xf5) [0x7fe234624ec5]
 18: (()+0x25f28a) [0x55b2ed22328a]
2016-04-14 16:19:30.301995 7fe2370a24c0 -1 *** Caught signal (Aborted) **
 in thread 7fe2370a24c0 thread_name:ceph-mon

 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x4f3712) [0x55b2ed4b7712]
 2: (()+0x10340) [0x7fe2363b1340]
 3: (gsignal()+0x39) [0x7fe234639cc9]
 4: (abort()+0x148) [0x7fe23463d0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fe234f44535]
 6: (()+0x5e6d6) [0x7fe234f426d6]
 7: (()+0x5e703) [0x7fe234f42703]
 8: (()+0x5e922) [0x7fe234f42922]
 9: (()+0x618f15) [0x55b2ed5dcf15]
10: (FSMap::decode(ceph::buffer::list::iterator&)+0x101f) [0x55b2ed4faebf]
 11: (MDSMonitor::update_from_paxos(bool*)+0x178) [0x55b2ed321738]
 12: (PaxosService::refresh(bool*)+0x19a) [0x55b2ed2958da]
 13: (Monitor::refresh_from_paxos(bool*)+0x143) [0x55b2ed232643]
 14: (Monitor::init_paxos()+0x85) [0x55b2ed232a55]
 15: (Monitor::preinit()+0x925) [0x55b2ed242505]
 16: (main()+0x236d) [0x55b2ed1d10ed]
 17: (__libc_start_main()+0xf5) [0x7fe234624ec5]
 18: (()+0x25f28a) [0x55b2ed22328a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

0> 2016-04-14 16:19:30.301995 7fe2370a24c0 -1 *** Caught signal (Aborted) **
 in thread 7fe2370a24c0 thread_name:ceph-mon

 ceph version 10.1.2 (4a2a6f72640d6b74a3bbd92798bb913ed380dcd4)
 1: (()+0x4f3712) [0x55b2ed4b7712]
 2: (()+0x10340) [0x7fe2363b1340]
 3: (gsignal()+0x39) [0x7fe234639cc9]
 4: (abort()+0x148) [0x7fe23463d0d8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fe234f44535]
 6: (()+0x5e6d6) [0x7fe234f426d6]
 7: (()+0x5e703) [0x7fe234f42703]
 8: (()+0x5e922) [0x7fe234f42922]
 9: (()+0x618f15) [0x55b2ed5dcf15]
10: (FSMap::decode(ceph::buffer::list::iterator&)+0x101f) [0x55b2ed4faebf]
 11: (MDSMonitor::update_from_paxos(bool*)+0x178) [0x55b2ed321738]
 12: (PaxosService::refresh(bool*)+0x19a) [0x55b2ed2958da]
 13: (Monitor::refresh_from_paxos(bool*)+0x143) [0x55b2ed232643]
 14: (Monitor::init_paxos()+0x85) [0x55b2ed232a55]
 15: (Monitor::preinit()+0x925) [0x55b2ed242505]
 16: (main()+0x236d) [0x55b2ed1d10ed]
 17: (__libc_start_main()+0xf5) [0x7fe234624ec5]
 18: (()+0x25f28a) [0x55b2ed22328a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.


*ceph.conf*
[global]
#enable experimental unrecoverable data corrupting features = *
fsid = xxxxxx-xxxxxx-xxxxxx-xxxx-xxxxxx
public_network = 172.xxx.xx.x/xx
cluster_network = 10.xxx.xx.x/xx
#mon_initial_members = srv1, srv2, srv3
#mon_host = 172.xxx.xx.1,172.xxx.xx.2,172.xxx.xx.3
mon_initial_members = mon1,mon2,mon3
mon_host = 172.xxx.xx.118,172.xxx.xx.119,172.xxx.xx.120
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

#enable_experimental_unrecoverable_data_corrupting_features = *

#osd_objectstore = bluestore

[osd]
osd mount options xfs = "rw,noatime,nobarrier,inode64"

[client.radosgw.mon1]
host = mon1
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /tmp/ceph.radosgw.gateway.fastcgi.sock
log file = /tmp/client.radosgw.gateway.log
rgw print continue = false


*mon3*
today i planned to add this as our third mon to the testcluster. But this didn't worked for the same reason "described" above. So i thought there is perhaps an issue with the server "mon3" and give it a try on *mon2*. Now out Testcluster has only 1 active mon and stopped all activity (as expected).


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux