osd was crashing on start - journal flush

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I had a problem with an osd starting - log seemed to show the journal was a problem. When I tried to flush the journal I got the errors below.

 

I was in a hurry so attached a spare ssd partion as a new journal, which fixed the problem and let it heal.

 

To fix it for the original ssd journal should I have cleared the ssd partition using dd?

 

errors:

 

 

ceph-osd -i 1 --flush-journal

2014-12-01 01:16:05.387607 7f4133d18780 -1 filestore(/var/lib/ceph/osd/ceph-1) FileStore::_setattrs: chain_setxattr returned -14

os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f4133d18780 time 2014-12-01 01:16:05.387752

os/FileStore.cc: 2559: FAILED assert(0 == "unexpected error")

ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)

1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x9ff) [0x9c63ff]

2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x6c) [0x9c9efc]

3: (JournalingObjectStore::journal_replay(unsigned long)+0x985) [0x9dfdc5]

4: (FileStore::mount()+0x32dd) [0x9b654d]

5: (main()+0xe20) [0x731dc0]

6: (__libc_start_main()+0xfd) [0x7f4131db8ead]

7: ceph-osd() [0x736ea9]

NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2014-12-01 01:16:05.390458 7f4133d18780 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f4133d18780 time 2014-12-01 01:16:05.387752

os/FileStore.cc: 2559: FAILED assert(0 == "unexpected error")

 

ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)

1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x9ff) [0x9c63ff]

2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x6c) [0x9c9efc]

3: (JournalingObjectStore::journal_replay(unsigned long)+0x985) [0x9dfdc5]

4: (FileStore::mount()+0x32dd) [0x9b654d]

5: (main()+0xe20) [0x731dc0]

6: (__libc_start_main()+0xfd) [0x7f4131db8ead]

7: ceph-osd() [0x736ea9]

NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

 

-4> 2014-12-01 01:16:05.387607 7f4133d18780 -1 filestore(/var/lib/ceph/osd/ceph-1) FileStore::_setattrs: chain_setxattr returned -14

0> 2014-12-01 01:16:05.390458 7f4133d18780 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7f4133d18780 time 2014-12-01 01:16:05.387752

os/FileStore.cc: 2559: FAILED assert(0 == "unexpected error")

 

ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)

1: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x9ff) [0x9c63ff]

2: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x6c) [0x9c9efc]

3: (JournalingObjectStore::journal_replay(unsigned long)+0x985) [0x9dfdc5]

4: (FileStore::mount()+0x32dd) [0x9b654d]

5: (main()+0xe20) [0x731dc0]

6: (__libc_start_main()+0xfd) [0x7f4131db8ead]

7: ceph-osd() [0x736ea9]

NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

 

terminate called after throwing an instance of 'ceph::FailedAssertion'

*** Caught signal (Aborted) **

in thread 7f4133d18780

ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)

1: ceph-osd() [0xab54a2]

2: (()+0xf0a0) [0x7f413325b0a0]

3: (gsignal()+0x35) [0x7f4131dcc165]

4: (abort()+0x180) [0x7f4131dcf3e0]

5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f413262389d]

6: (()+0x63996) [0x7f4132621996]

7: (()+0x639c3) [0x7f41326219c3]

8: (()+0x63bee) [0x7f4132621bee]

9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0xb8f97a]

10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x9ff) [0x9c63ff]

11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x6c) [0x9c9efc]

12: (JournalingObjectStore::journal_replay(unsigned long)+0x985) [0x9dfdc5]

13: (FileStore::mount()+0x32dd) [0x9b654d]

14: (main()+0xe20) [0x731dc0]

15: (__libc_start_main()+0xfd) [0x7f4131db8ead]

16: ceph-osd() [0x736ea9]

2014-12-01 01:16:05.394614 7f4133d18780 -1 *** Caught signal (Aborted) **

in thread 7f4133d18780

 

ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)

1: ceph-osd() [0xab54a2]

2: (()+0xf0a0) [0x7f413325b0a0]

3: (gsignal()+0x35) [0x7f4131dcc165]

4: (abort()+0x180) [0x7f4131dcf3e0]

5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f413262389d]

6: (()+0x63996) [0x7f4132621996]

7: (()+0x639c3) [0x7f41326219c3]

8: (()+0x63bee) [0x7f4132621bee]

9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0xb8f97a]

10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x9ff) [0x9c63ff]

11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x6c) [0x9c9efc]

12: (JournalingObjectStore::journal_replay(unsigned long)+0x985) [0x9dfdc5]

13: (FileStore::mount()+0x32dd) [0x9b654d]

14: (main()+0xe20) [0x731dc0]

15: (__libc_start_main()+0xfd) [0x7f4131db8ead]

16: ceph-osd() [0x736ea9]

NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

 

0> 2014-12-01 01:16:05.394614 7f4133d18780 -1 *** Caught signal (Aborted) **

in thread 7f4133d18780

 

ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)

1: ceph-osd() [0xab54a2]

2: (()+0xf0a0) [0x7f413325b0a0]

3: (gsignal()+0x35) [0x7f4131dcc165]

4: (abort()+0x180) [0x7f4131dcf3e0]

5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f413262389d]

6: (()+0x63996) [0x7f4132621996]

7: (()+0x639c3) [0x7f41326219c3]

8: (()+0x63bee) [0x7f4132621bee]

9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0xb8f97a]

10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x9ff) [0x9c63ff]

11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long, ThreadPool::TPHandle*)+0x6c) [0x9c9efc]

12: (JournalingObjectStore::journal_replay(unsigned long)+0x985) [0x9dfdc5]

13: (FileStore::mount()+0x32dd) [0x9b654d]

14: (main()+0xe20) [0x731dc0]

15: (__libc_start_main()+0xfd) [0x7f4131db8ead]

16: ceph-osd() [0x736ea9]

NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

 

 

 

--

Lindsay

Attachment: signature.asc
Description: This is a digitally signed message part.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux