AH! Sorry for the false alarm, I clearly have a hard drive problem....
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x24
ata2.00: failed command: READ DMA
ata2.00: cmd c8/00:08:38:bd:70/00:00:00:00:00/ef tag 0 dma 4096 in
res 51/40:00:3f:bd:70/40:00:21:00:00/ef Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x24
ata2.00: failed command: READ DMA
ata2.00: cmd c8/00:08:38:bd:70/00:00:00:00:00/ef tag 0 dma 4096 in
res 51/40:00:3f:bd:70/40:00:21:00:00/ef Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x24
ata2.00: failed command: READ DMA
ata2.00: cmd c8/00:08:38:bd:70/00:00:00:00:00/ef tag 0 dma 4096 in
res 51/40:00:3f:bd:70/40:00:21:00:00/ef Emask 0x9 (media error)
ata2.00: status: { DRDY ERR }
ata2.00: error: { UNC }
ata2.00: configured for UDMA/133
ata2: EH complete
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata2.00: BMDMA stat 0x24
ata2.00: failed command: READ DMA
On Thu, Nov 13, 2014 at 9:28 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote:
Hmm, looks like leveldb is hitting a problem. Is there anything in the
kernel log (dmesg) that suggests a disk or file system problem? Are you
able to, say, tar up the current/omap directory without problems?
This is a single OSD, right? None of the others have been upgraded yet?
sage
On Thu, 13 Nov 2014, Joshua McClintock wrote:
>
> [root@ceph-node20 ~]# ls /var/lib/ceph/osd/us-west01-0/current
>
> 0.10_head 0.1a_head 0.23_head 0.2c_head 0.37_head 0.3_head 0.b_head
> 1.10_head 1.1b_head 1.24_head 1.2c_head 1.3a_head 1.b_head 2.16_head
> 2.1_head 2.2a_head 2.32_head 2.3a_head 2.a_head omap
>
> 0.11_head 0.1d_head 0.25_head 0.2e_head 0.38_head 0.4_head 0.c_head
> 1.13_head 1.1d_head 1.26_head 1.2f_head 1.3b_head 1.e_head 2.1a_head
> 2.22_head 2.2c_head 2.33_head 2.3e_head 2.b_head
>
> 0.13_head 0.1f_head 0.26_head 0.2f_head 0.3b_head 0.5_head 0.d_head
> 1.16_head 1.1f_head 1.27_head 1.31_head 1.3e_head 2.0_head 2.1b_head
> 2.25_head 2.2e_head 2.36_head 2.3f_head 2.c_head
>
> 0.16_head 0.20_head 0.27_head 0.30_head 0.3c_head 0.6_head 0.e_head
> 1.18_head 1.20_head 1.29_head 1.36_head 1.3_head 2.10_head 2.1c_head
> 2.26_head 2.2f_head 2.37_head 2.4_head commit_op_seq
>
> 0.18_head 0.21_head 0.28_head 0.33_head 0.3e_head 0.7_head 0.f_head
> 1.19_head 1.22_head 1.2a_head 1.37_head 1.4_head 2.11_head 2.1d_head
> 2.27_head 2.30_head 2.38_head 2.7_head meta
>
> 0.19_head 0.22_head 0.29_head 0.35_head 0.3f_head 0.9_head 1.0_head
> 1.1a_head 1.23_head 1.2b_head 1.39_head 1.a_head 2.12_head 2.1e_head
> 2.28_head 2.31_head 2.39_head 2.8_head nosnap
>
>
> The output from the other command was too long to post, here's the link to
> the full dump:
>
> http://pastee.co/Kd1BlP
>
> Here's the last 100-200 lines:
>
> ...
>
> ...
>
> ...
>
> ...
>
> _HOBJTOSEQ_:pglog%u2%e2e...0.none.516B9E4C
>
> _HOBJTOSEQ_:pglog%u2%e2f...0.none.516B9F1C
>
> _HOBJTOSEQ_:pglog%u2%e30...0.none.516BFD4B
>
> _HOBJTOSEQ_:pglog%u2%e31...0.none.516BF21B
>
> _HOBJTOSEQ_:pglog%u2%e32...0.none.516BF3AB
>
> _HOBJTOSEQ_:pglog%u2%e33...0.none.516BF37B
>
> _HOBJTOSEQ_:pglog%u2%e36...0.none.516BF16B
>
> _HOBJTOSEQ_:pglog%u2%e37...0.none.516BF63B
>
> _HOBJTOSEQ_:pglog%u2%e38...0.none.516BF7CB
>
> _HOBJTOSEQ_:pglog%u2%e39...0.none.516BF49B
>
> _HOBJTOSEQ_:pglog%u2%e3a...0.none.516B933C
>
> _HOBJTOSEQ_:pglog%u2%e3e...0.none.516B96FC
>
> _HOBJTOSEQ_:pglog%u2%e3f...0.none.516B978C
>
> _HOBJTOSEQ_:pglog%u2%e4...0.none.103ABD8E
>
> _HOBJTOSEQ_:pglog%u2%e7...0.none.103AB3BE
>
> _HOBJTOSEQ_:pglog%u2%e8...0.none.103AB34E
>
> _HOBJTOSEQ_:pglog%u2%ea...0.none.103A5CBF
>
> _HOBJTOSEQ_:pglog%u2%eb...0.none.103A5C4F
>
> _HOBJTOSEQ_:pglog%u2%ec...0.none.103A5D1F
>
> _SYS_:HEADER
>
> *** Caught signal (Bus error) **
>
> in thread 7f64e92ce760
>
> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
>
> 1: ceph-kvstore-tool() [0x4bf2e1]
>
> 2: (()+0xf710) [0x7f64e86e0710]
>
> 3: (leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions
> const&, leveldb::BlockHandle const&, leveldb::BlockContents*)+0x1cb)
> [0x7f64e8e9f73b]
>
> 4: (leveldb::Table::BlockReader(void*, leveldb::ReadOptions const&,
> leveldb::Slice const&)+0x291) [0x7f64e8ea0de1]
>
> 5: (()+0x3a412) [0x7f64e8ea3412]
>
> 6: (()+0x3a6f8) [0x7f64e8ea36f8]
>
> 7: (()+0x3a78d) [0x7f64e8ea378d]
>
> 8: (()+0x3761a) [0x7f64e8ea061a]
>
> 9: (()+0x20fd2) [0x7f64e8e89fd2]
>
> 10: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::next()+0x47) [0x4ba417]
>
> 11: (StoreTool::traverse(std::string const&, bool, std::ostream*)+0x1da)
> [0x4b65fa]
>
> 12: (main()+0x2cc) [0x4b26fc]
>
> 13: (__libc_start_main()+0xfd) [0x7f64e77a2d1d]
>
> 14: ceph-kvstore-tool() [0x4b21b9]
>
> 2014-11-13 21:19:18.318941 7f64e92ce760 -1 *** Caught signal (Bus error) **
>
> in thread 7f64e92ce760
>
>
> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
>
> 1: ceph-kvstore-tool() [0x4bf2e1]
>
> 2: (()+0xf710) [0x7f64e86e0710]
>
> 3: (leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions
> const&, leveldb::BlockHandle const&, leveldb::BlockContents*)+0x1cb)
> [0x7f64e8e9f73b]
>
> 4: (leveldb::Table::BlockReader(void*, leveldb::ReadOptions const&,
> leveldb::Slice const&)+0x291) [0x7f64e8ea0de1]
>
> 5: (()+0x3a412) [0x7f64e8ea3412]
>
> 6: (()+0x3a6f8) [0x7f64e8ea36f8]
>
> 7: (()+0x3a78d) [0x7f64e8ea378d]
>
> 8: (()+0x3761a) [0x7f64e8ea061a]
>
> 9: (()+0x20fd2) [0x7f64e8e89fd2]
>
> 10: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::next()+0x47) [0x4ba417]
>
> 11: (StoreTool::traverse(std::string const&, bool, std::ostream*)+0x1da)
> [0x4b65fa]
>
> 12: (main()+0x2cc) [0x4b26fc]
>
> 13: (__libc_start_main()+0xfd) [0x7f64e77a2d1d]
>
> 14: ceph-kvstore-tool() [0x4b21b9]
>
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
>
>
> --- begin dump of recent events ---
>
> -13> 2014-11-13 21:19:04.689722 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command perfcounters_dump hook 0x1f1b510
>
> -12> 2014-11-13 21:19:04.689754 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command 1 hook 0x1f1b510
>
> -11> 2014-11-13 21:19:04.689771 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command perf dump hook 0x1f1b510
>
> -10> 2014-11-13 21:19:04.689778 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command perfcounters_schema hook 0x1f1b510
>
> -9> 2014-11-13 21:19:04.689787 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command 2 hook 0x1f1b510
>
> -8> 2014-11-13 21:19:04.689793 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command perf schema hook 0x1f1b510
>
> -7> 2014-11-13 21:19:04.689803 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command config show hook 0x1f1b510
>
> -6> 2014-11-13 21:19:04.689811 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command config set hook 0x1f1b510
>
> -5> 2014-11-13 21:19:04.689818 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command config get hook 0x1f1b510
>
> -4> 2014-11-13 21:19:04.689821 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command log flush hook 0x1f1b510
>
> -3> 2014-11-13 21:19:04.689831 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command log dump hook 0x1f1b510
>
> -2> 2014-11-13 21:19:04.689837 7f64e92ce760 5 asok(0x1f1b5b0)
> register_command log reopen hook 0x1f1b510
>
> -1> 2014-11-13 21:19:04.689940 7f64e92ce760 -1 did not load config file,
> using default settings.
>
> 0> 2014-11-13 21:19:18.318941 7f64e92ce760 -1 *** Caught signal (Bus
> error) **
>
> in thread 7f64e92ce760
>
>
> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
>
> 1: ceph-kvstore-tool() [0x4bf2e1]
>
> 2: (()+0xf710) [0x7f64e86e0710]
>
> 3: (leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions
> const&, leveldb::BlockHandle const&, leveldb::BlockContents*)+0x1cb)
> [0x7f64e8e9f73b]
>
> 4: (leveldb::Table::BlockReader(void*, leveldb::ReadOptions const&,
> leveldb::Slice const&)+0x291) [0x7f64e8ea0de1]
>
> 5: (()+0x3a412) [0x7f64e8ea3412]
>
> 6: (()+0x3a6f8) [0x7f64e8ea36f8]
>
> 7: (()+0x3a78d) [0x7f64e8ea378d]
>
> 8: (()+0x3761a) [0x7f64e8ea061a]
>
> 9: (()+0x20fd2) [0x7f64e8e89fd2]
>
> 10: (LevelDBStore::LevelDBWholeSpaceIteratorImpl::next()+0x47) [0x4ba417]
>
> 11: (StoreTool::traverse(std::string const&, bool, std::ostream*)+0x1da)
> [0x4b65fa]
>
> 12: (main()+0x2cc) [0x4b26fc]
>
> 13: (__libc_start_main()+0xfd) [0x7f64e77a2d1d]
>
> 14: ceph-kvstore-tool() [0x4b21b9]
>
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
> interpret this.
>
>
> --- logging levels ---
>
> 0/ 5 none
>
> 0/ 1 lockdep
>
> 0/ 1 context
>
> 1/ 1 crush
>
> 1/ 5 mds
>
> 1/ 5 mds_balancer
>
> 1/ 5 mds_locker
>
> 1/ 5 mds_log
>
> 1/ 5 mds_log_expire
>
> 1/ 5 mds_migrator
>
> 0/ 1 buffer
>
> 0/ 1 timer
>
> 0/ 1 filer
>
> 0/ 1 striper
>
> 0/ 1 objecter
>
> 0/ 5 rados
>
> 0/ 5 rbd
>
> 0/ 5 journaler
>
> 0/ 5 objectcacher
>
> 0/ 5 client
>
> 0/ 5 osd
>
> 0/ 5 optracker
>
> 0/ 5 objclass
>
> 1/ 3 filestore
>
> 1/ 3 keyvaluestore
>
> 1/ 3 journal
>
> 0/ 5 ms
>
> 1/ 5 mon
>
> 0/10 monc
>
> 1/ 5 paxos
>
> 0/ 5 tp
>
> 1/ 5 auth
>
> 1/ 5 crypto
>
> 1/ 1 finisher
>
> 1/ 5 heartbeatmap
>
> 1/ 5 perfcounter
>
> 1/ 5 rgw
>
> 1/ 5 javaclient
>
> 1/ 5 asok
>
> 1/ 1 throttle
>
> -2/-2 (syslog threshold)
>
> 99/99 (stderr threshold)
>
> max_recent 500
>
> max_new 1000
>
> log_file
>
> --- end dump of recent events ---
>
> Bus error
>
>
>
> Joshua
>
>
>
> On Thu, Nov 13, 2014 at 8:52 PM, Sage Weil <sage@xxxxxxxxxxxx> wrote:
> On Thu, 13 Nov 2014, Joshua McClintock wrote:
> > I upgraded my mons to the latest version and they appear to
> work, I then
> > upgraded my mds and it seems fine.
> > I then upgraded one OSD node and the OSD fails to start with
> the following
> > dump, any help is appreciated:
> >
> > --- begin dump of recent events ---
> >
> > 0> 2014-11-13 18:20:15.625793 7fbd973ce7a0 -1 *** Caught
> signal
> > (Aborted) **
> >
> > in thread 7fbd973ce7a0
> >
> >
> > ceph version 0.80.7
> (6c0127fcb58008793d3c8b62d925bc91963672a3)
> > 1: /usr/bin/ceph-osd() [0x9bd2a1]
> > 2: (()+0xf710) [0x7fbd96373710]
> > 3: (gsignal()+0x35) [0x7fbd95245925]
> > 4: (abort()+0x175) [0x7fbd95247105]
> > 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d)
> [0x7fbd95affa5d]
> > 6: (()+0xbcbe6) [0x7fbd95afdbe6]
> > 7: (()+0xbcc13) [0x7fbd95afdc13]
> > 8: (()+0xbcd0e) [0x7fbd95afdd0e]
> > 9: (ceph::__ceph_assert_fail(char const*, char const*, int,
> char
> > const*)+0x7f2) [0xafbe22]
> > 10: (PG::peek_map_epoch(ObjectStore*, coll_t, hobject_t&,
> > ceph::buffer::list*)+0x4ea) [0x7f729a]
> > 11: (OSD::load_pgs()+0x18f1) [0x64f2b1]
> > 12: (OSD::init()+0x22c0) [0x6536f0]
> > 13: (main()+0x35bc) [0x5fe39c]
> > 14: (__libc_start_main()+0xfd) [0x7fbd95231d1d]
> > 15: /usr/bin/ceph-osd() [0x5f9e49]
> > NOTE: a copy of the executable, or `objdump -rdS
> <executable>` is needed to
> > interpret this.
>
> Hey, this looks like a different report we saw recently off-list! In
> that
> case, they were upgrading from 0.80.4 to 0.80.7. Opening #10105.
>
> You only restarting a single OSD? I would hold off on restarting any
> more
> for the time being.
>
> Can you attach the output from
>
> ls /var/lib/ceph/osd/ceph-NNN/current
> and
> ceph-kvstore-tool /var/lib/ceph/osd/ceph-NNN/current/omap list
>
> (you may need to install the ceph-tests rpm to get ceph-kvstore-tool).
>
> Thanks!
> sage
>
>
> >
> >
> > --- logging levels ---
> >
> > 0/ 5 none
> >
> > 0/ 1 lockdep
> >
> > 0/ 1 context
> >
> > 1/ 1 crush
> >
> > 1/ 5 mds
> >
> > 1/ 5 mds_balancer
> >
> > 1/ 5 mds_locker
> >
> > 1/ 5 mds_log
> >
> > 1/ 5 mds_log_expire
> >
> > 1/ 5 mds_migrator
> >
> > 0/ 1 buffer
> >
> > 0/ 1 timer
> >
> > 0/ 1 filer
> >
> > 0/ 1 striper
> >
> > 0/ 1 objecter
> >
> > 0/ 5 rados
> >
> > 0/ 5 rbd
> >
> > 0/ 5 journaler
> >
> > 0/ 5 objectcacher
> >
> > 0/ 5 client
> >
> > 0/ 5 osd
> >
> > 0/ 5 optracker
> >
> > 0/ 5 objclass
> >
> > 1/ 3 filestore
> >
> > 1/ 3 keyvaluestore
> >
> > 1/ 3 journal
> >
> > 0/ 5 ms
> >
> > 1/ 5 mon
> >
> > 0/10 monc
> >
> > 1/ 5 paxos
> >
> > 0/ 5 tp
> >
> > 1/ 5 auth
> >
> > 1/ 5 crypto
> >
> > 1/ 1 finisher
> >
> > 1/ 5 heartbeatmap
> >
> > 1/ 5 perfcounter
> >
> > 1/ 5 rgw
> >
> > 1/ 5 javaclient
> >
> > 1/ 5 asok
> >
> > 1/ 1 throttle
> >
> > -2/-2 (syslog threshold)
> >
> > -1/-1 (stderr threshold)
> >
> > max_recent 10000
> >
> > max_new 1000
> >
> > log_file /var/log/ceph/us-west01-osd.0.log
> >
> > --- end dump of recent events ---
> >
> >
> >
>
>
>
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
