Hey Gregory,
On 10-09-13 20:21, Gregory Farnum wrote:
> On Tue, Sep 10, 2013 at 10:54 AM, Oliver Daudey <oliver@xxxxxxxxx> wrote:
>> Hey list,
>>
>> I just upgraded to Ceph 0.67.3. What I did on every node of my 3-node
>> cluster was:
>> - Unmount CephFS everywhere.
>> - Upgrade the Ceph-packages.
>> - Restart MON.
>> - Restart OSD.
>> - Restart MDS.
>>
>> As soon as I got to the second node, the MDS crashed right after startup.
>>
>> Part of the logs (more on request):
>>
>> -> 194.109.43.12:6802/53419 -- osd_op(mds.0.58:4 mds_snaptable [read
>> 0~0] 1.d902
>> 70ad e37647) v4 -- ?+0 0x1e48d80 con 0x1e5d9a0
>> -11> 2013-09-10 19:35:02.798962 7fd1ba81f700 2 mds.0.58 boot_start
>> 1: openin
>> g mds log
>> -10> 2013-09-10 19:35:02.798968 7fd1ba81f700 5 mds.0.log open
>> discovering lo
>> g bounds
>> -9> 2013-09-10 19:35:02.798988 7fd1ba81f700 1 mds.0.journaler(ro)
>> recover s
>> tart
>> -8> 2013-09-10 19:35:02.798990 7fd1ba81f700 1 mds.0.journaler(ro)
>> read_head
>> -7> 2013-09-10 19:35:02.799028 7fd1ba81f700 1 --
>> 194.109.43.12:6800/67277 -
>> -> 194.109.43.11:6800/16562 -- osd_op(mds.0.58:5 200.00000000 [read 0~0]
>> 1.844f3
>> 494 e37647) v4 -- ?+0 0x1e48b40 con 0x1e5db00
>> -6> 2013-09-10 19:35:02.799053 7fd1ba81f700 1 --
>> 194.109.43.12:6800/67277 <
>> == mon.2 194.109.43.13:6789/0 16 ==== mon_subscribe_ack(300s) v1 ====
>> 20+0+0 (42
>> 35168662 0 0) 0x1e93380 con 0x1e5d580
>> -5> 2013-09-10 19:35:02.799099 7fd1ba81f700 10 monclient:
>> handle_subscribe_a
>> ck sent 2013-09-10 19:35:02.796448 renew after 2013-09-10 19:37:32.796448
>> -4> 2013-09-10 19:35:02.800907 7fd1ba81f700 5 mds.0.58
>> ms_handle_connect on
>> 194.109.43.12:6802/53419
>> -3> 2013-09-10 19:35:02.800927 7fd1ba81f700 5 mds.0.58
>> ms_handle_connect on
>> 194.109.43.13:6802/45791
>> -2> 2013-09-10 19:35:02.801176 7fd1ba81f700 5 mds.0.58
>> ms_handle_connect on
>> 194.109.43.11:6800/16562
>> -1> 2013-09-10 19:35:02.803546 7fd1ba81f700 1 --
>> 194.109.43.12:6800/67277 <
>> == osd.2 194.109.43.13:6802/45791 1 ==== osd_op_reply(3 mds_anchortable
>> [read 0~
>> 0] ack = -2 (No such file or directory)) v4 ==== 114+0+0 (3107677671 0
>> 0) 0x1e4d
>> e00 con 0x1e5ddc0
>> 0> 2013-09-10 19:35:02.805611 7fd1ba81f700 -1 mds/MDSTable.cc: In
>> function
>> 'void MDSTable::load_2(int, ceph::bufferlist&, Context*)' thread
>> 7fd1ba81f700 ti
>> me 2013-09-10 19:35:02.803673
>> mds/MDSTable.cc: 152: FAILED assert(r >= 0)
>>
>> ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a)
>> 1: (MDSTable::load_2(int, ceph::buffer::list&, Context*)+0x44f) [0x77ce7f]
>> 2: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe3b) [0x7d891b]
>> 3: (MDS::handle_core_message(Message*)+0x987) [0x56f527]
>> 4: (MDS::_dispatch(Message*)+0x2f) [0x56f5ef]
>> 5: (MDS::ms_dispatch(Message*)+0x19b) [0x5710bb]
>> 6: (DispatchQueue::entry()+0x592) [0x92e432]
>> 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x8a59bd]
>> 8: (()+0x68ca) [0x7fd1bed298ca]
>> 9: (clone()+0x6d) [0x7fd1bda5cb6d]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> When trying to mount CephFS, it just hangs now. Sometimes, an MDS stays
>> up for a while, but will eventually crash again. This CephFS was
>> created on 0.67 and I haven't done anything but mount and use it under
>> very light load in the mean time.
>>
>> Any ideas, or if you need more info, let me know. It would be nice to
>> get my data back, but I have backups too.
>
> Does the filesystem have any data in it? Every time we've seen this
> error it's been on an empty cluster which had some weird issue with
> startup.
This one certainly had some data on it, yes. A couple of 100's of GBs
of disk-images and a couple of trees of smaller files. Most of them
accessed very rarely since being copied on.
Regards,
Oliver
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
