On Thu, Oct 24, 2013 at 6:44 AM, Michael <michael@xxxxxxxxxxxxxxxxxx> wrote:
> Tying to gather some more info.
>
> CentOS - hanging ls
> [root@srv ~]# cat /proc/14614/stack
> [<ffffffffa02d3e81>] wait_answer_interruptible+0x81/0xc0 [fuse]
> [<ffffffffa02d415b>] fuse_request_send+0x1cb/0x290 [fuse]
> [<ffffffffa02d652c>] fuse_do_getattr+0x10c/0x2c0 [fuse]
> [<ffffffffa02d6755>] fuse_update_attributes+0x75/0x80 [fuse]
> [<ffffffffa02d67b3>] fuse_getattr+0x53/0x60 [fuse]
> [<ffffffff81186d51>] vfs_getattr+0x51/0x80
> [<ffffffff81186de0>] vfs_fstatat+0x60/0x80
> [<ffffffff81186f2b>] vfs_stat+0x1b/0x20
> [<ffffffff81186f54>] sys_newstat+0x24/0x50
> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> Ubuntu - hanging ls
> root@srv:~# cat /proc/30012/stack
> [<ffffffffa061d04b>] ceph_mdsc_do_request+0xcb/0x1a0 [ceph]
> [<ffffffffa0608f37>] ceph_do_getattr+0xe7/0x120 [ceph]
> [<ffffffffa0608f94>] ceph_getattr+0x24/0x100 [ceph]
> [<ffffffff8118d42e>] vfs_getattr+0x4e/0x80
> [<ffffffff8118d4ae>] vfs_fstatat+0x4e/0x70
> [<ffffffff8118d4ee>] vfs_lstat+0x1e/0x20
> [<ffffffff8118d68a>] sys_newlstat+0x1a/0x40
> [<ffffffff816a6ba9>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> Started occurring shortly (within an hour or so) after adding a pool, not
> sure if that's relevant yet.
>
> -Michael
>
> On 23/10/2013 21:10, Michael wrote:
>>
>> I have a filesystem shared by several systems mounted on 2 ceph nodes with
>> a 3rd as a reference monitor.
>> It's been used for a couple of months now but suddenly the root directory
>> for the mount has become inaccessible and requests to files in it just hang,
>> there's no ceph errors reported before/after and subdirectories of the
>> directory can be used (and still are currently being used by VM's still
>> running from it). It's being mounted in a mixed kernel driver (ubuntu) and
>> centos (ceph-fuse) environment.
kernel, ceph-fuse and ceph-mds version? the hang was likely caused by an known
bug in kernel 3.10.
Regards
Yan, Zheng
>>
>> cluster ab3f7bc0-4cf7-4489-9cde-1af11d68a834
>> health HEALTH_OK
>> monmap e1: 3 mons at {srv10=##:6789/0,srv11=##:6789/0,srv8=##:6789/0},
>> election epoch 96, quorum 0,1,2 srv10,srv11,srv8
>> osdmap e2873: 6 osds: 6 up, 6 in
>> pgmap v2451618: 728 pgs: 728 active+clean; 128 GB data, 260 GB used,
>> 3929 GB / 4189 GB avail; 30365B/s wr, 5op/s
>> mdsmap e51: 1/1/1 up {0=srv10=up:active}
>>
>> Have done a full deep scrub/repair cycle on all of the osd which has come
>> back fine so not really sure where to start looking to find out what's wrong
>> with it.
>>
>> Any ideas?
>>
>> -Michael
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
