OCFS2 on RBD

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi all,

I have a requirement for a highly available, high performance storage environment to serve data for webheads. (lots of reads on small files, limited writes) Having looked at all available options, OCFS2 on top of RBD appears to be the only solution that meets all my needs. I have used OCFS2 on top of an ISCSI SAN in the past with great success. Also, Ceph is is sexy. I would have like to try CephFS but the issue with no redundant / HA MD servers means it doesn't tick my boxes right now.

I built a Ceph system (three nodes with 3 1TB OSD's). The nodes are KVM Virtual Machines with storage and networking delivered via Infiniband on the hypervisor/host level. The VM's use VirtIO to access storage and networking. I mounted an RBD on some different nodes, ran up OCFS2 on top of that, copied some files onto the mount, and everything worked just peachy up to this point.

I then proceeded to rsync some files (about 70000 files worth about 15GB) across from a remote system which isn't very close or fast, and things broke about half way through. At some point, things started hanging, and the machine used to rsync (an OCFS2 node) spontaneously rebooted. This is probably because when OCFS2 doesn't like something, it will panic and reboot (this can of course be configured). The actual node that rebooted didn't produce any useful logs, but this showed up on a different OCFS2 node:
 (Node 90 is the node that was doing the rsyncing and rebooting):

[Mon Nov 24 04:00:26 2014] libceph: loaded (mon/osd proto 15/24)
[Mon Nov 24 04:00:26 2014] rbd: loaded rbd (rados block device)
[Mon Nov 24 04:00:26 2014] libceph: client4305 fsid b157f972-771b-47aa-826a-3bc97cf8d853
[Mon Nov 24 04:00:26 2014] libceph: mon0 192.168.5.31:6789 session established
[Mon Nov 24 04:00:26 2014]  rbd1: unknown partition table
[Mon Nov 24 04:00:26 2014] rbd: rbd1: added with size 0x1900000000
[Mon Nov 24 04:15:27 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 04:25:30 2014] OCFS2 Node Manager 1.5.0
[Mon Nov 24 04:25:30 2014] OCFS2 DLM 1.5.0
[Mon Nov 24 04:25:30 2014] ocfs2: Registered cluster interface o2cb
[Mon Nov 24 04:25:30 2014] OCFS2 DLMFS 1.5.0
[Mon Nov 24 04:25:30 2014] OCFS2 User DLM kernel interface loaded
[Mon Nov 24 04:30:27 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 04:39:49 2014] OCFS2 1.5.0
[Mon Nov 24 04:39:49 2014] o2dlm: Joining domain A04D7A74D30641B595E50B8C0CAAA157 ( 21 ) 1 nodes
[Mon Nov 24 04:39:49 2014] JBD2: Ignoring recovery information on journal
[Mon Nov 24 04:39:49 2014] ocfs2: Mounting device (251,0) on (node 21, slot 0) with ordered data mode.
[Mon Nov 24 04:39:55 2014] o2net: Accepted connection from node phptst01 (num 80) at 192.168.5.80:7777
[Mon Nov 24 04:39:57 2014] o2net: Accepted connection from node wwwtst01 (num 90) at 192.168.5.90:7777
[Mon Nov 24 04:40:02 2014] o2dlm: Node 80 joins domain A04D7A74D30641B595E50B8C0CAAA157 ( 21 80 ) 2 nodes
[Mon Nov 24 04:40:04 2014] o2dlm: Node 90 joins domain A04D7A74D30641B595E50B8C0CAAA157 ( 21 80 90 ) 3 nodes
[Mon Nov 24 04:44:58 2014] o2dlm: Node 90 leaves domain A04D7A74D30641B595E50B8C0CAAA157 ( 21 80 ) 2 nodes
[Mon Nov 24 04:45:00 2014] o2net: Connection to node wwwtst01 (num 90) at 192.168.5.90:7777 shutdown, state 8
[Mon Nov 24 04:45:00 2014] o2net: No longer connected to node wwwtst01 (num 90) at 192.168.5.90:7777
[Mon Nov 24 04:45:30 2014] o2net: Accepted connection from node wwwtst01 (num 90) at 192.168.5.90:7777
[Mon Nov 24 04:45:34 2014] o2dlm: Node 90 joins domain A04D7A74D30641B595E50B8C0CAAA157 ( 21 80 90 ) 3 nodes
[Mon Nov 24 04:54:49 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 05:09:49 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 05:24:50 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 05:39:50 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 05:54:50 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 06:09:51 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 06:24:51 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 06:39:51 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 06:54:51 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 07:09:52 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 07:24:52 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 07:39:52 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 07:54:52 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 08:09:53 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 08:24:53 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 08:39:53 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 08:54:54 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 09:09:54 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 09:24:54 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 09:39:54 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 09:54:55 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 10:09:55 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 10:24:55 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 10:39:56 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 10:54:56 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 11:09:56 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 11:24:57 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 11:39:57 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 11:54:57 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 12:09:57 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 12:24:58 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 12:39:58 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 12:54:58 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 13:09:59 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 13:24:59 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 13:39:59 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 13:54:59 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 14:10:00 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 14:25:00 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 14:40:00 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 14:55:00 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 15:10:01 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 15:25:01 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 15:40:01 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 15:55:02 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 16:10:02 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 16:25:02 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 16:40:02 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 16:55:03 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 17:10:03 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 17:25:03 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 17:40:04 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 17:55:04 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 18:10:04 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 18:25:04 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 18:40:05 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 18:55:05 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 19:10:05 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 19:25:06 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 19:40:06 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 19:55:06 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 20:10:06 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 20:25:07 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 20:40:07 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 20:55:07 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 21:10:07 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 21:25:08 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 21:40:08 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 21:55:08 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 22:10:09 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 22:25:09 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 22:40:09 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 22:55:09 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 23:10:10 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 23:25:10 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 23:40:10 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Mon Nov 24 23:55:11 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 00:10:11 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 00:25:11 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 00:40:11 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 00:55:12 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 01:10:12 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 01:25:12 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 01:40:13 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 01:55:13 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 02:10:13 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 02:25:13 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 02:40:14 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 02:55:14 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 03:10:14 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 03:25:14 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 03:40:15 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 03:55:15 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 04:10:15 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 04:25:16 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 04:40:16 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 04:55:16 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 05:10:17 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 05:25:17 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 05:40:17 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 05:55:17 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 06:10:18 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 06:25:18 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 06:36:48 2014] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[Tue Nov 25 06:36:48 2014] JFS: nTxBlock = 8192, nTxLock = 65536
[Tue Nov 25 06:36:48 2014] NTFS driver 2.1.30 [Flags: R/O MODULE].
[Tue Nov 25 06:36:48 2014] QNX4 filesystem 0.2.3 registered.
[Tue Nov 25 06:36:48 2014] xor: automatically using best checksumming function:
[Tue Nov 25 06:36:48 2014]    avx       :  3936.000 MB/sec
[Tue Nov 25 06:36:48 2014] raid6: sse2x1    5201 MB/s
[Tue Nov 25 06:36:49 2014] raid6: sse2x2    8978 MB/s
[Tue Nov 25 06:36:49 2014] raid6: sse2x4   10215 MB/s
[Tue Nov 25 06:36:49 2014] raid6: using algorithm sse2x4 (10215 MB/s)
[Tue Nov 25 06:36:49 2014] raid6: using ssse3x2 recovery algorithm
[Tue Nov 25 06:36:49 2014] bio: create slab <bio-1> at 1
[Tue Nov 25 06:36:49 2014] Btrfs loaded
[Tue Nov 25 06:40:18 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 06:55:19 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 07:10:19 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 07:25:19 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 07:31:08 2014] INFO: task nrsysmond:38979 blocked for more than 120 seconds.
[Tue Nov 25 07:31:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:31:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:31:08 2014] nrsysmond       D ffff88013ae34480     0 38979  35317 0x00000000
[Tue Nov 25 07:31:08 2014]  ffff880134e85ba0 0000000000000082 ffff8801380cb000 ffff880134e85fd8
[Tue Nov 25 07:31:08 2014]  0000000000014480 0000000000014480 ffff8801380cb000 ffff880134e85d48
[Tue Nov 25 07:31:08 2014]  ffff880134e85d50 7fffffffffffffff ffff8801380cb000 ffff880112c89f38
[Tue Nov 25 07:31:08 2014] Call Trace:
[Tue Nov 25 07:31:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:31:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:31:08 2014]  [<ffffffff811c802e>] ? generic_permission+0x15e/0x240
[Tue Nov 25 07:31:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:31:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:31:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:31:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:31:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:31:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:31:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:31:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:31:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:31:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:31:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:31:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:31:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:31:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:31:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:33:08 2014] INFO: task nrsysmond:38979 blocked for more than 120 seconds.
[Tue Nov 25 07:33:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:33:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:33:08 2014] nrsysmond       D ffff88013ae34480     0 38979  35317 0x00000000
[Tue Nov 25 07:33:08 2014]  ffff880134e85ba0 0000000000000082 ffff8801380cb000 ffff880134e85fd8
[Tue Nov 25 07:33:08 2014]  0000000000014480 0000000000014480 ffff8801380cb000 ffff880134e85d48
[Tue Nov 25 07:33:08 2014]  ffff880134e85d50 7fffffffffffffff ffff8801380cb000 ffff880112c89f38
[Tue Nov 25 07:33:08 2014] Call Trace:
[Tue Nov 25 07:33:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:33:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:33:08 2014]  [<ffffffff811c802e>] ? generic_permission+0x15e/0x240
[Tue Nov 25 07:33:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:33:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:33:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:33:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:33:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:33:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:33:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:33:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:33:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:33:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:33:08 2014] INFO: task df:38963 blocked for more than 120 seconds.
[Tue Nov 25 07:33:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:33:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:33:08 2014] df              D ffff88013ae34480     0 38963  38962 0x00000000
[Tue Nov 25 07:33:08 2014]  ffff88000cd8dba0 0000000000000082 ffff880036a8b000 ffff88000cd8dfd8
[Tue Nov 25 07:33:08 2014]  0000000000014480 0000000000014480 ffff880036a8b000 ffff88000cd8dd48
[Tue Nov 25 07:33:08 2014]  ffff88000cd8dd50 7fffffffffffffff ffff880036a8b000 ffff880112c89f38
[Tue Nov 25 07:33:08 2014] Call Trace:
[Tue Nov 25 07:33:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:33:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:33:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:33:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:33:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:33:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:33:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:33:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:33:08 2014]  [<ffffffff811cba43>] ? path_lookupat+0x73/0x790
[Tue Nov 25 07:33:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:33:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:33:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:33:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:33:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:35:08 2014] INFO: task nrsysmond:38979 blocked for more than 120 seconds.
[Tue Nov 25 07:35:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:35:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:35:08 2014] nrsysmond       D ffff88013ae34480     0 38979  35317 0x00000000
[Tue Nov 25 07:35:08 2014]  ffff880134e85ba0 0000000000000082 ffff8801380cb000 ffff880134e85fd8
[Tue Nov 25 07:35:08 2014]  0000000000014480 0000000000014480 ffff8801380cb000 ffff880134e85d48
[Tue Nov 25 07:35:08 2014]  ffff880134e85d50 7fffffffffffffff ffff8801380cb000 ffff880112c89f38
[Tue Nov 25 07:35:08 2014] Call Trace:
[Tue Nov 25 07:35:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:35:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:35:08 2014]  [<ffffffff811c802e>] ? generic_permission+0x15e/0x240
[Tue Nov 25 07:35:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:35:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:35:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:35:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:35:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:35:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:35:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:35:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:35:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:35:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:35:08 2014] INFO: task df:38963 blocked for more than 120 seconds.
[Tue Nov 25 07:35:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:35:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:35:08 2014] df              D ffff88013ae34480     0 38963  38962 0x00000000
[Tue Nov 25 07:35:08 2014]  ffff88000cd8dba0 0000000000000082 ffff880036a8b000 ffff88000cd8dfd8
[Tue Nov 25 07:35:08 2014]  0000000000014480 0000000000014480 ffff880036a8b000 ffff88000cd8dd48
[Tue Nov 25 07:35:08 2014]  ffff88000cd8dd50 7fffffffffffffff ffff880036a8b000 ffff880112c89f38
[Tue Nov 25 07:35:08 2014] Call Trace:
[Tue Nov 25 07:35:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:35:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:35:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:35:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:35:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:35:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:35:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:35:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:35:08 2014]  [<ffffffff811cba43>] ? path_lookupat+0x73/0x790
[Tue Nov 25 07:35:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:35:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:35:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:35:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:35:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:37:08 2014] INFO: task nrsysmond:38979 blocked for more than 120 seconds.
[Tue Nov 25 07:37:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:37:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:37:08 2014] nrsysmond       D ffff88013ae34480     0 38979  35317 0x00000000
[Tue Nov 25 07:37:08 2014]  ffff880134e85ba0 0000000000000082 ffff8801380cb000 ffff880134e85fd8
[Tue Nov 25 07:37:08 2014]  0000000000014480 0000000000014480 ffff8801380cb000 ffff880134e85d48
[Tue Nov 25 07:37:08 2014]  ffff880134e85d50 7fffffffffffffff ffff8801380cb000 ffff880112c89f38
[Tue Nov 25 07:37:08 2014] Call Trace:
[Tue Nov 25 07:37:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:37:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:37:08 2014]  [<ffffffff811c802e>] ? generic_permission+0x15e/0x240
[Tue Nov 25 07:37:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:37:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:37:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:37:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:37:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:37:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:37:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:37:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:37:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:37:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:37:08 2014] INFO: task df:38963 blocked for more than 120 seconds.
[Tue Nov 25 07:37:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:37:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:37:08 2014] df              D ffff88013ae34480     0 38963  38962 0x00000000
[Tue Nov 25 07:37:08 2014]  ffff88000cd8dba0 0000000000000082 ffff880036a8b000 ffff88000cd8dfd8
[Tue Nov 25 07:37:08 2014]  0000000000014480 0000000000014480 ffff880036a8b000 ffff88000cd8dd48
[Tue Nov 25 07:37:08 2014]  ffff88000cd8dd50 7fffffffffffffff ffff880036a8b000 ffff880112c89f38
[Tue Nov 25 07:37:08 2014] Call Trace:
[Tue Nov 25 07:37:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:37:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:37:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:37:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:37:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:37:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:37:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:37:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:37:08 2014]  [<ffffffff811cba43>] ? path_lookupat+0x73/0x790
[Tue Nov 25 07:37:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:37:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:37:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:37:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:37:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:39:08 2014] INFO: task nrsysmond:38979 blocked for more than 120 seconds.
[Tue Nov 25 07:39:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:39:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:39:08 2014] nrsysmond       D ffff88013ae34480     0 38979  35317 0x00000000
[Tue Nov 25 07:39:08 2014]  ffff880134e85ba0 0000000000000082 ffff8801380cb000 ffff880134e85fd8
[Tue Nov 25 07:39:08 2014]  0000000000014480 0000000000014480 ffff8801380cb000 ffff880134e85d48
[Tue Nov 25 07:39:08 2014]  ffff880134e85d50 7fffffffffffffff ffff8801380cb000 ffff880112c89f38
[Tue Nov 25 07:39:08 2014] Call Trace:
[Tue Nov 25 07:39:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:39:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:39:08 2014]  [<ffffffff811c802e>] ? generic_permission+0x15e/0x240
[Tue Nov 25 07:39:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:39:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:39:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:39:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:39:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:39:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:39:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:39:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:39:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:39:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:39:08 2014] INFO: task df:38963 blocked for more than 120 seconds.
[Tue Nov 25 07:39:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:39:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:39:08 2014] df              D ffff88013ae34480     0 38963  38962 0x00000000
[Tue Nov 25 07:39:08 2014]  ffff88000cd8dba0 0000000000000082 ffff880036a8b000 ffff88000cd8dfd8
[Tue Nov 25 07:39:08 2014]  0000000000014480 0000000000014480 ffff880036a8b000 ffff88000cd8dd48
[Tue Nov 25 07:39:08 2014]  ffff88000cd8dd50 7fffffffffffffff ffff880036a8b000 ffff880112c89f38
[Tue Nov 25 07:39:08 2014] Call Trace:
[Tue Nov 25 07:39:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:39:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:39:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:39:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:39:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:39:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:39:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:39:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:39:08 2014]  [<ffffffff811cba43>] ? path_lookupat+0x73/0x790
[Tue Nov 25 07:39:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:39:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:39:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:39:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:39:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:40:19 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 07:41:08 2014] INFO: task nrsysmond:38979 blocked for more than 120 seconds.
[Tue Nov 25 07:41:08 2014]       Tainted: G           OX 3.13.0-39-generic #66-Ubuntu
[Tue Nov 25 07:41:08 2014] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Nov 25 07:41:08 2014] nrsysmond       D ffff88013ae34480     0 38979  35317 0x00000000
[Tue Nov 25 07:41:08 2014]  ffff880134e85ba0 0000000000000082 ffff8801380cb000 ffff880134e85fd8
[Tue Nov 25 07:41:08 2014]  0000000000014480 0000000000014480 ffff8801380cb000 ffff880134e85d48
[Tue Nov 25 07:41:08 2014]  ffff880134e85d50 7fffffffffffffff ffff8801380cb000 ffff880112c89f38
[Tue Nov 25 07:41:08 2014] Call Trace:
[Tue Nov 25 07:41:08 2014]  [<ffffffff81723109>] schedule+0x29/0x70
[Tue Nov 25 07:41:08 2014]  [<ffffffff81722359>] schedule_timeout+0x239/0x2d0
[Tue Nov 25 07:41:08 2014]  [<ffffffff811c802e>] ? generic_permission+0x15e/0x240
[Tue Nov 25 07:41:08 2014]  [<ffffffff812d4a1c>] ? security_inode_permission+0x1c/0x30
[Tue Nov 25 07:41:08 2014]  [<ffffffff811c8162>] ? __inode_permission+0x52/0xc0
[Tue Nov 25 07:41:08 2014]  [<ffffffff81723c26>] wait_for_completion+0xa6/0x160
[Tue Nov 25 07:41:08 2014]  [<ffffffff8109a8d0>] ? wake_up_state+0x20/0x20
[Tue Nov 25 07:41:08 2014]  [<ffffffffa03e518a>] __ocfs2_cluster_lock.isra.34+0x21a/0x820 [ocfs2]
[Tue Nov 25 07:41:08 2014]  [<ffffffff811c734d>] ? complete_walk+0x4d/0xe0
[Tue Nov 25 07:41:08 2014]  [<ffffffff811cd05f>] ? getname_flags+0x4f/0x190
[Tue Nov 25 07:41:08 2014]  [<ffffffffa03e7974>] ocfs2_inode_lock_full_nested+0x1f4/0x4f0 [ocfs2]
[Tue Nov 25 07:41:08 2014]  [<ffffffffa042ae65>] ocfs2_statfs+0x65/0x320 [ocfs2]
[Tue Nov 25 07:41:08 2014]  [<ffffffff811ee331>] statfs_by_dentry+0xa1/0x140
[Tue Nov 25 07:41:08 2014]  [<ffffffff811ee3eb>] vfs_statfs+0x1b/0xb0
[Tue Nov 25 07:41:08 2014]  [<ffffffff811ee4c6>] user_statfs+0x46/0x80
[Tue Nov 25 07:41:08 2014]  [<ffffffff811ee515>] SYSC_statfs+0x15/0x30
[Tue Nov 25 07:41:08 2014]  [<ffffffff811ee64e>] SyS_statfs+0xe/0x10
[Tue Nov 25 07:41:08 2014]  [<ffffffff8172f7ed>] system_call_fastpath+0x1a/0x1f
[Tue Nov 25 07:43:58 2014] o2net: No longer connected to node wwwtst01 (num 90) at 192.168.5.90:7777
[Tue Nov 25 07:43:58 2014] (dlm_thread,48693,1):dlm_send_proxy_ast_msg:482 ERROR: A04D7A74D30641B595E50B8C0CAAA157: res S000000000000000000000                                                                     200000000, error -112 send AST to node 90
[Tue Nov 25 07:43:58 2014] (dlm_thread,48693,1):dlm_flush_asts:596 ERROR: status = -112
[Tue Nov 25 07:43:58 2014] o2cb: o2dlm has evicted node 90 from domain A04D7A74D30641B595E50B8C0CAAA157
[Tue Nov 25 07:44:00 2014] o2dlm: Begin recovery on domain A04D7A74D30641B595E50B8C0CAAA157 for node 90
[Tue Nov 25 07:44:00 2014] o2dlm: Node 21 (me) is the Recovery Master for the dead node 90 in domain A04D7A74D30641B595E50B8C0CAAA157
[Tue Nov 25 07:44:05 2014] o2dlm: End recovery on domain A04D7A74D30641B595E50B8C0CAAA157
[Tue Nov 25 07:48:06 2014] o2net: Accepted connection from node wwwtst01 (num 90) at 192.168.5.90:7777
[Tue Nov 25 07:48:10 2014] o2dlm: Node 90 joins domain A04D7A74D30641B595E50B8C0CAAA157 ( 21 80 90 ) 3 nodes
[Tue Nov 25 07:55:20 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 08:10:20 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 08:25:20 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 08:40:21 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 08:55:21 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)
[Tue Nov 25 09:10:21 2014] libceph: osd2 192.168.5.31:6800 socket closed (con state OPEN)


Whilst this looks like an OCFS2 issue, I am posting this here as I have seen some bugs in the Ceph tracker with similar patterns: ceph socket closed, combined with [TASK] blocked for more than 120 seconds.

I would appreciate any pointers as to where to even begin looking for resolving this issue. besides the random "host rebooting" there is also a question of performance, but I am testing the various independent components to figure out where the bottlenecks are.

Many thanks!

Martijn



_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux