On Sat, Jun 23, 2012 at 3:43 AM, David Blundell <David.Blundell@xxxxxxxxxxxxxxxx> wrote: > The logs on all three servers are full of messages like: > Jun 23 04:02:19 Store2 kernel: [63811.494955] ceph-osd: page allocation failure: order:3, mode:0x4020 > > The difference between the lines is that order: varies between 2, 3, 4 or 5 > > Is this likely to be a btrfs bug? That means you're running out of memory, in kernelspace. The order is the power-of-two (2**n) of how many 4kB pages were requested, 0x4020 = GFP_COMP|GFP_HIGH (compound & access emergency pools). Btrfs may be indirectly related, it's not clear what's consuming all the memory, but that doesn't sound all that likely. That message should be followed by a stack dump, that might tell us more. Are you using the Ceph distributed filesystem, or just the RADOS level, e.g. RBD images? -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html