Hi,
After our trouble with ext4/xattr soft lockup kernel bug we started
moving some of our OSD to XFS, we're using ubuntu 14.04 3.19 kernel
and ceph 0.94.5.
We have two out of 28 rotational OSD running XFS and
they both get restarted regularly because they're terminating with
"ENOSPC":
2015-11-25 16:51:08.015820 7f6135153700 0 filestore(/var/lib/ceph/osd/ceph-11) error (28) No space left on device not handled on operation 0xa0f4d520 (12849173.0.4, or op 4, counting from 0)
2015-11-25 16:51:08.015837 7f6135153700 0 filestore(/var/lib/ceph/osd/ceph-11) ENOSPC handling not implemented
2015-11-25 16:51:08.015838 7f6135153700 0 filestore(/var/lib/ceph/osd/ceph-11) transaction dump:
...
{
"op_num": 4,
"op_name": "write",
"collection": "58.2d5_head",
"oid": "53e4fed5\/rbd_data.11f20f75aac8266.00000000000a79eb\/head\/\/58",
"length": 73728,
"offset": 4120576,
"bufferlist length": 73728
},
(Writing the last 73728 bytes = 72 kbytes of 4 Mbytes if I'm reading
this correctly)
Mount options:
/dev/sdb1 /var/lib/ceph/osd/ceph-11 xfs rw,noatime,attr2,inode64,noquota
Space and Inodes:
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/sdb1 xfs 1947319356 1624460408 322858948 84% /var/lib/ceph/osd/ceph-11
Filesystem Type Inodes IUsed IFree IUse% Mounted on
/dev/sdb1 xfs 48706752 1985587 46721165 5% /var/lib/ceph/osd/ceph-11
We're only using rbd devices, so max 4 MB/object write, how
can we get ENOSPC for a 4MB operation with 322 GB free space?
The most surprising thing is that after the automatic restart
disk usage keep increasing and we no longer get ENOSPC for a while.
Did we miss a needed XFS mount option? Did other ceph users
encounter this issue with XFS?
We have no such issue with ~96% full ext4 OSD (after setting the right
value for the various ceph "fill" options).
Thanks in advance,
Laurent
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
