On Thu, 2015-11-26 at 22:13 +0300, Andrey Korolyov wrote:
> On Thu, Nov 26, 2015 at 1:29 AM, Laurent GUERBY <laurent@xxxxxxxxxx> wrote:
> > Hi,
> >
> > After our trouble with ext4/xattr soft lockup kernel bug we started
> > moving some of our OSD to XFS, we're using ubuntu 14.04 3.19 kernel
> > and ceph 0.94.5.
> >
> > We have two out of 28 rotational OSD running XFS and
> > they both get restarted regularly because they're terminating with
> > "ENOSPC":
> >
> > 2015-11-25 16:51:08.015820 7f6135153700 0 filestore(/var/lib/ceph/osd/ceph-11) error (28) No space left on device not handled on operation 0xa0f4d520 (12849173.0.4, or op 4, counting from 0)
> > 2015-11-25 16:51:08.015837 7f6135153700 0 filestore(/var/lib/ceph/osd/ceph-11) ENOSPC handling not implemented
> > 2015-11-25 16:51:08.015838 7f6135153700 0 filestore(/var/lib/ceph/osd/ceph-11) transaction dump:
> > ...
> > {
> > "op_num": 4,
> > "op_name": "write",
> > "collection": "58.2d5_head",
> > "oid": "53e4fed5\/rbd_data.11f20f75aac8266.00000000000a79eb\/head\/\/58",
> > "length": 73728,
> > "offset": 4120576,
> > "bufferlist length": 73728
> > },
> >
> > (Writing the last 73728 bytes = 72 kbytes of 4 Mbytes if I'm reading
> > this correctly)
> >
> > Mount options:
> >
> > /dev/sdb1 /var/lib/ceph/osd/ceph-11 xfs rw,noatime,attr2,inode64,noquota
> >
> > Space and Inodes:
> >
> > Filesystem Type 1K-blocks Used Available Use% Mounted on
> > /dev/sdb1 xfs 1947319356 1624460408 322858948 84% /var/lib/ceph/osd/ceph-11
> >
> > Filesystem Type Inodes IUsed IFree IUse% Mounted on
> > /dev/sdb1 xfs 48706752 1985587 46721165 5% /var/lib/ceph/osd/ceph-11
> >
> > We're only using rbd devices, so max 4 MB/object write, how
> > can we get ENOSPC for a 4MB operation with 322 GB free space?
> >
> > The most surprising thing is that after the automatic restart
> > disk usage keep increasing and we no longer get ENOSPC for a while.
> >
> > Did we miss a needed XFS mount option? Did other ceph users
> > encounter this issue with XFS?
> >
> > We have no such issue with ~96% full ext4 OSD (after setting the right
> > value for the various ceph "fill" options).
> >
> > Thanks in advance,
> >
> > Laurent
> >
>
> Hi, from given numbers one can conclude that you are facing some kind
> of XFS preallocation bug, because ((raw space divided by number of
> files)) is four times lower than the ((raw space divided by 4MB
> blocks)). At a glance it could be avoided by specifying relatively
> small allocsize= mount option, of course by impacting overall
> performance, appropriate benchmarks could be found through
> ceph-users/ceph-devel. Also do you plan to preserve overcommit ratio
> to be that high forever?
Hi again,
Looks like we hit a bug in image deletion leaving objects undeleted on
disk:
http://tracker.ceph.com/issues/13894
I assume we'll get a lot more free space when it's fixed :).
Laurent
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
