Hi,
What is your "allocsize"? If it's default you're allocating 64KiB per file. If you have lots of small files (check file size distribution) you may need to use a smaller allocsize (eg. 4KiB).
Best,
On Tue, Oct 25, 2016 at 4:37 AM, Василий Ангапов <angapov@xxxxxxxxx> wrote:
Hello,
I got Ceph 10.2.1 cluster with 10 nodes, each having 29 * 6TB OSDs.
Yesterday I found that 3 OSDs were down and out with 89% space
utilization.
In logs there is:
2016-10-24 22:36:37.599253 7f8309c5e800 0 ceph version 10.2.1
(3a66dd4f30852819c1bdaa8ec23c79 5d4ad77269), process ceph-osd, pid
2602081
2016-10-24 22:36:37.600129 7f8309c5e800 0 pidfile_write: ignore empty
--pid-file
2016-10-24 22:36:37.635769 7f8309c5e800 0
filestore(/var/lib/ceph/osd/ceph-123) backend xfs (magic 0x58465342)
2016-10-24 22:36:37.635805 7f8309c5e800 -1
genericfilestorebackend(/var/lib/ceph/osd/ceph-123) detect_features:
unable to create /var/lib/ceph/osd/ceph-123/fiemap_test: (28) No space
left on device
2016-10-24 22:36:37.635814 7f8309c5e800 -1
filestore(/var/lib/ceph/osd/ceph-123) _detect_fs: detect_features
error: (28) No space left on device
2016-10-24 22:36:37.635818 7f8309c5e800 -1
filestore(/var/lib/ceph/osd/ceph-123) FileStore::mount: error in
_detect_fs: (28) No space left on device
2016-10-24 22:36:37.635824 7f8309c5e800 -1 osd.123 0 OSD:init: unable
to mount object store
2016-10-24 22:36:37.635827 7f8309c5e800 -1 ESC[0;31m ** ERROR: osd
init failed: (28) No space left on deviceESC[0m
root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ df -h /var/lib/ceph/osd/ceph-123
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/disk23p1 5.5T 4.9T 651G 89% /var/lib/ceph/osd/ceph-123
root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ df -i /var/lib/ceph/osd/ceph-123
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/disk23p1 146513024 22074752 124438272 16%
/var/lib/ceph/osd/ceph-123
root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ touch 123
touch: cannot touch ‘123’: No space left on device
root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ grep ceph-123 /proc/mounts
/dev/mapper/disk23p1 /var/lib/ceph/osd/ceph-123 xfs
rw,noatime,attr2,inode64,noquota 0 0
The same situation is for all three down OSDs. OSD can be unmounted
and mounted without problem:
root@ed-ds-c178:[~]:$ umount /var/lib/ceph/osd/ceph-123
root@ed-ds-c178:[~]:$
root@ed-ds-c178:[~]:$ mount /var/lib/ceph/osd/ceph-123
root@ed-ds-c178:[~]:$ touch /var/lib/ceph/osd/ceph-123/123
touch: cannot touch ‘/var/lib/ceph/osd/ceph-123/123’: No space left on device
xfs_repair gives no error for FS.
Kernel is
root@ed-ds-c178:[~]:$ uname -r
4.7.0-1.el7.wg.x86_64
What else can I do to rectify that situation?
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
Łukasz Jagiełło
lukasz<at>jagiello<dot>org
lukasz<at>jagiello<dot>org
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
