Actually all OSDs are already mounted with inode64 option. Otherwise I could not write beyond 1TB. 2016-10-25 14:53 GMT+03:00 Ashley Merrick <ashley@xxxxxxxxxxxxxx>: > Sounds like 32bit Inode limit, if you mount with -o inode64 (not 100% how you would do in ceph), would allow data to continue to be wrote. > > ,Ashley > > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of ??????? ??????? > Sent: 25 October 2016 12:38 > To: ceph-users <ceph-users@xxxxxxxxxxxxxx> > Subject: XFS no space left on device > > Hello, > > I got Ceph 10.2.1 cluster with 10 nodes, each having 29 * 6TB OSDs. > Yesterday I found that 3 OSDs were down and out with 89% space utilization. > In logs there is: > 2016-10-24 22:36:37.599253 7f8309c5e800 0 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid > 2602081 > 2016-10-24 22:36:37.600129 7f8309c5e800 0 pidfile_write: ignore empty --pid-file > 2016-10-24 22:36:37.635769 7f8309c5e800 0 > filestore(/var/lib/ceph/osd/ceph-123) backend xfs (magic 0x58465342) > 2016-10-24 22:36:37.635805 7f8309c5e800 -1 > genericfilestorebackend(/var/lib/ceph/osd/ceph-123) detect_features: > unable to create /var/lib/ceph/osd/ceph-123/fiemap_test: (28) No space left on device > 2016-10-24 22:36:37.635814 7f8309c5e800 -1 > filestore(/var/lib/ceph/osd/ceph-123) _detect_fs: detect_features > error: (28) No space left on device > 2016-10-24 22:36:37.635818 7f8309c5e800 -1 > filestore(/var/lib/ceph/osd/ceph-123) FileStore::mount: error in > _detect_fs: (28) No space left on device > 2016-10-24 22:36:37.635824 7f8309c5e800 -1 osd.123 0 OSD:init: unable to mount object store > 2016-10-24 22:36:37.635827 7f8309c5e800 -1 ESC[0;31m ** ERROR: osd init failed: (28) No space left on deviceESC[0m > > root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ df -h /var/lib/ceph/osd/ceph-123 > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/disk23p1 5.5T 4.9T 651G 89% /var/lib/ceph/osd/ceph-123 > > root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ df -i /var/lib/ceph/osd/ceph-123 > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/mapper/disk23p1 146513024 22074752 124438272 16% > /var/lib/ceph/osd/ceph-123 > > root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ touch 123 > touch: cannot touch ‘123’: No space left on device > > root@ed-ds-c178:[/var/lib/ceph/osd/ceph-123]:$ grep ceph-123 /proc/mounts > /dev/mapper/disk23p1 /var/lib/ceph/osd/ceph-123 xfs rw,noatime,attr2,inode64,noquota 0 0 > > The same situation is for all three down OSDs. OSD can be unmounted and mounted without problem: > root@ed-ds-c178:[~]:$ umount /var/lib/ceph/osd/ceph-123 root@ed-ds-c178:[~]:$ root@ed-ds-c178:[~]:$ mount /var/lib/ceph/osd/ceph-123 root@ed-ds-c178:[~]:$ touch /var/lib/ceph/osd/ceph-123/123 > touch: cannot touch ‘/var/lib/ceph/osd/ceph-123/123’: No space left on device > > xfs_repair gives no error for FS. > > Kernel is > root@ed-ds-c178:[~]:$ uname -r > 4.7.0-1.el7.wg.x86_64 > > What else can I do to rectify that situation? > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |