Looks like the same problem like mine: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-January/032054.html The free space is total while Ceph uses the smallest free space (worst OSD). Please check your (re-)weights. Kevin Am Di., 8. Jan. 2019 um 14:32 Uhr schrieb Rodrigo Embeita <rodrigo@xxxxxxxxxxxxxxx>: > > Hi guys, I need your help. > I'm new with Cephfs and we started using it as file storage. > Today we are getting no space left on device but I'm seeing that we have plenty space on the filesystem. > Filesystem Size Used Avail Use% Mounted on > 192.168.51.8,192.168.51.6,192.168.51.118:6789:/pagefreezer/smhosts 73T 39T 35T 54% /mnt/cephfs > > We have 35TB of disk space. I've added 2 additional OSD disks with 7TB each but I'm getting the error "No space left on device" every time that I want to add a new file. > After adding the 2 additional OSD disks I'm seeing that the load is beign distributed among the cluster. > Please I need your help. > > root@pf-us1-dfs1:/etc/ceph# ceph -s > cluster: > id: 609e9313-bdd3-449e-a23f-3db8382e71fb > health: HEALTH_ERR > 2 backfillfull osd(s) > 1 full osd(s) > 7 pool(s) full > 197313040/508449063 objects misplaced (38.807%) > Degraded data redundancy: 2/508449063 objects degraded (0.000%), 2 pgs degraded > Degraded data redundancy (low space): 16 pgs backfill_toofull, 3 pgs recovery_toofull > > services: > mon: 3 daemons, quorum pf-us1-dfs2,pf-us1-dfs1,pf-us1-dfs3 > mgr: pf-us1-dfs3(active), standbys: pf-us1-dfs2 > mds: pagefs-2/2/2 up {0=pf-us1-dfs3=up:active,1=pf-us1-dfs1=up:active}, 1 up:standby > osd: 10 osds: 10 up, 10 in; 189 remapped pgs > rgw: 1 daemon active > > data: > pools: 7 pools, 416 pgs > objects: 169.5 M objects, 3.6 TiB > usage: 39 TiB used, 34 TiB / 73 TiB avail > pgs: 2/508449063 objects degraded (0.000%) > 197313040/508449063 objects misplaced (38.807%) > 224 active+clean > 168 active+remapped+backfill_wait > 16 active+remapped+backfill_wait+backfill_toofull > 5 active+remapped+backfilling > 2 active+recovery_toofull+degraded > 1 active+recovery_toofull > > io: > recovery: 1.1 MiB/s, 31 objects/s > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |