Thanks, we found that even after I clean some disk space and make sure all the osd disk usage is less than 90%, I still can not write any data to the filesystem(not a single byte). Henry have check the kernel client debug info and found the osdmap is out of date and the flag is keep at full. He will post more detail information later. I guess there are some error handling problem there so that the client didn't keep updating the osdmap when disk is fulled. Any suggestion for further trouble shooting ? Regards, Leander Yu. On Wed, Oct 6, 2010 at 12:48 PM, Gregory Farnum <gregf@xxxxxxxxxxxxxxx> wrote: > On Tue, Oct 5, 2010 at 9:40 PM, Leander Yu <leander.yu@xxxxxxxxx> wrote: >> Hi, >> I just found my ceph cluster report no space left error. I check the >> df and every osd disk. it still has space available and after delete >> some file, I still can't write any data to the file system. >> Any suggestion for trouble shooting this case? > As with all distributed filesystems, Ceph still doesn't handle things > very well when even one disk runs out of space. Some sort of solution > will appear, but isn't on the roadmap yet. The most likely cause is > that you have disks of different sizes and haven't balanced their > input (via the CRUSH map) to match. Unfortunately, the best fix is > either to keep deleting data or to put a larger disk in whichever OSD > is full. The logs should tell you which one reported full. > > Keep in mind that to prevent more nasty badness from the local > filesystem, Ceph reports a disk "full" at some percentage below full > (I think it's 95%, but it may actually be less). > -Greg > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
