>OSD write errors are not usual events: any issues with the underlying >storage are expected to be handled by RADOS, and write operations to >an unhealthy cluster should block, rather than returning an error. It >would not be correct for CephFS to throw away metadata updates in the >case of unexpected write errors -- this is a strongly consistent >system, so when we can't make progress consistently (i.e. respecting >all the ops we've seen in order), then we have to stop. Thank you for that explanation; that all makes sense. I have to get used to the idea of responding to broken storage by waiting indefinitely until it is isn't broken. I wasn't thinking in those terms. >I'm guessing that you changed some related settings (like >mds_log_segment_size) to get into this situation? Otherwise, an error >like this would definitely be a bug. What I changed (from default) was osd_max_write_size. I set it to its legal minimum, 1M. I've discovered that there are clients all around that expect to be able to write 4M and don't respond nicely when they can't. Rather than try to find and change them all, I'm going to capitulate and go ahead and make osd_max_write_size 4M. Does manually tuning every client to make it consistent with the OSD's maximum write size have to be what avoids crashes like this? It sure would be nice if an MDS could detect much earlier that the log is on an OSD that's incapable of hosting that log. But I found the filesystem driver is the same way - I have to tell it how big a write it can do; it can't figure it out from the OSDs. So maybe its a fundamental architecture thing. -- Bryan Henderson San Jose, California _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |