nfs-ganesha can store its client recovery database in a RADOS object. To do this it uses librados with a write_op to store the value in the omap. This generally works, but I've been playing with containerizing ganesha and there I've noticed that occasionally the string that is stored in the value field of the omap is truncated. On my test rig, the value should be 63 bytes, but ends up only being 29. Looking at the wire traffic, it's clear that the omap set operation is not done until the daemon is being shut down, well after the rados_write_op_operate call was issued. What I think is happening is that we generally end up with exclusive caps on this object, and the client just caches the write operation. When we go to shut down, we don't shut down the connection properly (currently) and that causes it to miss writing out the object. I think I can probably fix this by shutting things down cleanly in the clean shutdown case, but in this case, we really do need to synchronously write out the omap key to the database even if we hold exclusive caps on on the object. We're storing info to be used after a major outage, and we need this data to be stored properly before we can issue state based on it. So with all of that... how do I force the librados client to not buffer things when rados_write_op_operate is called? I had an initial hope that LIBRADOS_OPERATION_IGNORE_CACHE might do the right thing, but that seems to have more to do with internal OSD operation. Thanks, -- Jeff Layton <jlayton@xxxxxxxxxx> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
