On 11/27/2012 11:07 AM, Gregory Farnum wrote:
On Tuesday, November 27, 2012 at 8:45 AM, Sam Lang wrote:
Hi Noah,
I was able to reproduce your issue with a similar test using the fuse
client and the clock_offset option for the mds. This is what I see
happening:
clientA's clock is a few seconds behind the mds clock
clientA creates the file
- the mds sets the mtime from its current time
- clientA acquires the exclusive capability (cap) for the file
clientA writes to the file
- the mtime is updated locally (at clientA with its current time)
clientA closes the file
- the exclusive cap is flushed to the mds, but the mtime is less
than the create mtime because of the clock skew, so the mds
doesn't update it to the mtime from clientA's write
clientA stats the file
- the mtime from the write (still cached) gets returned. I saw a
race in my tests, where sometimes the mtime was from the cache
(if the flush hadn't completed I assume), and sometimes it was
from the mds.
clientB stats the file
- the exclusive cap is revoked at clientA, but the mtime returned
to clientB is from the mds
Hurray, I think we all agree about what's happening now! :)
Have you checked to see if the MDS ever sets mtime after create, or is it always dictated by the client following that?
It sets it on truncate as well.
-sam
The goal of the current implementation is to provide an mtime that is
non-decreasing, but that conflicts with using mtime as a version in this
case. Using mtime as a version has its own set of problems, but I won't
go into that here. I think there are a few alternatives if we want to
try to have a more consistent mtime value across clients.
1. Let the client set the create mtime. This avoids the issue that the
mds and client clocks are out of sync, but in other cases where the
client has a clock a few seconds ahead of other clients, we run into a
similar problem. This might be reasonable considering clients that
share state will more likely have synchronized clocks than the clients
and mds.
2. Provide a config option to always set the mtime on cap flush/revoke,
even if its less than the current mtime. This breaks the non-decreasing
behavior, and requires the user set a config option across the cluster
if they want this.
3. When a client acquires the cap for a file, have the mds provide its
current time as well. As the client updates the mtime, it uses the
timestamp provided by the mds and the time since the cap was acquired.
Except for the skew caused by the message latency, this approach allows
the mtime to be based off the mds time, so it will be consistent across
clients and the mds. It does however, allow a client to set an mtime to
the future (based off of its local time), which might be undesirable,
but that is more like how NFS behaves. Message latency probably won't
be much of an issue either, as the granularity of mtime is a second.
Also, the client can set its cap acquired timestamp to the time at which
the cap was requested, ensuring that the relative increment includes the
round trip latency so that the mtime will always be set further ahead.
Of course, this approach would be a lot more intrusive to implement. :-)
I actually like this third approach of letting the MDS be authoritative about time even if it's not directly involved. Given that, I wonder if perhaps the client should just have time translation functions it uses everywhere?
However, the problem with that is that the different MDS daemons might also disagree about time. Perhaps they could adopt the master MDS clock or something skanky like that… :/
The fundamental issue of time resolution is why things are the way they are now, of course — you usually don't want things going "backwards" in time, but clock skews are a real problem in large clusters so we just decided to not let things go back on the assumption it would be a temporary and little-noticed disparity, with easy-to-understand behavior. Obviously we were wrong about it being little-noticed.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
