On 3/16/16 7:15 PM, Theodore Ts'o wrote: > On Wed, Mar 16, 2016 at 03:45:49PM -0600, Andreas Dilger wrote: >>> Clearly, the performance hit of unwritten extent conversion is large >>> enough to tempt people to ask for no-hide-stale. But I'd rather hear >>> that directly from a developer, Ceph or otherwise. >> >> I suspect that this gets significantly worse if you are running with >> random writes instead of sequential overwrites. With sequential overwrites >> there is only a single boundary between init and uninit extents, so at >> most one extra extent in the tree. The above performance deltas will also >> be much larger when real disks are involved and seek latency is a factor. > > It will vary a lot depending on your use case. If you are running > with data=ordered, and with journalled enabled, then even if it is a > single extent that is modified, the fact that a journal transaction > involved, with a forced data block flush to avoid revealing stale > data, that is certainly going to be measurable. > > The other thing is if you are worried about tail latency, which is a > major concern at Google[1], and you are running your disks close to > flat out, the fact that you have to do an extra seek to update the > extent tree is a seek that you can't be using for useful work --- and > worse, could delay a low-latency read from completing within your SLO. > > [1] https://research.google.com/pubs/pub44830.html > > Part of what's challenging with giving numbers is that it's trivially > easy to give some worst case scneario where the numbers are really > terrible. A random 4k random write benchmark into an fallocated file, > eeven with XFS, would have pretty bad numbers, But of course people > wouldn't say that it's very realistic. But those are the easiest to > get. > > The most realistic numbers are going to be a lot harder to get, and > wouldn't necessarily make a lot of sense without revealing a lot > proprietary information. I will say that Google does have a fairly > large number of disks[2] and so even a small fractional percentage > gain multipled by gazillions of disks starts turning into a dollar > number with enough zeros that people really sit up and take notice. > I'll also note that map reduce can be quite nasty as far as random I/O > is concerned[3], and while map reduce jobs are often not high priority > jobs, they can interfere with low-latency reads from important > applications (e.g., web search, user-visible gmail operations, etc.) > > [2] https://what-if.xkcd.com/63/ > [3] https://pdfs.semanticscholar.org/6238/e5f0fd807f634f5999701c7aa6a09d88dfc8.pdf > > So I'm not sure what numbers I can really give that would satisfy > people. Doing a random write fio job is not hard, and will result in > fairly impressive numbers. If that's enough, then either I can do > this, or Chris Mason can reproduce his experiment using XFS (which > would presumably eliminate the excuse that it's because ext4 sucks at > extent operations). But if that's not going to convince people, then > I'd much rather not waste my time. > > Besides, at Google it's easy enough for me to maintain the patch > out-of-tree. It's the Ceph folks who would need to at the very least, > have such a patch ship in Red Hat Enterprise Linux. So it's probably > better for them to justify it, if numbers are really necessary. I may have lost the thread at this point, with poor Darrick's original patch submission devolving into a long thread about a NO_HIDE_STALE patch used at Google, but I don't *think* Ceph ever asked for NO_HIDE_STALE. At least I can't find any indication of that. Am I missing something? cc'ing Greg on this one in case I am. -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-api" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
