On Wed, 2016-03-02 at 13:01 -0600, Frank Sorenson wrote: > Currently, the only visibility into filesystem freeze or > thaw activity is when an error is returned from the actual > call to freeze or thaw. If the action itself hangs, there > is no indication that the freeze or thaw was in-progress, > short of collecting a vmcore. > > There is also no record of what process froze a filesystem > or when it happened, so if the process does not thaw it > later, debugging the issue is difficult. > Thanks for getting the ball rolling Frank! For some background, Frank can correct me if I'm wrong, but I think the main "use case" which spurred these patches is as follows. Frank, myself and others supporting enterprise customers on Linux very frequently see vmcores come in with customers reporting "hung systems". We analyze the vmcore and see that the root filesystem, and often every other local filesystem is frozen. In the vmcore we see many processes blocked waiting on a filesystem to become unfrozen. In short, as a result of the filesystems being frozen, eventually the system grinds to a halt and users wonder what happened. So the discussion then turns to "Who froze the filesystem, and why didn't they unfreeze it"? One approach we have is to give the customer a systemtap script which basically does what these patches do and shows which process issued a 'freeze' call and then 'unfreeze'. As "simple" as systemtap is, many customers run into problems with installing the requirements and/or they don't want to do it. It also requires they reproduce the issue and give us another vmcore so we can see the output. The discussion almost always ends with "Contact those responsible for backup / snapshotting of your hung system, as this is most likely an issue with snapshots / backup", but unfortunately we can't give many more details than that since we don't have any log of what happened. I thought about this some more and I realized the fact that if the customer gives us a vmcore and it contains frozen filesystem, all parties involved have "already lost". This indicates usually a couple things: 1. They didn't know what happened 2. There's a bug in freeze / thaw, or backup/snapshot agent, which needs fixed On problem #1, I wonder now if a 'freeze' operation should be treated more like a "shutdown" where all users are notified this is about to happen. One downside of this is I can imagine this defeats the intent of such "VM snapshots" or "transparent backups" where they probably don't want people to notice a backup / snapshot is being taken. One the diagnostic side though, we definitely want to know what happened when something goes wrong, so there's probably a conflict there. The other downside is such notification may not be possible since many of these problems occur with 3rd party add-ons. Then again, the leading virtualization vendor has recently open sourced their agent so it may be possible. On problem #2, this patchset or something similar will help only to hone in on who is to blame faster, but it won't fix anything of course. In closing, I want to point out that based on the above use case, a simple one line printk in just the freeze path would help. That is, if we just knew who the last process was which issued a freeze on a given filesystem was, and that was always present in any system log / vmcore, we'd be able to help customers in a more definitive way. All this said, I don't think Frank, myself or anyone else working on such problems for customers wants to add slightly better diagnostics, but create a nightmare of maintenance of tracepoints or unknown side effects that outweigh any benefit. -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
