On Tue, Oct 26, 2021 at 08:59:44PM +0300, Amir Goldstein wrote: > On Tue, Oct 26, 2021 at 7:18 PM Vivek Goyal <vgoyal@xxxxxxxxxx> wrote: > > > > On Tue, Oct 26, 2021 at 06:23:50PM +0300, Amir Goldstein wrote: > > > > [..] > > > > 3) The lifetime of the local watch in the guest kernel is very > > > > important. Specifically, there is a possibility that the guest does not > > > > receive remote events on time, if it removes its local watch on the > > > > target or deletes the inode (and thus the guest kernel removes the watch). > > > > In these cases the guest kernel removes the local watch before the > > > > remote events arrive from the host (virtiofsd) and as such the guest > > > > kernel drops all the remote events for the target inode (since the > > > > corresponding local watch does not exist anymore). > > > > So this is one of the issues which has been haunting us in virtiofs. If > > a file is removed, for local events, event is generated first and > > then watch is removed. But in case of remote filesystems, it is racy. > > It is possible that by the time event arrives, watch is already gone > > and application never sees the delete event. > > > > Not sure how to address this issue. > > Can you take me through the scenario step by step. > I am not sure I understand the exact sequence of the race. Ioannis, please correct me If I get something wrong. You know exact details much more than me. A. Say a guest process unlinks a file. B. Fuse sends an unlink request to server (virtiofsd) C. File is unlinked on host. Assume there are no other users so inode will be freed as well. And event will be generated on host and watch removed. D. Now Fuse server will send a unlink request reply. unlink notification might still be in kernel buffers or still be in virtiofsd or could be in virtiofs virtqueue. E. Fuse client will receive unlink reply and remove local watch. Fuse reply and notification event are now traveling in parallel on different virtqueues and there is no connection between these two. And it could very well happen that fuse reply comes first, gets processed first and local watch is removed. And notification is processed right after but by then local watch is gone and filesystem will be forced to drop event. As of now situation is more complicated in virtiofsd. We don't keep file handle open for file and keep an O_PATH fd open for each file. That means in step D above, inode on host is not freed yet and unlink event is not generated yet. When unlink reply reaches fuse client, it sends FORGET messages to server, and then server closes O_PATH fd and then host generates unlink events. By that time its too late, guest has already remove local watches (and triggered removal of remote watches too). This second problem probably can be solved by using file handles, but basic race will still continue to be there. > If it is local file removal that causes watch to be removed, > then don't drop local events and you are good to go. > Is it something else? - If remote events are enabled, then idea will be that user space gets and event when file is actually removed from server, right? Now it is possible that another VM has this file open and file has not been yet removed. So local event only tells you that file has been removed in guest VM (or locally) but does not tell anything about the state of file on server. (It has been unlinked on server but inode continues to be alive internall). - If user receives both local and remote delete event, it will be confusing. I guess if we want to see both the events, then there has to be some sort of info in event which classifies whether event is local or remote. And let application act accordingly. Thanks Vivek
