Changes since v5: In cachefiles_daemon_poll(), replace xa_for_each_marked with xas_for_each_marked. [Background] ============ In ondemand read mode, if user daemon closes anonymous fd(e.g. daemon crashes), subsequent read and inflight requests based on these fd will return -EIO. Even if above mentioned case is tolerable for some individual users, but when it happenens in real cloud service production environment, such IO errors will be passed to cloud service users and impact its working jobs. It's terrible for cloud service stability. [Design] ======== The main idea of daemon failover is reopen the inflight req related object, thus the newly started daemon could process the req as usual. To implement that, we need to support: 1. Store inflight requests during daemon crash. 2. Hold the handle of /dev/cachefiles(by container snapshotter/systemd). BTW, if user chooses not to keep /dev/cachefiles fd, failover is not enabled. Inflight requests return error and passed it to container.(same behavior as now). [Flow Path] =========== This patchset introduce three states for ondemand object: CLOSE: Object which just be allocated or closed by user daemon. OPEN: Object which related OPEN request has been processed correctly. REOPENING: Object which has been closed, and is drived to open by a read request. 1. Daemon use UDS send/receive fd to keep and pass the fd reference of "/dev/cachefiles". 2. User daemon crashes -> restart and recover dev fd's reference. 3. User daemon write "restore" to device. 2.1 Reset the object's state from CLOSE to REOPENING. 2.2 Init a work which reinit the object and add it to wq. (daemon can get rid of kernel space and handle that open request). 4. The user of upper filesystem won't notice that the daemon ever crashed since the inflight IO is restored and handled correctly. [Test] ====== There is a testcase for above mentioned scenario. A user process read the file by fscache ondemand reading. At the same time, we kill the daemon constantly. The expected result is that the file read by user is consistent with original, and the user doesn't notice that daemon has ever been killed. https://github.com/userzj/demand-read-cachefilesd/commits/failover-test [GitWeb] ======== https://github.com/userzj/linux/tree/fscache-failover-v6 RFC: https://lore.kernel.org/all/20220818135204.49878-1-zhujia.zj@xxxxxxxxxxxxx/ V1: https://lore.kernel.org/all/20221011131552.23833-1-zhujia.zj@xxxxxxxxxxxxx/ V2: https://lore.kernel.org/all/20221014030745.25748-1-zhujia.zj@xxxxxxxxxxxxx/ V3: https://lore.kernel.org/all/20221014080559.42108-1-zhujia.zj@xxxxxxxxxxxxx/ V4: https://lore.kernel.org/all/20230111052515.53941-1-zhujia.zj@xxxxxxxxxxxxx/ V5: https://lore.kernel.org/all/20230329140155.53272-1-zhujia.zj@xxxxxxxxxxxxx/ Jia Zhu (5): cachefiles: introduce object ondemand state cachefiles: extract ondemand info field from cachefiles_object cachefiles: resend an open request if the read request's object is closed cachefiles: narrow the scope of triggering EPOLLIN events in ondemand mode cachefiles: add restore command to recover inflight ondemand read requests fs/cachefiles/daemon.c | 15 +++- fs/cachefiles/interface.c | 7 +- fs/cachefiles/internal.h | 59 +++++++++++++- fs/cachefiles/ondemand.c | 166 ++++++++++++++++++++++++++++---------- 4 files changed, 201 insertions(+), 46 deletions(-) -- 2.20.1 -- Linux-cachefs mailing list Linux-cachefs@xxxxxxxxxx https://listman.redhat.com/mailman/listinfo/linux-cachefs