On 4/21/22 10:54 PM, David Howells wrote: > Jeffle Xu <jefflexu@xxxxxxxxxxxxxxxxx> wrote: > >> + fd_install(fd, file); > > Do you need to mitigate potential EMFILE/ENFILE problems? You're potentially > trebling up the number of accounted systemwide fds: one for erofs itself, one > anonfd per cache object file to communicate with the daemon and one in the > daemon to talk to the server. Cachefiles has a fourth internally, but it's > kept off the books - further, cachefiles closes them fairly quickly after a > period of nonuse. > Hi, thanks for pointing it out. 1. Actually in our using scenarios, one erofs filesystem is formed of several chunk-deduplicated blobs (which are really cached by Cachefiles), while each blob can contain many files of erofs. For example, one container image for node.js will correspond to ~20 blob files in total. Only these blob files are cached by Cachefiles. In densely employed environment, there could be hundreds of containers and thus thousands of backing files on one machine. That is, only tens of thousands of fds/files is needed in this case. 2. Our user daemon will configure rlimit-nofile to a reasonably large (e.g. 1 million) value, so that it won't fail when trying to allocate fds. https://github.com/dragonflyoss/image-service/blob/master/src/bin/nydusd/main.rs#L152 3. Our user daemon will close the anonymous fd once the corresponding backing file has fully downloaded, to free the fd resources. 4. Even if fd/file allocation fails (in cachefiles_ondemand_get_fd()), the INIT request will fail, and thus the erofs mount will fail then. That is, it won't break the upper erofs in this case. 5. If later we find that the number of fds/files is indeed an issue, then we also plan to make the user daemon close some fds to spare some free resources. And then the Cachefiles kernel module needs to reallocate an anonymous fd for the backing file when cache miss. But it remains to be done later if it's really needed. -- Thanks, Jeffle
