On Wed, Jan 9, 2019 at 8:29 PM Keith Packard <keithp@xxxxxxxxxx> wrote: > > Akira TAGOH <akira@xxxxxxxxx> writes: > > > As of the discussion on the list, Keith's changes doesn't address the > > original purpose - allow sharing caches on bind-mounts in flatpaks. > > particularly for the case where flatpaks uses same location in > > sandbox. > > I'm probably forgetting a bunch of context here, but I think the problem > was that flatpaks have /run/host/fonts pointing to the "real" > /usr/share/fonts and then a separate /usr/share/fonts of their own with > a small set of fonts, and so you end up with collisions in the cache > file namespace as both directories end up generating the same cache file > name. Yes, and there is two aspects of this problems with this: A sandbox has something in /usr/share/fonts that isn't the same as what is on the host, but the cache is shared, so we pick up host cache for the sandbox directory (or the reverse), causing misrendering of fonts and whatnot. A version of the host /usr/share/fonts is available to the sandbox under a different pathname, causing the host cache to not be used and thus have to be regenerated, which is slow first time the app is started. I'd like to repeat that this is not really flatpak specific as such. The issue can happen in multiple cases like nfs mounts, multi-boot systems, docker containers, etc. Also, the focus on /usr/share/fonts here is mainly for illustration, the same issue could happen with any other path. For instance, separate containers with shared $HOME would have its font path (whatever it is as long as it is the same) conflict between the containers. Flatpak has an additional weakness here, which is that we don't store mtimes (to maximize content sharing abilities the mtime is not part of the content addressing). This means the mtime can't be used to detect a stale cache, so we use the uuid to detect such changes. > Making the build reproducible means having all content generated > deterministically based only on the source package and toolchain. The > current UUID files are generated randomly making them > non-deterministic. > > For the font cache, making it reproducible requires that the keys > mapping directories to cache filenames be the same each time the cache > is built. This means we cannot use the current randomly generated UUID > values and also have a reproducible system. I think this misrepresents what the UUID is for. The UUID represents a uniqe identifier for the *location*, not the contents, and the goal is to make it independent on how you found the directory. If you add fonts to that directory, then you're supposed to keep the UUID because you want to regenerate the same cache for the new content. I realize this is not what the reproducible builds project wants, but it is what the UUID was added for. > I considered whether we might provide a mechanism to generate UUID > values deterministically for purposes of packaging. However, this would > mean that we couldn't use these same packages when creating a flatpak as > the deterministic UUID values would collide if those same packages were > used in the outer system. I think this furthers the misunderstanding from above, but lets continue this idea. If the UUID really *was* content addressed, then it would change each time some font was added, and old font caches would become stale (and reaped via some other way like mtimes). In this case the fact that caches between the sandbox and the hosts collide is not even a problem, since the cached data is identical and could be shared. The problem is rather that the font directory is mutable, and if it changes without immediately updating the uuid you run into issues. > Without deterministic UUID values, I'm left with the feeling that > our only available solutions involve changing how flatpaks reference > fonts. > > If we agree that a solution to this involves changing the flatpak > mechanism, I'd like to suggest that the most straightforward fix for the > overall system would be to expose the external fonts using the external > path names -- bind mounting the external /usr/share/fonts as > /usr/share/fonts within the flatpak, and creating a new > /usr/share/fonts-minimal (or whatever) to hold the fonts provided by the > flatpak itself. With this change, we can simply delete the UUID code > from fontconfig and go back to using global font paths as keys to the > font cache database. I'm willing to make *some* changes to flatpak, but I'm not sure this is the right approach. First of all it just looks at a tiny subset of the problem (only flatpak, and only one directory). Secondly, it is likely to run into issues having non-standard paths. For example, the fedora flatpak runtime is created from the standard fedora rpms, so it will have to be tweaked post install, and its possible that some code hard-coded /usr/share/fonts/some-specific-font which is in the app but not the host.. Also, we'll be guaranteeing that caches for /usr/share/fonts and /usr/share/fonts-minimal don't conflict, but there is no guarantee that different versions of /usr/share/fonts-minimal don't conflict. In flatpak we set XDG_CACHE_HOME separate for each app, so this doesn't happen cross-apps like it would with e.g. docker. However it will cause flatpak to fail to detect an update of /usr/share/fonts-minimal due to the mtime issue. I can imagine changing flatpak to modify the mtime of the fonts directory after install (since only files are hardlink-shared between apps, not directories), but this would cause problems for all outstanding flatpak deployments that don't do this. As a side point, flatpak also maps /run/host/user-fonts, which points to the ~/.fonts or ~/.local/share/fonts directory even when the app doesn't have regular homedir access. It would similarly have to be changed to use the real path if the above approach is used. > I'd love to hear about alternative ideas which might lead to solutions > that make builds involving fontconfig reproducible. I'd be happy to take > even vague hints at this point; all I've got at this point are a > collection of dead ends. Here is my proposal: Make the uuid *generation* optional and manual. Then, when we create the flatpak runtime we run fc-cache --make-uuid (or something) to generate the uuid files. Then fontconfig would never confuse the sandboxed /usr/share/fonts with any other, and since we would get a new uuid each time we regenerated the runtime it would correctly pick up stale caches when we update the runtime (even with no mtime change). This would make the default installation of fontconfig reproducible, and it would solve the first problem (don't mix up sandboxed and host font dirs). It would also let you opt-in to the uuid in other cases where it makes sense. For instance, you could have a uuid file on a NFS share or USB drive font dir, so that any caches for it will always be the same no matter how it happens to be mounted. We still wouldn't have a way to reuse host caches which were mounted in a different way, but if we assume all conflicting directories use uuids (like they would in the flatpak case), then we could solve this in a pretty simple way by a config file saying "treat all instances of /run/host/fonts as /usr/share/fonts", and I could make flatpak generate such a file. _______________________________________________ Fontconfig mailing list Fontconfig@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/fontconfig
