On Fri, Nov 05 2021, Emily Shaffer wrote: > On Fri, Oct 29, 2021 at 10:31:10AM -0700, Jonathan Tan wrote: > [...] >> >> One alternative is to rerun the config parsing mechanism upon noticing >> the first URL-conditional include in order to find all URLs. This would >> require the config files to be read from disk twice, though. > > What's the easiest way to "try it and see", to add tooling and find out > whether the config files would be reopened during the second parse? > Because I suspect that we won't actually reopen those files, due to the > config cache. strace -f? > So couldn't we do something like.... > > pass #1: > if (include) > if (not hasRemoteUrl) > open up path & parse > put config into in-memory cache normally > pass #2: (and this pass would need to be added to repo_config() probably) > if (include) > if (hasRemoteUrl) > open up path & parse > insert in-order into in-memory cache > don't touch existing configs otherwise > > I think it's in practice similar to the approach you're using (getting > around the weird ordering with a cache in memory), but we could reuse > the existing config cache rather than creating a new and different one. I don't know enough to say if this two-step approach is better (although I'm slightly biased in that direction, since it seems simpler), but this just seems like premature optimization. I.e. let's just read the files twice, they'll be in the OS's FS cache, which is unlikely to be a bottleneck for the amount of files involved. That being said we do have exactly this cache already. See [1] and 3c8687a73ee (add `config_set` API for caching config-like files, 2014-07-28). But I think that was added due to *very* frequent re-parsing of the entire config every time someone needed a config variable, not due to the I/O overhead (but I may be wrong). So if we've got 100 config variables we need and 10 config files then 10*100 is probably starting to hurt, but if for whatever reason we needed 2*10 here that's probably no big deal, and in any case would only happen if this new include mechanism was in play. 1. https://lore.kernel.org/git/1404631162-18556-1-git-send-email-tanayabh@xxxxxxxxx/
