# State of sparsity developments and future plans (Victoria) - (Victoria) Integrating commands with sparse index, making them compatible with sparse indexes, not un-sparsifying the index before executing themselves - Have worked on some more recently - GSoC student has worked on a handful as well - Near-term future is about finding commands that need to touch the index, don't support sparse index, and then make them compatible - In some cases, that is going to require expanding the index to be non-sparse, especially if you are touching something outside of the sparsity cone - That is somewhat straightforward - More interesting questions: what is the future of sparsity? Recently, Elijah pushed a change to make sparse-checkout's cone mode the default (nice, since it is required by sparse index) - As we move forward, what should we change the defaults of? - Sparse index for sparse checkouts in cone mode? - Scalar as a testing ground for larger features, including sparse index - Could make sparse index the default in Scalar for cone-mode sparse checkouts, and then see how it goes - Or, could just go for it sooner (after we have integrated sparse index with enough commands) - A handful of internal, logistical things that would have to happen for sparse index to become the default. Currently, commands are assumed to not work with the sparse index. - Question for everybody: what is a good balance between pushing sparse index, and waiting to introduce it to more users by holding off on changing the default. - (Stolee): sparse checkout and submodules became a difficulty when mentoring their GSoC student. - (JTan): possible to decouple sparse index and cone-mode sparse checkouts from each other? This would be easy to test - turn it on, all of the test suite automatically uses it. Jrnieder: This sounds ok for the filesystem, but I don't know how this would work for this "VFS-backed Git" idea on the spreadsheet. (other things…) - (Stolee): We need cone mode today because they're the only way to definitively say that we've reached the boundary. But we can also expand the idea of "cone" to allow more paths (files instead of directories) in the cone. - (Taylor): What do we need to tell subcommands to assume that sparse index is supported? (Victoria) Gut feeling for the most part. (Jrnieder): I'd prefer this to happen sooner rather than later. This is easier for maintainability since we don't have to worry about commands being in two possible modes of operation. We can break these incompatible APIs by renaming them to prevent them from being misused by new commands. - (Victoria): So just break things that always use the full index? Sounds ok. (Stolee) This sounds similar to the_index macros, which we've tried to remove for the most part but we've stopped. Doing this conversion everywhere sounds extremely difficult - we've done an audit on this. (Jrnieder): Oh, I just meant renaming the API without changing semantics. Intentionally break everything. - (Victoria): We'd need to write new tests for lots of commands because the existing tests don't actually interact with the "sparse" parts of the index. - (Ævar): Is this just a matter of telling `git init` to initialize a sparse index? (Stolee): No, we need to force the tests to work on sparse directory entries. - (Jrnieder): This sounds like a good fit for feature.experimental - (VIctoria): Is sparse index a good git default instead of just "for large repos"? I'd think yes. (Jrnieder) Yes I think any sparse checkout user would want this. (Stolee): LIterally every command that touches the index has been converted (used for Microsoft Office monorepo), so it's just a matter of doing this for the whole project. - (Elijah): I would like partial filters from sparse patterns. --filter=blob:none doesn't let you disconnect from the server. - (Jrnieder) The DX of sparse checkout + blob:none has been pretty good. (Elijah): but you need to stay connected to the server. (Jrnieder) Ah, thanks for explaining, sorry for the confusion (Elijah) It would be great to have "sparse clone"s and have commands that work just inside of that cone when disconnected. Make "grep", "log", etc respect the sparse pattern - (Stolee): We've thought about this, but it is very expensive on the server side and makes bitmaps unusable. Alternatively, we could start with blob:none and then backfill. That sounds more promising, but that's not just a plain partial clone. - (Jonathantanmy): FYI there's a protocol feature that already allows clones to specify a sparse filter (referencing a blob with sparse patterns that's present on the server), but I don't know of any implementation that has this enabled. - (Jrnieder): Can we delete this? (Github folks): We don't like it, we invented the uploadpackfilter config to disable it :) Is this just cleanup? (Jrnieder): Yes, and this dead end will stop being a distraction. - (Peff) We could already implement most of the backfilling using current commands, but that might skip over some delta-ing optimizations. We could have a protocol change to provide the path as a hint to the server.
