On Thu, 6 Sep 2018, Noah Watkins wrote: > On Thu, Sep 6, 2018 at 4:30 PM Sage Weil <sweil@xxxxxxxxxx> wrote: > > > > The problem is that those generate_test_instances are (1) sparse and > > minimal, and (2) would require huge developer investment to fill in with > > "realistic" instances, and (3) even though wouldn't necessarily be > > If the goal (and maybe i'm missing the point in this thread) is to > check that old/new code can read new/old objects, what are some > examples of unrealistic instances that would throw a wrench in the > testing? The case that comes to mind is when decoding behavior is a > function of the content being decoded (beyond the standard struct > version info). If the testing extends to higher levels where the > relationship between values in a decoded object can be checked for > consistency / sanity then this makes sense but I didn't get that from > reading through the thread. Right now the object corpus tests only verify that we can decode all prior versions. And there is a special way to indicate that an incompatible encoding change has been made (that tends to happens at once for most major releases it seems). And all the test is doing is verifying that we can decode without throwing an exception. Surprising it catches lots of careless errors. This could expand that to ensure that newer object instances are decodable by older versions. Mostly that just will just make sure the struct length stuff is working properly, so probably not likely to catch much. When encoding changes are feature dependent we could do more: we could extend the object corpus to include objects encoded with an older release's feature bits. That's a bit more involved. As is often the case, though, just because something should always work doesn't mean it will when you test it :). This probably isn't the highest-value time investment, but it does also help address the problem of forgetting to add objects from every release to the corpus (which is traditionally an annoying manual process), so I like it... sage
