On Mon, May 21 2018 at 10:52am -0400, Jens Axboe <axboe@xxxxxxxxx> wrote: > On 5/21/18 8:47 AM, Mike Snitzer wrote: > > On Mon, May 21 2018 at 10:36am -0400, > > Jens Axboe <axboe@xxxxxxxxx> wrote: > > > >> On 5/21/18 8:31 AM, Mike Snitzer wrote: > >>> On Mon, May 21 2018 at 10:19am -0400, > >>> Jens Axboe <axboe@xxxxxxxxx> wrote: > >>> > >>>> On 5/21/18 8:03 AM, Mike Snitzer wrote: > >>>>> On Sun, May 20 2018 at 6:25pm -0400, > >>>>> Kent Overstreet <kent.overstreet@xxxxxxxxx> wrote: > >>>>> > >>>>>> Jens - this series does the rest of the conversions that Christoph wanted, and > >>>>>> drops bioset_create(). > >>>>>> > >>>>>> Only lightly tested, but the changes are pretty mechanical. Based on your > >>>>>> for-next tree. > >>>>> > >>>>> By switching 'mempool_t *' to 'mempool_t' and 'bio_set *' to 'bio_set' > >>>>> you've altered the alignment of members in data structures. So I'll > >>>>> need to audit all the data structures you've modified in DM. > >>>>> > >>>>> Could we get the backstory on _why_ you're making this change? > >>>>> Would go a long way to helping me appreciate why this is a good use of > >>>>> anyone's time. > >>>> > >>>> Yeah, it's in the first series, it gets rid of a pointer indirection. > >>> > >>> "Allows mempools to be embedded in other structs, getting rid of a > >>> pointer indirection from allocation fastpaths." > >>> > >>> So this is about using contiguous memory or avoiding partial allocation > >>> failure? Or both? > >>> > >>> Or more to it? Just trying to fully appreciate the theory behind the > >>> perceived associated benefit. > >> > >> It's about avoiding a pointer indirection. Instead of having to > >> follow a pointer to get to that struct, it's simple offset math off > >> your main structure. > >> > >>> I do think the increased risk of these embedded bio_set and mempool_t > >>> themselves crossing cachelines, or struct members that follow them doing > >>> so, really detracts from these types of changes. > >> > >> Definitely something to look out for, though most of them should be > >> per-dev structures and not in-flight structures. That makes it a bit > >> less sensitive. But can't hurt to audit the layouts and adjust if > >> necessary. This is why it's posted for review :-) > > > > This isn't something that is easily caught upfront. Yes we can all be > > busy little beavers with pahole to audit alignment. But chances are > > most people won't do it. > > > > Reality is there is potential for a regression due to false sharing to > > creep in if a hot struct member suddenly starts straddling a cacheline. > > That type of NUMA performance killer is pretty insidious and somewhat > > tedious to hunt down even when looking for it with specialized tools: > > https://joemario.github.io/blog/2016/09/01/c2c-blog/ > > IMHO you're making a big deal out of something that should not be. I raised an issue that had seemingly not been considered at all. Not making a big deal. Raising it for others' benefit. > If the dm bits are that sensitive and cache line honed to perfection > already due to previous regressions in that area, then it might > not be a bad idea to have some compile checks for false cacheline > sharing between sensitive members, or spilling of a sub-struct > into multiple cachelines. > > It's not like this was pushed behind your back. It's posted for > review. It's quite possible the net change is a win for dm. Let's > focus on getting it reviewed, rather than pontificate on what > could potentially go all wrong with this. Why are you making this personal? Or purely about DM? I'm merely pointing out this change isn't something that can be given a quick blanket "looks good". Mike
