I'll leave it to others to figure out the exact details. But afaict it should be possible to have fine-grained-randomization and preserve the workaround in the end.
the most obvious "solution" is to compile with an alignment of 4 bytes (so tight packing) and then in the randomizer preserve the offset within 32 bytes, no matter what it is that would get you an average padding of 16 bytes which is a bit more than now but not too insane (queue Kees' argument that tiny bits of padding are actually good)
