> Hello, > > I had found recently that the OSD daemons under certain conditions > (moderate vm pressure, moderate I/O, slightly altered vm settings) can > go into loop involving isolate_freepages and effectively hit Ceph > cluster performance. Hi! I'm the creator of the server fault issue you reference: http://serverfault.com/questions/642883/cause-of-page-fragmentation-on-large-server-with-xfs-20-disks-and-ceph I'd like to get to the bottom of this very much, I'm seeing a very similar pattern on 3.10.0-123.9.3.el7.x86_64, if this is fixed in later versions perhaps we could backport something. Here is some perf output: http://ponies.io/raw/compaction.png Looks pretty similar. I also have hundreds of MB logs and traces should we need some specific question answered. I've managed to reproduce many failed compactions with this: https://gist.github.com/christian-marie/cde7e80c5edb889da541 I took some compaction stress test code and bolted on a little loop to mmap a large sparse file and read every PAGE_SIZEth byte. Run it once, compactions seem to do okay, run it again and they're really slow. This seems to be because my little trick to fill up cache memory only seems to work exactly half the time. Note that transhuge pages are only used to introduce fragmentation/pressure here, turning transparent huge pages off doesn't seem to make the slightest difference to the spinning-in-reclaim issue. We are using Mellanox ipoib drivers which do not do scatter-gather, so I'm currently working on adding support for that (the hardware supports it). Are you also using ipoib or have something else doing high order allocations? It's a bit concerning for me if you don't as it would suggest that cutting down on those allocations won't help.
Attachment:
pgpvIv6SnIUlG.pgp
Description: PGP signature