On Tue, Oct 03, 2017 at 02:10:57PM -0400, R. Jason Adams wrote: > Reformated the drive and it's refilling. With the changes > suggested (100 dir, 512 nodes, defaults) it already seems better. > We’re currently at 6% full and the reads are quite a bit less > than they were before at similar fullness. Ok, that's good - we're making progress :) > One thing I’m > noticing in Grafana, the read request/s seem to keep increasing > (up to ~8/s) for around an 15 minutes, then they drop down 1/s for > 10-15 minutes.. then over the next 15 minutes they build back up.. > etc I'm still curious as to what the reads are - can you run a blktrace to try to capture some of them? That will give us a better idea of what might be done to further reduce this. It might be worth checking what is happening with the xfs inode cache and overall memory usage and see if this change in read behaviour correlates with memory reclaim being active? > > FWIW, how much RAM do you have in the system, and what does > > 'echo 200 > /proc/sys/fs/xfs/xfssyncd_centisecs' do to the > > behaviour? > > System has 24G of ram. I’m guessing a move to 96 or 192G would > help a lot.. in the end the system will have 36 of these 10TB > drives. 24G should be enough if you're not walking the entire data set periodically. The working set of inodes should be pretty small, so the amount of memory being used to cache the working set shouldn't be huge nor cause much trouble in terms of turnover. Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
