On Tue, Feb 26, 2013 at 9:57 AM, Kevin Decherf <kevin@xxxxxxxxxxxx> wrote: > On Tue, Feb 19, 2013 at 05:09:30PM -0800, Gregory Farnum wrote: >> On Tue, Feb 19, 2013 at 5:00 PM, Kevin Decherf <kevin@xxxxxxxxxxxx> wrote: >> > On Tue, Feb 19, 2013 at 10:15:48AM -0800, Gregory Farnum wrote: >> >> Looks like you've got ~424k dentries pinned, and it's trying to keep >> >> 400k inodes in cache. So you're still a bit oversubscribed, yes. This >> >> might just be the issue where your clients are keeping a bunch of >> >> inodes cached for the VFS (http://tracker.ceph.com/issues/3289). >> > >> > Thanks for the analyze. We use only one ceph-fuse client at this time >> > which makes all "high-load" commands like rsync, tar and cp on a huge >> > amount of files. Well, I will replace it by the kernel client. >> >> Oh, that bug is just an explanation of what's happening; I believe it >> exists in the kernel client as well. > > After setting the mds cache size to 900k, storms are gone. > However we continue to observe high latency on some clients (always the > same clients): each IO takes between 40 and 90ms (for example with > Wordpress, it takes ~20 seconds to load all needed files...). > With a non-laggy client, IO requests take less than 1ms. I can't be sure from that description, but it sounds like you've got one client which is generally holding the RW "caps" on the files, and then another client which comes in occasionally to read those same files. That requires the first client to drop its caps, and involves a couple round-trip messages and is going to take some time — this is an unavoidable consequence if you have clients sharing files, although there's probably still room for us to optimize. Can you describe your client workload in a bit more detail? -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
