On Mon, Jan 12, 2015 at 10:17 PM, Michael Sevilla <mikesevilla3@xxxxxxxxx> wrote: > I can't get consistent performance with 1 MDS. I have 2 clients create > 100,000 files (separate directories) in a CephFS mount. I ran the > experiment 5 times (deleting the pools/fs and restarting the MDS in > between each run). I graphed the metadata throughput (requests per > second): https://github.com/michaelsevilla/mds/blob/master/graphs/thruput.png So that top line is ~20,000 processed requests/second, as measured at the MDS? (Looking at perfcounters?) And the fast run is doing 10k create requests/second? (This number is much higher than I expected!) > Sometimes (run0, run3), both clients issue 2 lookups per create to the > MDS - this makes throughput high but the runtime long since the MDS > processes many more requests. > Sometimes (run2, run4), 1 client does 2 lookups per create and the > other doesn't do any lookups. > Sometimes (run1), neither client does any lookups - this has the > fastest runtime. > > Does anyone know why the client behaves differently for the same exact > experiment? Reading the client logs, it looks like sometimes the > client enters add_update_cap() and clears the inode->flags in > check_cap_issue(), then when a lookup occurs (in _lookup()), the > client can't return ENOENT locally -- forcing it ask the MDS to do the > lookup. But this only happens sometimes (e.g., run0 and run3). If you provide the logs I can check more carefully, but my guess is that you've got another client mounting it, or are looking at both directories from one of the clients, and this is inadvertently causing them to go into shared rather than exclusive mode. How are you trying to keep the directories private during the workload? Some of the more naive solutions won't stand up to repetitive testing given how various components of the system currently behave. > > Details of the experiment: > Workload: 2 clients, 100,000 creates in separate directories, using > the FUSE client > MDS config: client_cache_size = 100000000, mds_cache_size = 16384000 That client_cache_size only has any effect if it's applied to the client-side config. ;) -Greg -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
