Re: MDS has inconsistent performance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Let me know if this works and/or you need anything else:

https://www.dropbox.com/s/fq47w6jebnyluu0/lookup-logs.tar.gz?dl=0

Beware - the clients were on debug=10. Also, I tried this with the
kernel client and it is more consistent; it does the 2 lookups per
create on 1 client every single time.

On Thu, Jan 15, 2015 at 11:28 AM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
> Can you post the full logs somewhere to look at? These bits aren't
> very helpful on their own (except to say, yes, the client cleared its
> I_COMPLETE for some reason).
>
> On Tue, Jan 13, 2015 at 3:45 PM, Michael Sevilla <mikesevilla3@xxxxxxxxx> wrote:
>> On Tue, Jan 13, 2015 at 11:13 AM, Gregory Farnum <greg@xxxxxxxxxxx> wrote:
>>> On Mon, Jan 12, 2015 at 10:17 PM, Michael Sevilla
>>> <mikesevilla3@xxxxxxxxx> wrote:
>>>> I can't get consistent performance with 1 MDS. I have 2 clients create
>>>> 100,000 files (separate directories) in a CephFS mount. I ran the
>>>> experiment 5 times (deleting the pools/fs and restarting the MDS in
>>>> between each run). I graphed the metadata throughput (requests per
>>>> second): https://github.com/michaelsevilla/mds/blob/master/graphs/thruput.png
>>>
>>> So that top line is ~20,000 processed requests/second, as measured at
>>> the MDS? (Looking at perfcounters?) And the fast run is doing 10k
>>> create requests/second? (This number is much higher than I expected!)
>>
>> Yes - top line was 20K req/s from perf counter dump and the fast run
>> does about 13K creates/s. We were surprised, too... In fact, the
>> performance of 1 client per MDS gives us similar performance to
>> IndexFS - a system that came out in a paper at Supercomputing this
>> year. Here is a throughput graph, normalized to the # of clients, that
>> shows how powerful one MDS can actually be:
>> https://github.com/michaelsevilla/mds/blob/master/graphs/thruput-norm.png
>>
>> Keep in mind that runs with more than 1 client aren't creates/s, but ops/sec. ;)
>>
>>>
>>>> Sometimes (run0, run3), both clients issue 2 lookups per create to the
>>>> MDS - this makes throughput high but the runtime long since the MDS
>>>> processes many more requests.
>>>> Sometimes (run2, run4), 1 client does 2 lookups per create and the
>>>> other doesn't do any lookups.
>>>> Sometimes (run1), neither client does any lookups - this has the
>>>> fastest runtime.
>>>>
>>>> Does anyone know why the client behaves differently for the same exact
>>>> experiment? Reading the client logs, it looks like sometimes the
>>>> client enters add_update_cap() and clears the inode->flags in
>>>> check_cap_issue(), then when a lookup occurs (in _lookup()), the
>>>> client can't return ENOENT locally -- forcing it ask the MDS to do the
>>>> lookup. But this only happens sometimes (e.g., run0 and run3).
>>>
>>> If you provide the logs I can check more carefully, but my guess is
>>> that you've got another client mounting it, or are looking at both
>>> directories from one of the clients, and this is inadvertently causing
>>> them to go into shared rather than exclusive mode.
>>
>> I think you are right! Here is a subset of the client log:
>> https://github.com/michaelsevilla/mds/blob/master/scratch/client0.log
>>
>> These snippets are zoomed into when the client stops sending "create,
>> create, create, create..." and starts sending "lookup, lookup, create,
>> lookup, lookup, create..."
>>
>> $ cat client0.log | grep "send_request client"
>> create ...file.2098
>> create ...file.2099
>> create ...file.2100
>> create ...file.2101
>> lookup ...file.2102
>> lookup ...file.2102
>> create ...file.2102
>> lookup ...file.2103
>> lookup ...file.2103
>> create ...file.2103
>> lookup ...file.2104
>> lookup ...file.2104
>> create ...file.2104
>>
>> I think what you are looking for is on line 687:
>> ... clearing (I_COMPLETE|I_DIR_ORDERED)
>> ... add_update_cap issued pAsLsXs -> pAsLsXsFsx
>>
>> It looks like we lose the exclusive mode on the file... but I don't
>> understand why the MDS revokes it for 1 client but not the other. The
>> MDS log is here:
>> https://raw.githubusercontent.com/michaelsevilla/mds/master/scratch/mds.log
>>
>>
>>>
>>> How are you trying to keep the directories private during the
>>> workload? Some of the more naive solutions won't stand up to
>>> repetitive testing given how various components of the system
>>> currently behave.
>> Is there a way to keep the directories private (i.e. keep the always
>> in exclusive mode? That'd be perfect... In my runs, one client does
>> mkdir /mnt/cephfs/dir0 and there other does mdkir /mnt/cephfs/dir1...
>>
>>>
>>>>
>>>> Details of the experiment:
>>>> Workload: 2 clients, 100,000 creates in separate directories, using
>>>> the FUSE client
>>>> MDS config: client_cache_size = 100000000, mds_cache_size = 16384000
>>>
>>> That client_cache_size only has any effect if it's applied to the
>>> client-side config. ;)
>> Yes - I copy the ceph.conf to the client, too. I think it works
>> because the 1 client, 1 MDS test caches all the inodes, according the
>> perf counters.
>>
>> Thanks so much, Greg!
>>
>> Mike
>>
>>> -Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux