On Tue 01-08-17 15:02:42, Jan Kara wrote: > Hi Andrew, > > On Fri 23-06-17 02:43:44, Andrew Perepechko wrote: > > The original workload was 50 threads sequentially creating files, each > > > > thread in its own directory, over a fast RAID array. > > OK, I can reproduce this. Actually I can reproduce on normal SATA drive. > Originally I've tried on ramdisk to simulate really fast drive but there > dq_list_lock and dq_data_lock contention is much more visible and the > contention on dqio_mutex is minimal (two orders of magnitude smaller). On > SATA drive we spend ~45% of runtime contending on dqio_mutex when creating > empty files. So this was just me misinterpretting lockstat data (forgot to divide the wait time by number of processes) - then the result would be that each process waits only ~1% of its runtime for dqio_mutex. Anyway, my patches show ~10% improvement in runtime when 50 different processes create empty files for 50 different users. As expected there's not measurable benefit when all processes create files for the same user. > The problem is that if it is single user that is creating all these files, > it is not clear how we could do much better - all processes contend to > update the same location on disk with quota information for that user and > they have to be synchronized somehow. If there are more users, we could do > better by splitting dqio_mutex on per-dquot basis (I have some preliminary > patches for that). > > One idea I have how we could make things faster is that instead of having > dquot dirty flag, we would have a sequence counter. So currently dquot > modification looks like: > > update counters in dquot > dquot_mark_dquot_dirty(dquot); > dquot_commit(dquot) > mutex_lock(dqio_mutex); > if (!clear_dquot_dirty(dquot)) > nothing to do -> bail > ->commit_dqblk(dquot) > mutex_unlock(dqio_mutex); > > When several processes race updating the same dquot, they very often all > end up updating dquot on disk even though another process has already > written dquot for them while they were waiting for dqio_sem - in my test > above the ratio of commit_dqblk / dquot_commit calls was 59%. What we could > do is that dquot_mark_dquot_dirty() would return "current sequence of > dquot", dquot_commit() would then get sequence that is required to be > written and if that is already written (we would also store in dquot latest > written sequence), it would bail out doing nothing. This should cut down > dqio_mutex hold times and thus wait times but I need to experiment and > measure that... I've been experimenting with this today but this idea didn't bring any benefit in my testing. Was your setup with multiple users or a single user? Could you give some testing to my patches to see whether they bring some benefit to you? Honza -- Jan Kara <jack@xxxxxxxx> SUSE Labs, CR
