Hi Andrew, On Fri 23-06-17 02:43:44, Andrew Perepechko wrote: > The original workload was 50 threads sequentially creating files, each > > thread in its own directory, over a fast RAID array. OK, I can reproduce this. Actually I can reproduce on normal SATA drive. Originally I've tried on ramdisk to simulate really fast drive but there dq_list_lock and dq_data_lock contention is much more visible and the contention on dqio_mutex is minimal (two orders of magnitude smaller). On SATA drive we spend ~45% of runtime contending on dqio_mutex when creating empty files. The problem is that if it is single user that is creating all these files, it is not clear how we could do much better - all processes contend to update the same location on disk with quota information for that user and they have to be synchronized somehow. If there are more users, we could do better by splitting dqio_mutex on per-dquot basis (I have some preliminary patches for that). One idea I have how we could make things faster is that instead of having dquot dirty flag, we would have a sequence counter. So currently dquot modification looks like: update counters in dquot dquot_mark_dquot_dirty(dquot); dquot_commit(dquot) mutex_lock(dqio_mutex); if (!clear_dquot_dirty(dquot)) nothing to do -> bail ->commit_dqblk(dquot) mutex_unlock(dqio_mutex); When several processes race updating the same dquot, they very often all end up updating dquot on disk even though another process has already written dquot for them while they were waiting for dqio_sem - in my test above the ratio of commit_dqblk / dquot_commit calls was 59%. What we could do is that dquot_mark_dquot_dirty() would return "current sequence of dquot", dquot_commit() would then get sequence that is required to be written and if that is already written (we would also store in dquot latest written sequence), it would bail out doing nothing. This should cut down dqio_mutex hold times and thus wait times but I need to experiment and measure that... Honza > > On Fri 03-03-17 11:08:42, Jan Kara wrote: > > > > Hello! > > > > > > > > On Thu 02-02-17 15:23:44, Andrew Perepechko wrote: > > > > > We have a heavy metadata related workload (ext4, quota journalling) > > > > > and profiling shows that there's significant dqio_mutex contention. > > > > > > > > > > From the quota code, it looks like every time dqio_mutex is taken > > > > > it protects access to only one quota file. > > > > > > > > > > Is it possible to split dqio_mutex for each of MAXQUOTAS so that > > > > > e.g. 2 parallel dquot_commit()'s can be running for user and group > > > > > quota update? Am I missing any dqio_mutex function that requires > > > > > dqio_mutex to be monolithic? > > > > > > > > So we can certainly make dqio_mutex less heavy. Making it per-quota-type > > > > would OK but I suspect it will not bring a big benefit. What would likely > > > > be more noticeable is if we avoided dqio_mutex for updates of quota > > > > information - that should not be that hard to do since we update that > > > > in-place and so don't really need the serialization for anything > > > > substantial. However we will need some restructuring of the code to make > > > > such locking scheme possible in a clean way... > > > > > > So I'm experimenting with some patches. However I have trouble creating > > > a workload where quota updates would show significant overhead. Can you > > > share which workload is problematic for you? Thanks! > > > > > > Honza > > > > > -- Jan Kara <jack@xxxxxxxx> SUSE Labs, CR