On Wed, Aug 21, 2019 at 09:04:57AM +0800, Joseph Qi wrote: > Hi Ted， > > On 19/8/21 00:08, Theodore Y. Ts'o wrote: > > On Tue, Aug 20, 2019 at 11:00:39AM +0800, Joseph Qi wrote: > >> > >> I've tested parallel dio reads with dioread_nolock, it doesn't have > >> significant performance improvement and still poor compared with reverting > >> parallel dio reads. IMO, this is because with parallel dio reads, it take > >> inode shared lock at the very beginning in ext4_direct_IO_read(). > > > > Why is that a problem? It's a shared lock, so parallel threads should > > be able to issue reads without getting serialized? > > > The above just tells the result that even mounting with dioread_nolock, > parallel dio reads still has poor performance than before (w/o parallel > dio reads). > > > Are you using sufficiently fast storage devices that you're worried > > about cache line bouncing of the shared lock? Or do you have some > > other concern, such as some other thread taking an exclusive lock? > > > The test case is random read/write described in my first mail. And Regardless of dioread_nolock, ext4_direct_IO_read() is taking inode_lock_shared() across the direct IO call. And writes in ext4 _always_ take the inode_lock() in ext4_file_write_iter(), even though it gets dropped quite early when overwrite && dioread_nolock is set. But just taking the lock exclusively in write fro a short while is enough to kill all shared locking concurrency... > from my preliminary investigation, shared lock consumes more in such > scenario. If the write lock is also shared, then there should not be a scalability issue. The shared dio locking is only half-done in ext4, so perhaps comparing your workload against XFS would be an informative exercise... Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx
