On Mon, Feb 11, 2013 at 7:02 PM, Pranith Kumar K <pkarampu@xxxxxxxxxx> wrote:
hi,
Problem:
When there are multiple fds writing to same file with eager-lock enabled, the fd which acquires the eager-lock waits for
post-op-delay secs before doing the unlock. Because of this all other fds opened on the file face extra delay when
performing writes. Eager-locking, post-op-delay need to be disabled when there are multiple fds opened on the file.
Here is the profile info output for the case above:
Execute the following command on the mount point.
for n in `seq 1 50` ; do eval "/home/pranithk/workspace/gerrit-repo/append2log.py ./ben.log 10000 0.001 &" ; done ; wait
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 50 RELEASE
0.00 0.00 us 0.00 us 0.00 us 60 RELEASEDIR
0.00 55.00 us 55.00 us 55.00 us 1 GETXATTR
0.00 31.50 us 27.00 us 36.00 us 2 STATFS
0.00 41.00 us 29.00 us 53.00 us 2 ENTRYLK
0.00 198.00 us 198.00 us 198.00 us 1 CREATE
0.00 124.00 us 108.00 us 140.00 us 2 READDIR
0.00 27.04 us 17.00 us 95.00 us 49 OPEN
0.00 74.89 us 13.00 us 206.00 us 47 STAT
0.01 87.02 us 11.00 us 391.00 us 50 FLUSH
0.01 102.43 us 20.00 us 268.00 us 60 OPENDIR
0.02 344.27 us 22.00 us 940.00 us 44 WRITE
0.02 228.80 us 52.00 us 345.00 us 82 FXATTROP
0.03 199.89 us 19.00 us 404.00 us 120 READDIRP
0.05 91.41 us 23.00 us 832.00 us 421 LOOKUP
99.86 632698.45 us 17.00 us 1999724.00 us 126 FINODELK
Observe that most of the delay is in FINODELK fop.
Possible Solution:
With the patch: http://review.gluster.org/4468 we started maintaining open-fd count in the inode. We need to implement xdata based xattr retrieval in write-fop and get open-fd-count in write fop. Remember the open-fd-count received from the write-fops and maintain it in afr-fd-ctx. If the open-fd count is >1 post-op-delay is immediately disabled for that write fop. All write-fops take into consideration this count to determine whether to enable eager-lock, post-op-delay for that write fop.
Let me know if you foresee any issues with this approach.
https://bugzilla.redhat.com/show_bug.cgi?id=910217 is tracking this issue.
Ideally you would want open-fd count to be retrieved in all fops, and only when an eager lock has been acquired. Any fop callback's xattr_rsp inspection should potentially wake up the sleeping post-op-delay in that inode (and disable further eager locking temporarily).
Avati
