On 05/21/2015 09:33 AM, Vijaikumar M wrote:
On Thursday 21 May 2015 06:48 PM, Shyam wrote:
On 05/21/2015 04:04 AM, Vijaikumar M wrote:
On Tuesday 19 May 2015 09:50 PM, Shyam wrote:
On 05/19/2015 11:23 AM, Vijaikumar M wrote:
Did that (in the attached script that I sent) and it still failed.
Please note:
- This dd command passes (or fails with EDQUOT)
- dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=512 count=10240
oflag=append oflag=sync conv=fdatasync
- We can even drop append and fdatasync, as sync sends a commit per
block written which is better for the test and quota enforcement,
whereas fdatasync does one in the end and sometimes fails (with larger
block sizes, say 1M)
- We can change bs to [512 - 256k]
- This dd command fails (or writes all the data)
- dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2 oflag=append
oflag=sync conv=fdatasync
The reasoning is that when we write a larger block size, NFS sends in
multiple 256k chunks to write and then sends the commit before the
next block. As a result if we exceed quota in the *last block* that we
are writing, we *may* fail. If we exceed quota in the last but one
block we will pass.
Hope this shorter version explains it better.
(VijayM is educating me on quota (over IM), and it looks like the
quota update happens as a synctask in the background, so post the
flush (NFS commit) we may still have a race)
Post education solution:
- Quota updates on disk xattr as a sync task, as a result if we
exceeded quota in the n-1th block there is no guarantee that the nth
block would fail, as the sync task may not have completed
So I think we need to do the following for the quota based tests
(expanding on the provided patch,
http://review.gluster.org/#/c/10811/ )
- First dd that exceeds quota (with either oflag=sync or
conv=fdatasync so that we do not see any flush behind or write behind
effects) to be done without checks
- Next check in an EXPECT_WITHIN that quota is exceeded (maybe add
checks on the just created/appended file w.r.t its minimum size that
would make it exceed the quota)
- Then do a further dd to a new file or append to an existing file to
get the EDQUOT error
- Proceed with whatever the test case needs to do next
Suggestions?
Here is my analysis on spurious failure with testcase:
tests/bugs/distribute/bug-1161156.t
In release-3.7, marker is re-factored to use synctask to do background
accounting.
I have done below tests with different combination and found that
parallel writes is causing the spurious failure.
I have filed a bug# 1223658 to track parallel write issue with quota.
Agreed with the observations, tallies with mine. Just one addition,
when we write 256k or less, the writes become serial as NFS writes in
256k chunks, and due to oflag=sync it follows up with a flush, correct?
Yes
Test (2) is interesting, even with marker foreground updates (which is
still in the UNWIND path), we observe failures. Do we know why? My
analysis/understanding of the same is that we have more in flight IOs
that passed quota enforcement (due to accounting on the UNWIND path),
does this bear any merit post your tests?
Yes, my understanding is same that it could be because of more in-flight
IOs and there is not much impact if the marker is doing background updates.
jFYI, I ran the patch posted for the above, 10 times in my machine, and
it passed all times. One caveat is that at times the file size 0.5 MB
more (see attached). Just stating this so that as we set the rules for
quota tests, we should also add that the attempt should be to write
*more* than the quota limit, and what this *more* is should be at least
a few extra 256k blocks (say 16 at least?). This is to prevent future
tests ending up in a similar regression state.
1) Parallel writes and Marker background update (Test always fails)
TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2
conv=fdatasync oflag=sync oflag=append
NFS client breaks 3M writes into multiple 256k chunks and does
parallel writes
2) Parallel writes and Marker foreground update (Test always fails)
TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2
conv=fdatasync oflag=sync oflag=append
Made a marker code change to account quota in foreground (without
synctask)
3) Serial writes and Marker background update (Test passed 100/100
times)
TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=256k count=24
conv=fdatasync oflag=sync oflag=append
Using smaller block size (256k), so that NFS client reduces
parallel writes
4) Serial writes and Marker foreground update (Test passed 100/100
times)
TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=256k count=24
conv=fdatasync oflag=sync oflag=append
Using smaller block size (256k), so that NFS client reduces
parallel writes
Made a marker code change to account quota in foreground (without
synctask)
5) Parallel writes on release-3.6 (Test always fails)
TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2
conv=fdatasync oflag=sync oflag=append
Moved marker xlator above IO-Threads in the graph.
Thanks,
Vijay
Test modified as follows to check the file size post dd,
# wait for write behind to complete.
EXPECT_WITHIN $MARKER_UPDATE_TIMEOUT "15.0MB" usage "/"
TEST ! quota_dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=256k count=40
---> stat $N0/$mydir/newfile_2 > /dev/tty <---
# Test rename within a directory. It should pass even when the
# corresponding directory quota is filled.
TEST mv $N0/dir/file $N0/dir/newfile_3
Results:
--------
host# for i in {1..10}; do ./tests/bugs/distribute/bug-1161156.t; done
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 9671870521241932594 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:25:37.511565334 -0400
Modify: 2015-05-21 09:25:37.857565398 -0400
Change: 2015-05-21 09:25:37.857565398 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 10709475173786443596 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:26:39.816576835 -0400
Modify: 2015-05-21 09:26:40.139576895 -0400
Change: 2015-05-21 09:26:40.139576895 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 10363430981752982285 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:27:36.347586026 -0400
Modify: 2015-05-21 09:27:36.663585983 -0400
Change: 2015-05-21 09:27:36.663585983 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 11172862175538664493 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:28:33.815578272 -0400
Modify: 2015-05-21 09:28:34.139578229 -0400
Change: 2015-05-21 09:28:34.147578227 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 12608171046258204792 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:29:32.903570300 -0400
Modify: 2015-05-21 09:29:33.230570256 -0400
Change: 2015-05-21 09:29:33.230570256 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5505024 Blocks: 10240 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 9920617755958839421 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:30:30.724562499 -0400
Modify: 2015-05-21 09:30:31.017562460 -0400
Change: 2015-05-21 09:30:31.017562460 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 12864360325492940828 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:31:39.560553212 -0400
Modify: 2015-05-21 09:31:39.872553170 -0400
Change: 2015-05-21 09:31:39.872553170 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5505024 Blocks: 10240 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 11174885303917170550 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:33:11.187542186 -0400
Modify: 2015-05-21 09:33:11.495542150 -0400
Change: 2015-05-21 09:33:11.496542150 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 9914333059591019731 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:34:14.258534735 -0400
Modify: 2015-05-21 09:34:14.590534695 -0400
Change: 2015-05-21 09:34:14.590534695 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5505024 Blocks: 10240 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 13416120642134532771 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:35:13.752527706 -0400
Modify: 2015-05-21 09:35:14.063527669 -0400
Change: 2015-05-21 09:35:14.064527669 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel
