On Wed, Aug 13, 2014 at 9:22 PM, Theodore Ts'o <tytso@xxxxxxx> wrote:
> On Wed, Aug 13, 2014 at 10:32:31AM +0800, Li Xi wrote:
>> Yeah, we were using non-journaled quota. And we were doing this
>> benchmark to confirm that xattr based implementation has extra
>> overhead. We will run benchmarks on journaled-quota, and let's see
>> what is the performance difference between non-journaled and
>> journaled quotas.
>
> Can you give a lot of details about exactly how you ran the benchmark
> (and run future benchmarks)? Was this on a ramdisk? An SSD? A HDD?
> How many CPU's, how many threads were creating files, etc. And do you
> understand where the performance overhead was coming from? Was it CPU
> overhead? Locking overhead?
>
> It just doesn't make sense that storing the value in the xattr, when
> the xattr is stored in the on-disk inode, that it should make a huge
> difference; the cost of the I/O should completely dominate the cost of
> whether we format the bytes as an integer or storing it in the
> in-inode xattr. So either there is a bug in the benchmark, or a bug
> in our code somewhere. Either way, we should find and fix it.
All these tests were running on following environment:
Kernel: 3.16.0-rc5
Server: Dell R620 (2 x E5-2667@3.3GHz, 16 CPU cores, 256GB memory(1866MHz))
Storage: Dothill 3730(10 x 15K RPM SAS, RAID1+0, 2GB RAID Cache, 8Gbps FC)
Test tool: mdtest-1.9.3. Mdtest created 800K files in total. Each
thread created files in unique directory.
And we are using default settings of ext4, execept journal size is set
to 4GB. Following is the output of dumpe2fs:
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 98b77829-6154-4c5f-a0a5-da1306e82d8d
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index
filetype extent flex_bg sparse_super large_file huge_file uninit_bg
dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 6111232
Block count: 24414048
Reserved block count: 1220702
Free blocks: 22966646
Free inodes: 6111221
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1018
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
RAID stripe width: 256
Flex block group size: 16
Filesystem created: Wed Aug 13 23:52:54 2014
Last mount time: n/a
Last write time: Wed Aug 13 23:52:57 2014
Mount count: 0
Maximum mount count: -1
Last checked: Wed Aug 13 23:52:54 2014
Check interval: 0 (<none>)
Lifetime writes: 4104 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 96a5bb0c-3fcf-4114-bba3-049f15a2d8f4
Journal backup: inode blocks
Journal features: (none)
Journal size: 4096M
Journal length: 1048576
Journal sequence: 0x00000001
Journal start: 0
We did several rounds of file creation and removal:
1) vanilla kernel
1a) diabling quota
1b) enabling user/group quota
2) vanilla kernel with remove-dqptr_sem patches
2a) diabling quota
2b) enabling user/group quota
3) kernel with xattr based project quota
3a) diabling quota
3b) enabling user/group quota
3c) enabling user/group/project quota
4) kernel with remove-dqptr_sem patches as well as xattr based project quota
4a) diabling quota
4b) enabling user/group quota
4c) enabling user/group/project quota
5) kernel with remove-dqptr_sem patches as well as inode-internal project
quota
5a) diabling quota
5b) enabling user/group quota
5c) enabling user/group/project quota
6) kernel with remove-dqptr_sem patches as well as xattr based project quota,
but skipped writing the project ID xattr onto disk.
6a) diabling quota
6b) enabling user/group quota
6c) enabling user/group/project quota
7) kernel with remove-dqptr_sem patches, but test in a directory with
default ACL. That means file creation will have to inherit ACL xattr from
the directory too.
7a) diabling quota
7b) enabling user/group quota
We got following conclusion:
A) The results of 1a) and 1b) shows that quota enforcement brings
significant performance regression, especially to file removal, if
remove-dqptr_sem patches are not landed.
B) The results of 1b) and 2b) shows that remove-dqptr_sem patches indeed
improves performance when quota is enabled.
C) The results of 2a) and 2b) shows that remove-dqptr_sem patches elimits
the overhead of enforcing quota.
D) The results of 3a) and 1a) shows that xattr based quota brings
significant performance regression to file creation. Even the quota
enforcement is disabled, the iheriting of project ID harms the
performance.
E) The results of 3b) and 2b) confirms D).
F) The results of 3c) and 3b) shows that extra quota enforcement
brings performance regression both to file creation and removal, if
remove-dqptr_sem patches are not landed.
G) The results of 4b) and 3b) confirms B).
H) The results of 4a), 4b) and 4c) confirms C). Extra quota enforcement
does not necessarily harms performance if remove-dqptr_sem patches
are landed.
I) The results of 4c) and 3c) confirms C).
J) The results of 5a) and 4a) shows that inode-internal based project ID
improves performance, especially for file creation. Since the quota
enforcement is not disabled, the overhead of inheriting project ID harms
the performance of 4a).
K) The results of 5b) and 4b) confirms J).
L) The results of 5c) and 4c) confirms J).
M) The results of 6) and 4) confirms saving xattr of project ID is the main
cause of the performance regression.
N) The result of 7) and 2) shows that iheriting default ACL
causes performance regression for file creation. This result confirms M).
Following are the test results:
File Creation:
1thr 2thr 4thr 8thr 16thr
1) vanilla
1a) quota disabled 66094 105781 178968 186647 172536
1b) quotaon(ug) 60337 99582 157396 171463 162872
2) vanilla + remove-dqptr_sem patches
2a) quota disabled 65955 112082 185550 181511 171988
2b) quotaon(ug) 62391 101905 171013 190570 168914
3) prjquota(xattr)
3a) quota disabled 61396 97580 147852 146423 164895
3b) quotaon(ug) 57009 93435 140589 135748 153196
3c) quotaon(ugP) 57500 89419 133604 125291 105127
4) prjquota(xattr) + remove-dqptr_sem patches
4a) quota disabled 64053 100078 147608 139403 163960
4b) quotaon(ug) 60754 104726 149231 139053 165990
4c) quotaon(ugP) 59238 93606 148921 138434 163931
5) prjquota(internal) + remove-dqptr_sem patches
5a) quota disabled 65826 111828 181486 189227 171241
5b) quotaon(ug) 65418 107745 173584 180562 173752
5c) quotaon(ugP) 64669 103890 169176 186426 172192
6) prjquota(xattr) + remove-dqptr_sem patches + skip xattr saving
6a) quota disabled 68590 112022 181028 185626 174231
6b) quotaon(ug) 58189 99716 167318 179360 180188
6c) quotaon(ugP) 59049 99110 172885 181841 172034
7) vanilla + remove-dqptr_sem patches + in directory with default ACL
7a) quota disabled 63630 103930 134828 114534 137676
7b) quotaon(ug) 62274 93960 130317 111247 138620
File Removal:
1thr 2thr 4thr 8thr 16thr
1) vanilla
1a) quota disabled 118059 169825 234661 291812 345656
1b) quotaon(ug) 106675 135834 153532 100437 87489
2) vanilla + remove-dqptr_sem patches
2a) quota disabled 120374 168437 236818 291754 331141
2b) quotaon(ug) 110709 161954 238333 293700 329015
3) prjquota(xattr)
3a) quota disabled 116680 161662 229190 295642 332959
3b) quotaon(ug) 104783 134359 154950 100516 87923
3c) quotaon(ugP) 100240 125978 108653 68286 58991
4) prjquota(xattr) + remove-dqptr_sem patches
4a) quota disabled 116281 168938 233733 286663 344002
4b) quotaon(ug) 109775 164995 236001 299389 340683
4c) quotaon(ugP) 113935 162979 236112 300033 356117
5) prjquota(internal) + remove-dqptr_sem patches
5a) quota disabled 119537 171565 247418 291068 350138
5b) quotaon(ug) 121756 159580 240778 298012 342437
5c) quotaon(ugP) 118954 168022 241206 289055 334008
6) prjquota(xattr) + remove-dqptr_sem patches + skip xattr saving
6a) quota disabled 120573 170316 239318 300070 344562
6b) quotaon(ug) 102690 156876 233259 302307 330111
6c) quotaon(ugP) 104816 161109 239573 294188 332578
7) vanilla + remove-dqptr_sem patches + in directory with default ACL
7a) quota disabled 123580 173847 246271 302456 335549
7b) quotaon(ug) 115557 156577 236049 305340 339873
115557 156577 236049 305340 339873
Following is the script we used to get result 3), 4), 5) and 6). (Please
note that we wrote a xattr interface for inode-internal implementation too.
That is why we are able to use this script to get result of 5).)
--
#!/bin/sh
export PATH=/work/tools/openmpi-1.6.2/bin:/usr/local/projquota/bin:/usr/local/projquota/sbin:$PATH
export LD_LIBRARY_PATH=/work/tools/openmpi-1.6.2/lib/
NFILES=800000
DIR=/mnt/ext4/testdir/
DEV=/dev/mapper/LUN01
MNTPT=/mnt/ext4
MPIRUN="mpirun -mca btl ^openib"
USER_ID=1000
PROJ_ID=1000
do_mount()
{
local opts="$1"
sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
sudo umount $MNTPT
sudo sh -c "echo y | mkfs.ext4 -J size=4096 $DEV"
sudo mount -t ext4 $DEV $opts $MNTPT
sudo mkdir $DIR
#sudo setfacl -m d:user1:--- $DIR
sudo chmod 777 $DIR
}
quota_on()
{
local opts="$1"
sudo -i quotacheck $opts $MNTPT
sudo -i quotaon $opts $MNTPT
}
run_pretest()
{
local logfile=$1
local opts=$2
local nfiles=1000
$MPIRUN -np 1 ./mdtest -n $nfiles -u -i 1 -F -C -d $DIR
echo "#### quota $opts after file creation ####" > $logfile 2>&1
sudo -i quota $opts >> $logfile 2>&1
$MPIRUN -np 1 ./mdtest -n $nfiles -u -i 1 -F -r -d $DIR
echo "#### quota $opts after file removal ####" >> $logfile 2>&1
sudo -i quota $opts >> $logfile 2>&1
}
run_mdtest()
{
local logfile=$1
for n in 1 2 4 8 16; do
nfiles=$((NFILES/n))
$MPIRUN -np $n ./mdtest -n $nfiles -u -i 5 -F -d $DIR >> $logfile 2>&1
done
}
# test with disabled quota
test_disable_quota()
{
do_mount
run_mdtest mdtest-disable-noopt.log
}
# test with enabled quota, but only user/group quota enforced
test_enable_quota_ug()
{
do_mount "-o usrquota,grpquota"
quota_on "-ug"
sudo -i setquota -u $USER_ID 0 0 0 0 $MNTPT
run_pretest "mdtest-enable-ug.log" "-vu $USER_ID"
run_mdtest mdtest-enable-ug.log
}
# test with enabled quota and enforced project quota as well as user/group quota
test_enable_quota_ugP()
{
do_mount "-o usrquota,grpquota,prjquota"
quota_on "-ugP"
sudo -i setfattr -n "system.project" -v $PROJ_ID $DIR
sudo -i setquota -P $PROJ_ID 0 0 0 0 $MNTPT
run_pretest "mdtest-enable-ugP.log" "-vP $PROJ_ID"
run_mdtest mdtest-enable-ugP.log
}
sudo sh -c ./tune.sh
test_disable_quota
test_enable_quota_ug
test_enable_quota_ugP
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
