On 10/15/18 12:41 PM, Dietmar Rieder wrote: > On 10/15/18 12:02 PM, jesper@xxxxxxxx wrote: >>>> On Sun, Oct 14, 2018 at 8:21 PM <jesper@xxxxxxxx> wrote: >>>> how many cephfs mounts that access the file? Is is possible that some >>>> program opens that file in RW mode (even they just read the file)? >>> >>> >>> The nature of the program is that it is "prepped" by one-set of commands >>> and queried by another, thus the RW case is extremely unlikely. >>> I can change permission bits to rewoke the w-bit for the user, they >>> dont need it anyway... it is just the same service-users that generates >>> the data and queries it today. >> >> Just to remove the suspicion of other clients fiddling with the files I did a >> more structured test. I have 4 x 10GB files from fio-benchmarking, total >> 40GB . Hosted on >> >> 1) CephFS /ceph/cluster/home/jk >> 2) NFS /z/home/jk >> >> First I read them .. then sleep 900 seconds .. then read again (just with dd) >> >> jk@sild12:/ceph/cluster/home/jk$ time for i in $(seq 0 3); do echo "dd >> if=test.$i.0 of=/dev/null bs=1M"; done | parallel -j 4 ; sleep 900; time >> for i in $(seq 0 3); do echo "dd if=test.$i.0 of=/dev/null bs=1M"; done | >> parallel -j 4 >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.56413 s, 4.2 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.82234 s, 3.8 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.9361 s, 3.7 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 3.10397 s, 3.5 GB/s >> >> real 0m3.449s >> user 0m0.217s >> sys 0m11.497s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 315.439 s, 34.0 MB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 338.661 s, 31.7 MB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 354.725 s, 30.3 MB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 356.126 s, 30.2 MB/s >> >> real 5m56.634s >> user 0m0.260s >> sys 0m16.515s >> jk@sild12:/ceph/cluster/home/jk$ >> >> >> Then NFS: >> >> jk@sild12:~$ time for i in $(seq 0 3); do echo "dd if=test.$i.0 >> of=/dev/null bs=1M"; done | parallel -j 4 ; sleep 900; time for i in >> $(seq 0 3); do echo "dd if=test.$i.0 of=/dev/null bs=1M"; done | parallel >> -j 4 >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 1.60267 s, 6.7 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.18602 s, 4.9 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.47564 s, 4.3 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.54674 s, 4.2 GB/s >> >> real 0m2.855s >> user 0m0.185s >> sys 0m8.888s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 1.68613 s, 6.4 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 1.6983 s, 6.3 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.20059 s, 4.9 GB/s >> 10240+0 records in >> 10240+0 records out >> 10737418240 bytes (11 GB, 10 GiB) copied, 2.58077 s, 4.2 GB/s >> >> real 0m2.980s >> user 0m0.173s >> sys 0m8.239s >> jk@sild12:~$ >> >> >> Can I ask one of you to run the same "test" (or similar) .. and report back >> i you can reproduce it? > > here my test on e EC (6+3) pool using cephfs kernel client: > > 7061+1 records in > 7061+1 records out > 7404496985 bytes (7.4 GB) copied, 3.62754 s, 2.0 GB/s > 7450+1 records in > 7450+1 records out > 7812246720 bytes (7.8 GB) copied, 4.11908 s, 1.9 GB/s > 7761+1 records in > 7761+1 records out > 8138636188 bytes (8.1 GB) copied, 4.34788 s, 1.9 GB/s > 8212+1 records in > 8212+1 records out > 8611295220 bytes (8.6 GB) copied, 4.53371 s, 1.9 GB/s > > real 0m4.936s > user 0m0.275s > sys 0m16.828s > > 7061+1 records in > 7061+1 records out > 7404496985 bytes (7.4 GB) copied, 3.19726 s, 2.3 GB/s > 7761+1 records in > 7761+1 records out > 8138636188 bytes (8.1 GB) copied, 3.31881 s, 2.5 GB/s > 7450+1 records in > 7450+1 records out > 7812246720 bytes (7.8 GB) copied, 3.36354 s, 2.3 GB/s > 8212+1 records in > 8212+1 records out > 8611295220 bytes (8.6 GB) copied, 3.74418 s, 2.3 GB/s > > > No big difference here. > all CentOS 7.5 official kernel 3.10.0-862.11.6.el7.x86_64 ...forgot to mention: all is luminous ceph-12.2.7 ~Dietmar
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
