> Op 2 februari 2017 om 15:35 schreef Ahmed Khuraidah <abushihab@xxxxxxxxx>: > > > Hi all, > > I am still confused about my CephFS sandbox. > > When I am performing simple FIO test into single file with size of 3G I > have too many IOps: > > cephnode:~ # fio payloadrandread64k3G > test: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, > iodepth=2 > fio-2.13 > Starting 1 process > test: Laying out IO file(s) (1 file(s) / 3072MB) > Jobs: 1 (f=1): [r(1)] [100.0% done] [277.8MB/0KB/0KB /s] [4444/0/0 iops] > [eta 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3714: Thu Feb 2 07:07:01 2017 > read : io=3072.0MB, bw=181101KB/s, iops=2829, runt= 17370msec > slat (usec): min=4, max=386, avg=12.49, stdev= 6.90 > clat (usec): min=202, max=5673.5K, avg=690.81, stdev=361 > > > But if I will change size to file to 320G, looks like I skip the cache: > > cephnode:~ # fio payloadrandread64k320G > test: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, > iodepth=2 > fio-2.13 > Starting 1 process > Jobs: 1 (f=1): [r(1)] [100.0% done] [4740KB/0KB/0KB /s] [74/0/0 iops] [eta > 00m:00s] > test: (groupid=0, jobs=1): err= 0: pid=3624: Thu Feb 2 06:51:09 2017 > read : io=3410.9MB, bw=11641KB/s, iops=181, runt=300033msec > slat (usec): min=4, max=442, avg=14.43, stdev=10.07 > clat (usec): min=98, max=286265, avg=10976.32, stdev=14904.82 > > > For random write test such behavior not exists, there are almost the same > results - around 100 IOps. > > So my question: could please somebody clarify where this caching likely > happens and how to manage it? > The page cache of your kernel. The kernel will cache the file in memory and perform read operations from there. Best way is to reboot your client between test runs. Although you can drop kernel caches I always reboot to make sure nothing is cached locally. Wido > P.S. > This is latest SLES/Jewel based onenode setup which has: > 1 MON, 1 MDS (both data and metadata pools on SATA drive) and 1 OSD (XFS on > SATA and journal on SSD). > My FIO config file: > direct=1 > buffered=0 > ioengine=libaio > iodepth=2 > runtime=300 > > Thanks > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |