On 03/27/2015 04:22 PM, Trond Myklebust wrote: > On Fri, Mar 27, 2015 at 3:04 PM, Anna Schumaker > <Anna.Schumaker@xxxxxxxxxx> wrote: >> I did two separate dd tests with the same 5G file from yesterday, and still using the same virtual machines. First, I ran dd using direct IO for reads: >> dd if=/nfs/file iflag=direct of=/dev/null bs=128K >> >> Mixed file performance was awful, so I reran without direct IO enabled for comparison: >> dd if=/nfs/file iflag=nocache of=/dev/null oflag=nocache bs=128K >> >> bs=128K sets the block size used by dd to the NFS rsize, without this dd will only read 512 bytes at a time and take forever to complete. >> >> >> ########################## >> # # >> # Without READ_PLUS # >> # # >> ########################## >> >> >> NFS v4.1, iflag=direct: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 11.704s | 11.055s | 11.329s | 11.453s | 10.741s | 11.256s | >> | Hole | 9.839s | 9.326s | 9.381s | 9.430s | 8.875s | 9.370s | >> | Mixed | 19.150s | 19.468s | 18.650s | 18.537s | 19.312s | 19.023s | >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> NFS v4.2, iflag=direct: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 10.927s | 10.885s | 11.114s | 11.283s | 10.371s | 10.916s | >> | Hole | 9.515s | 9.039s | 9.116s | 8.867s | 8.905s | 9.088s | >> | Mixed | 19.149s | 18.656s | 19.400s | 18.834s | 20.041s | 19.216s | >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> >> >> NFS v4.1, iflag=nocache oflag=nocache: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 6.808s | 6.698s | 7.482s | 6.761s | 7.235s | 6.995s | >> | Hole | 5.350s | 5.148s | 5.161s | 5.070s | 5.089s | 5.164s | >> | Mixed | 9.316s | 8.731s | 9.072s | 9.145s | 8.627s | 8.978s | >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> NFS v4.2, iflag=nocache oflag=nocache: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 6.686s | 6.848s | 6.876s | 6.799s | 7.815s | 7.004s | >> | Hole | 5.092s | 5.330s | 5.050s | 5.280s | 5.030s | 5.156s | >> | Mixed | 8.142s | 7.897s | 8.040s | 7.960s | 8.050s | 8.018s | >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> >> >> >> >> ####################### >> # # >> # With READ_PLUS # >> # # >> ####################### >> >> >> NFS v4.1, iflag=direct: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 9.464s | 10.181s | 10.048s | 9.452s | 10.795s | 9.988s | >> | Hole | 7.954s | 8.486s | 7.762s | 7.969s | 8.299s | 8.094s | >> | Mixed | 19.037s | 18.323s | 18.965s | 18.156s | 19.185s | 18.733s | >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> NFS v4.2, iflag=direct: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 11.923s | 10.026s | 10.222s | 12.387s | 11.431s | 11.198s | >> | Hole | 3.247s | 3.155s | 3.191s | 3.243s | 3.202s | 3.208s | >> | Mixed | 54.677s | 54.697s | 52.978s | 53.704s | 54.054s | 54.022s | > > That's a bit nasty. Any idea what is going on with the Mixed case here? Not offhand, but my first guess would be something to do with extra seeks to find how long each hole and data segment is. Anna > >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> >> >> NFS v4.1, iflag=nocache oflag=nocache: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 6.788s | 6.802s | 6.750s | 6.756s | 6.852s | 6.790s | >> | Hole | 5.143s | 5.165s | 5.104s | 5.154s | 5.116s | 5.136s | >> | Mixed | 7.902s | 7.693s | 9.169s | 8.186s | 9.157s | 8.421s | >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> NFS v4.2, iflag=nocache oflag=nocache: >> Trial >> |---------|---------|---------|---------|---------|---------|---------| >> | | 1 | 2 | 3 | 4 | 5 | Average | >> |---------|---------|---------|---------|---------|---------|---------| >> | Data | 6.897s | 6.862s | 7.054s | 6.961s | 7.081s | 6.971s | >> | Hole | 1.690s | 1.673s | 1.553s | 1.554s | 1.490s | 1.592s | >> | Mixed | 9.009s | 7.840s | 7.661s | 8.945s | 7.649s | 8.221s | >> |---------|---------|---------|---------|---------|---------|---------| >> >> >> On 03/26/2015 12:13 PM, Trond Myklebust wrote: >>> On Thu, Mar 26, 2015 at 12:11 PM, Anna Schumaker >>> <Anna.Schumaker@xxxxxxxxxx> wrote: >>>> On 03/26/2015 12:06 PM, Trond Myklebust wrote: >>>>> On Thu, Mar 26, 2015 at 11:47 AM, Anna Schumaker >>>>> <Anna.Schumaker@xxxxxxxxxx> wrote: >>>>>> On 03/26/2015 11:38 AM, J. Bruce Fields wrote: >>>>>>> On Thu, Mar 26, 2015 at 11:32:25AM -0400, Trond Myklebust wrote: >>>>>>>> On Thu, Mar 26, 2015 at 11:21 AM, Anna Schumaker >>>>>>>> <Anna.Schumaker@xxxxxxxxxx> wrote: >>>>>>>>> Here are my updated numbers! I tested with files 5G in size: one 100% data, one 100% hole, and one alternating between hole and data every 4K. I collected data for both v4.1 and v4.2 with and without the READ_PLUS patches: >>>>>>>>> >>>>>>>>> ########################## >>>>>>>>> # # >>>>>>>>> # Without READ_PLUS # >>>>>>>>> # # >>>>>>>>> ########################## >>>>>>>>> >>>>>>>>> >>>>>>>>> NFS v4.1: >>>>>>>>> Trial >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | | 1 | 2 | 3 | 4 | 5 | Average | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | Data | 8.723s | 7.243s | 8.252s | 6.997s | 6.980s | 7.639s | >>>>>>>>> | Hole | 5.271s | 5.224s | 5.060s | 4.897s | 5.321s | 5.155s | >>>>>>>>> | Mixed | 8.050s | 10.057s | 7.919s | 8.060s | 9.557s | 8.729s | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> NFS v4.2: >>>>>>>>> Trial >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | | 1 | 2 | 3 | 4 | 5 | Average | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | Data | 6.707s | 7.070s | 6.722s | 6.761s | 6.810s | 6.814s | >>>>>>>>> | Hole | 5.152s | 5.149s | 5.213s | 5.206s | 5.312s | 5.206s | >>>>>>>>> | Mixed | 7.979s | 7.985s | 8.177s | 7.772s | 8.280s | 8.039s | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ####################### >>>>>>>>> # # >>>>>>>>> # With READ_PLUS # >>>>>>>>> # # >>>>>>>>> ####################### >>>>>>>>> >>>>>>>>> >>>>>>>>> NFS v4.1: >>>>>>>>> Trial >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | | 1 | 2 | 3 | 4 | 5 | Average | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | Data | 9.082s | 7.008s | 7.116s | 6.771s | 7.902s | 7.576s | >>>>>>>>> | Hole | 5.333s | 5.358s | 5.380s | 5.161s | 5.282s | 5.303s | >>>>>>>>> | Mixed | 8.189s | 8.308s | 9.540s | 7.937s | 8.420s | 8.479s | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> NFS v4.2: >>>>>>>>> Trial >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | | 1 | 2 | 3 | 4 | 5 | Average | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> | Data | 7.033s | 6.829s | 7.025s | 6.873s | 7.134s | 6.979s | >>>>>>>>> | Hole | 1.794s | 1.800s | 1.905s | 1.811s | 1.725s | 1.807s | >>>>>>>>> | Mixed | 7.590s | 8.777s | 9.423s | 10.366s | 8.024s | 8.836s | >>>>>>>>> |---------|---------|---------|---------|---------|---------|---------| >>>>>>>>> >>>>>>>> >>>>>>>> So there is a clear win in the 100% hole case here, but otherwise the >>>>>>>> statistical fluctuations are dominating the numbers. Can you get us a >>>>>>>> little more stats and then perhaps run the results through nfsometer? >>>>>>> >>>>>>> Also, could you describe the setup (are these still kvm's), and how >>>>>>> you're clearing the cache between runs? >>>>>> >>>>>> These are still KVMs and my server is exporting an xfs filesystem. I clear caches by running "echo 3 > /proc/sys/vm/drop_caches" on the server before every read, and I remount my client after reading each set of three files once. >>>>> >>>>> I agree that you have to use the 'drop_caches' interface on the >>>>> server, but why not just use O_DIRECT on the clients? >>>> >>>> I've been reading by using cat from my test shell script: `time cat /nfs/file > /dev/null`. I can write something to read files with O_DIRECT if that would be more useful! >>>> >>> >>> 'dd' can do that for you if the appropriate incantations are performed. >>> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > _______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
