Re: [RESEND] [PATCH] readahead:add blk_run_backing_dev

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Ronald Moesbergen, on 07/13/2009 04:12 PM wrote:
2009/7/10 Vladislav Bolkhovitin <vst@xxxxxxxx>:
Vladislav Bolkhovitin, on 07/10/2009 12:43 PM wrote:
Ronald Moesbergen, on 07/10/2009 10:32 AM wrote:
I've also long ago noticed that reading data from block devices is
slower
than from files from mounted on those block devices file systems. Can
anybody explain it?

Looks like this is strangeness #2 which we uncovered in our tests (the
first
one was earlier in this thread why the context RA doesn't work with
cooperative I/O threads as good as it should).

Can you rerun the same 11 tests over a file on the file system, please?
I'll see what I can do. Just te be sure: you want me to run
blockdev-perftest on a file on the OCFS2 filesystem which is mounted
on the client over iScsi, right?
Yes, please.
Forgot to mention that you should also configure your backend storage as a
big file on a file system (preferably, XFS) too, not as direct device, like
/dev/vg/db-master.

Ok, here are the results:

client kernel: 2.6.26-15lenny3 (debian)
server kernel: 2.6.29.5 with readahead patch

Test done with XFS on both the target and the initiator. This confirms
your findings, using files instead of block devices is faster, but
only when using the io_context patch.

Seems, correct, except case (2), which is still 10% faster.

Without io_context patch:
1) client: default, server: default
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  18.327   18.327   17.740   56.491    0.872    0.883
 33554432  18.662   18.311   18.116   55.772    0.683    1.743
 16777216  18.900   18.421   18.312   55.229    0.754    3.452
  8388608  18.893   18.533   18.281   55.156    0.743    6.895
  4194304  18.512   18.097   18.400   55.850    0.536   13.963
  2097152  18.635   18.313   18.676   55.232    0.486   27.616
  1048576  18.441   18.264   18.245   55.907    0.267   55.907
   524288  17.773   18.669   18.459   55.980    1.184  111.960
   262144  18.580   18.758   17.483   56.091    1.767  224.365
   131072  17.224   18.333   18.765   56.626    2.067  453.006
    65536  18.082   19.223   18.238   55.348    1.483  885.567
    32768  17.719   18.293   18.198   56.680    0.795 1813.766
    16384  17.872   18.322   17.537   57.192    1.024 3660.273

2) client: default, server: 64 max_sectors_kb, RA default
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  18.738   18.435   18.400   55.283    0.451    0.864
 33554432  18.046   18.167   17.572   57.128    0.826    1.785
 16777216  18.504   18.203   18.377   55.771    0.376    3.486
  8388608  22.069   18.554   17.825   53.013    4.766    6.627
  4194304  19.211   18.136   18.083   55.465    1.529   13.866
  2097152  18.647   17.851   18.511   55.866    1.071   27.933
  1048576  19.084   18.177   18.194   55.425    1.249   55.425
   524288  18.999   18.553   18.380   54.934    0.763  109.868
   262144  18.867   18.273   18.063   55.668    1.020  222.673
   131072  17.846   18.966   18.193   55.885    1.412  447.081
    65536  18.195   18.616   18.482   55.564    0.530  889.023
    32768  17.882   18.841   17.707   56.481    1.525 1807.394
    16384  17.073   18.278   17.985   57.646    1.689 3689.369

3) client: default, server: default max_sectors_kb, RA 2MB
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  18.658   17.830   19.258   55.162    1.750    0.862
 33554432  17.193   18.265   18.517   56.974    1.854    1.780
 16777216  17.531   17.681   18.776   56.955    1.720    3.560
  8388608  18.234   17.547   18.201   56.926    1.014    7.116
  4194304  18.057   17.923   17.901   57.015    0.218   14.254
  2097152  18.565   17.739   17.658   56.958    1.277   28.479
  1048576  18.393   17.433   17.314   57.851    1.550   57.851
   524288  18.939   17.835   18.972   55.152    1.600  110.304
   262144  18.562   19.005   18.069   55.240    1.141  220.959
   131072  19.574   17.562   18.251   55.576    2.476  444.611
    65536  19.117   18.019   17.886   55.882    1.647  894.115
    32768  18.237   17.415   17.482   57.842    1.200 1850.933
    16384  17.760   18.444   18.055   56.631    0.876 3624.391

4) client: default, server: 64 max_sectors_kb, RA 2MB
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  18.368   17.495   18.524   56.520    1.434    0.883
 33554432  18.209   17.523   19.146   56.052    2.027    1.752
 16777216  18.765   18.053   18.550   55.497    0.903    3.469
  8388608  17.878   17.848   18.389   56.778    0.774    7.097
  4194304  18.058   17.683   18.567   56.589    1.129   14.147
  2097152  18.896   18.384   18.697   54.888    0.623   27.444
  1048576  18.505   17.769   17.804   56.826    1.055   56.826
   524288  18.319   17.689   17.941   56.955    0.816  113.910
   262144  19.227   17.770   18.212   55.704    1.821  222.815
   131072  18.738   18.227   17.869   56.044    1.090  448.354
    65536  19.319   18.525   18.084   54.969    1.494  879.504
    32768  18.321   17.672   17.870   57.047    0.856 1825.495
    16384  18.249   17.495   18.146   57.025    1.073 3649.582

With io_context patch:
5) client: default, server: default
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  12.393   11.925   12.627   83.196    1.989    1.300
 33554432  11.844   11.855   12.191   85.610    1.142    2.675
 16777216  12.729   12.602   12.068   82.187    1.913    5.137
  8388608  12.245   12.060   14.081   80.419    5.469   10.052
  4194304  13.224   11.866   12.110   82.763    3.833   20.691
  2097152  11.585   12.584   11.755   85.623    3.052   42.811
  1048576  12.166   12.144   12.321   83.867    0.539   83.867
   524288  12.019   12.148   12.160   84.568    0.448  169.137
   262144  12.014   12.378   12.074   84.259    1.095  337.036
   131072  11.840   12.068   11.849   85.921    0.756  687.369
    65536  12.098   11.803   12.312   84.857    1.470 1357.720
    32768  11.852   12.635   11.887   84.529    2.465 2704.931
    16384  12.443   13.110   11.881   82.197    3.299 5260.620

6) client: default, server: 64 max_sectors_kb, RA default
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  13.033   12.122   11.950   82.911    3.110    1.295
 33554432  12.386   13.357   12.082   81.364    3.429    2.543
 16777216  12.102   11.542   12.053   86.096    1.860    5.381
  8388608  12.240   11.740   11.789   85.917    1.601   10.740
  4194304  11.824   12.388   12.042   84.768    1.621   21.192
  2097152  11.962   12.283   11.973   84.832    1.036   42.416
  1048576  12.639   11.863   12.010   84.197    2.290   84.197
   524288  11.809   12.919   11.853   84.121    3.439  168.243
   262144  12.105   12.649   12.779   81.894    1.940  327.577
   131072  12.441   12.769   12.713   81.017    0.923  648.137
    65536  12.490   13.308   12.440   80.414    2.457 1286.630
    32768  13.235   11.917   12.300   82.184    3.576 2629.883
    16384  12.335   12.394   12.201   83.187    0.549 5323.990

7) client: default, server: default max_sectors_kb, RA 2MB
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  12.017   12.334   12.151   84.168    0.897    1.315
 33554432  12.265   12.200   11.976   84.310    0.864    2.635
 16777216  12.356   11.972   12.292   83.903    1.165    5.244
  8388608  12.247   12.368   11.769   84.472    1.825   10.559
  4194304  11.888   11.974   12.144   85.325    0.754   21.331
  2097152  12.433   10.938   11.669   87.911    4.595   43.956
  1048576  11.748   12.271   12.498   84.180    2.196   84.180
   524288  11.726   11.681   12.322   86.031    2.075  172.062
   262144  12.593   12.263   11.939   83.530    1.817  334.119
   131072  11.874   12.265   12.441   84.012    1.648  672.093
    65536  12.119   11.848   12.037   85.330    0.809 1365.277
    32768  12.549   12.080   12.008   83.882    1.625 2684.238
    16384  12.369   12.087   12.589   82.949    1.385 5308.766

8) client: default, server: 64 max_sectors_kb, RA 2MB
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  12.664   11.793   11.963   84.428    2.575    1.319
 33554432  11.825   12.074   12.442   84.571    1.761    2.643
 16777216  11.997   11.952   10.905   88.311    3.958    5.519
  8388608  11.866   12.270   11.796   85.519    1.476   10.690
  4194304  11.754   12.095   12.539   84.483    2.230   21.121
  2097152  11.948   11.633   11.886   86.628    1.007   43.314
  1048576  12.029   12.519   11.701   84.811    2.345   84.811
   524288  11.928   12.011   12.049   85.363    0.361  170.726
   262144  12.559   11.827   11.729   85.140    2.566  340.558
   131072  12.015   12.356   11.587   85.494    2.253  683.952
    65536  11.741   12.113   11.931   85.861    1.093 1373.770
    32768  12.655   11.738   12.237   83.945    2.589 2686.246
    16384  11.928   12.423   11.875   84.834    1.711 5429.381

9) client: 64 max_sectors_kb, default RA. server: 64 max_sectors_kb, RA 2MB
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  13.570   13.491   14.299   74.326    1.927    1.161
 33554432  13.238   13.198   13.255   77.398    0.142    2.419
 16777216  13.851   13.199   13.463   75.857    1.497    4.741
  8388608  13.339   16.695   13.551   71.223    7.010    8.903
  4194304  13.689   13.173   14.258   74.787    2.415   18.697
  2097152  13.518   13.543   13.894   75.021    0.934   37.510
  1048576  14.119   14.030   13.820   73.202    0.659   73.202
   524288  13.747   14.781   13.820   72.621    2.369  145.243
   262144  14.168   13.652   14.165   73.189    1.284  292.757
   131072  14.112   13.868   14.213   72.817    0.753  582.535
    65536  14.604   13.762   13.725   73.045    2.071 1168.728
    32768  14.796   15.356   14.486   68.861    1.653 2203.564
    16384  13.079   13.525   13.427   76.757    1.111 4912.426

10) client: default max_sectors_kb, 2MB RA. server: 64 max_sectors_kb, RA 2MB
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  20.372   18.077   17.262   55.411    3.800    0.866
 33554432  17.287   17.620   17.828   58.263    0.740    1.821
 16777216  16.802   18.154   17.315   58.831    1.865    3.677
  8388608  17.510   18.291   17.253   57.939    1.427    7.242
  4194304  17.059   17.706   17.352   58.958    0.897   14.740
  2097152  17.252   18.064   17.615   58.059    1.090   29.029
  1048576  17.082   17.373   17.688   58.927    0.838   58.927
   524288  17.129   17.271   17.583   59.103    0.644  118.206
   262144  17.411   17.695   18.048   57.808    0.848  231.231
   131072  17.937   17.704   18.681   56.581    1.285  452.649
    65536  17.927   17.465   17.907   57.646    0.698  922.338
    32768  18.494   17.820   17.719   56.875    1.073 1819.985
    16384  18.800   17.759   17.575   56.798    1.666 3635.058

11) client: 64 max_sectors_kb, 2MB. RA server: 64 max_sectors_kb, RA 2MB
blocksize       R        R        R   R(avg,    R(std        R
  (bytes)     (s)      (s)      (s)    MB/s)   ,MB/s)   (IOPS)
 67108864  20.045   21.881   20.018   49.680    2.037    0.776
 33554432  20.768   20.291   20.464   49.938    0.479    1.561
 16777216  21.563   20.714   20.429   49.017    1.116    3.064
  8388608  21.290   21.109   21.308   48.221    0.205    6.028
  4194304  22.240   20.662   21.088   48.054    1.479   12.013
  2097152  20.282   21.098   20.580   49.593    0.806   24.796
  1048576  20.367   19.929   20.252   50.741    0.469   50.741
   524288  20.885   21.203   20.684   48.945    0.498   97.890
   262144  19.982   21.375   20.798   49.463    1.373  197.853
   131072  20.744   21.590   19.698   49.593    1.866  396.740
    65536  21.586   20.953   21.055   48.314    0.627  773.024
    32768  21.228   20.307   21.049   49.104    0.950 1571.327
    16384  21.257   21.209   21.150   48.289    0.100 3090.498

The drop with 64 max_sectors_kb on the client is a consequence of how CFQ is working. I can't find the exact code responsible for this, but from all signs, CFQ stops delaying requests if amount of outstanding requests exceeds some threshold, which is 2 or 3. With 64 max_sectors_kb and 5 SCST I/O threads this threshold is exceeded, so CFQ doesn't recover order of requests, hence the performance drop. With default 512 max_sectors_kb and 128K RA the server sees at max 2 requests at time.

Ronald, can you perform the same tests with 1 and 2 SCST I/O threads, please?

You can limit amount of SCST I/O threads by num_threads parameter of scst_vdisk module.

Thanks,
Vlad
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux