Hello Richard! (Thanks for all your NBD work over the years) On Thu, 18 Apr 2019 at 15:10, Richard W.M. Jones <rjones@xxxxxxxxxx> wrote: <snip> > NBD servers (generally, not always) expose a single block device per > server. You can usually make multiple connections to the server, and > all connections will see a single view of the device. There is of > course no concept of files. > > I'm having some problems understanding the execution model of fio: in > what order do engine methods get called? how do threads correspond to > files? are "jobs" the same as "files"? The order of method calls for a synchronous engine (IF they are defined) is roughly: ->load (engines are all loaded together) ->file_size ->init ->open (per I/O) ->prep ->queue ->close ->cleanup ->free <-close It's possible to define more functions than this though. A thread may have one or more files associated with it depending on what the engine can cope with and what the user requested (see https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-filename ). A "job" is the definition of an fio workload and via filename it may define multiple files to work with. A filename may be auto generated but this may not make sense with a diskless ioengine such as the one you're writing. > Nevertheless I think I have mapped a single job to the NBD block > device. You can test it using nbdkit plus a ramdisk like this: > > $ rm /tmp/socket > $ nbdkit -U /tmp/socket memory size=1G --run './fio examples/nbd.fio' > > If there are multiple jobs (files?) should these be striped over the > block device? It depends on how you define the job. If you were to just copy the job and give it a new name it would likely just overlap the first job (because it's just going to do the same thing at the same time). You would need to use something like numjobs (https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-numjobs ) offset_increment (https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-offset-increment ) and a workload that skipped (e.g. rw=write:12k - https://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-readwrite ) to achieve striping. > > For some reason fio doesn't recognize the new --sockname parameter: > > $ ./fio examples/nbd.fio --sockname=foo > ./fio: unrecognized option '--sockname=foo' > ./fio: unrecognized option '--sockname=foo' We'd have to debug this by looking at the code to see that the new option was registered correctly. > If this was fixed then you could use an implicit socket: > > $ nbdkit -U - memory size=1G \ > --run './fio examples/nbd.fio --sockname=$unixsocket' > > The .queue method in this engine is synchronous. It's possible for > multiple requests to be issued asynchronously on a single TCP socket, > but I don't know if it is worth doing this. > > Although I said above that all connections to an NBD server see the > "same" view of the device, the view is not guaranteed to be consistent > *unless* the server returns the NBD_FLAG_CAN_MULTI_CONN export flag. > We do not check this currently (mainly because qemu-nbd does not set it). > > Another problem is that we don't read the export size from the NBD > server until it's "too late" to use it to determine the file size. > Therefore users must take care to set a file size which is smaller > than the NBD export, although this isn't much of a problem in > practice. I notice that the RBD engine works around this by > connecting to their server early (during .setup). Yeah - I think I had to do something similar in my never submitted libiscsi engine. I think there I stash some information whenever the file is opened and then make use of that information in getsize (https://github.com/sitsofe/fio/blob/libiscsi/engines/libiscsi.c#L869 ). -- Sitsofe | http://sucs.org/~sits/
