On Mon, 2020-04-06 at 15:17 +0200, Jesper Krogh wrote: > Hi Jeff. > > No, because the client "bacula-fd" is reading from the local > filesystem - here CephFS and sending it over the network to the server > with the tape-libraries attached to it. Thus "ideal" receive == send > - which is also the pattern I see when using larger files (multiple > MB). > > Is the per-file overhead many KB? > Maybe not "many" but "several". CephFS is quite chatty. There can also be quite a bit of back and forth between the client and MDS. The protocol has a lot of extraneous fields for any given message. Writing also means cap flushes (particularly on size changes), and those can add up. Whether that accounts for what you're seeing though, I'm not sure. > > On Mon, Apr 6, 2020 at 1:45 PM Jeff Layton <jlayton@xxxxxxxxxx> wrote: > > On Mon, 2020-04-06 at 10:04 +0200, Jesper Krogh wrote: > > > This is a CephFS client - its only purpose is to run the "filedaemon" of bacula > > > and transport data to the tape-library from CephFS - below 2 threads is > > > essentially doing something equivalent to > > > > > > find /cephfs/ -type f | xargs cat | nc server > > > > > > 2 threads only, load exploding and the "net read vs net write" has > > > more than 100x difference. > > > > > > > Makes sense. You're basically just reading in all of the data on this > > cephfs, so the receive is going to be much larger than the send. > > > > > Can anyone explain this as "normal" behaviour? > > > Server is a VM with 16 "vCPU" and 16GB memory running libvirt/qemu > > > > > > jk@wombat:~$ w > > > 07:50:33 up 11:25, 1 user, load average: 206.43, 76.23, 50.58 > > > USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT > > > jk pts/0 10.194.133.42 06:54 0.00s 0.05s 0.00s w > > > jk@wombat:~$ dstat -ar > > > --total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system-- --io/total- > > > usr sys idl wai stl| read writ| recv send| in out | int csw | read writ > > > 0 0 98 1 0| 14k 34k| 0 0 | 3B 27B| 481 294 |0.55 0.73 > > > 1 1 0 98 0| 0 0 | 60M 220k| 0 0 |6402 6182 | 0 0 > > > 0 1 0 98 0| 0 0 | 69M 255k| 0 0 |7305 4339 | 0 0 > > > 1 2 0 98 0| 0 0 | 76M 282k| 0 0 |7914 4886 | 0 0 > > > 1 1 0 99 0| 0 0 | 70M 260k| 0 0 |7293 4444 | 0 0 > > > 1 1 0 98 0| 0 0 | 80M 278k| 0 0 |8018 4931 | 0 0 > > > 0 1 0 98 0| 0 0 | 60M 221k| 0 0 |6435 5951 | 0 0 > > > 0 1 0 99 0| 0 0 | 59M 211k| 0 0 |6163 3584 | 0 0 > > > 0 1 0 98 0| 0 0 | 64M 323k| 0 0 |6653 3881 | 0 0 > > > 1 0 0 99 0| 0 0 | 61M 243k| 0 0 |6822 4401 | 0 0 > > > 0 1 0 99 0| 0 0 | 55M 205k| 0 0 |5975 3518 | 0 0 > > > 1 1 0 98 0| 0 0 | 68M 242k| 0 0 |7094 6544 | 0 0 > > > 0 1 0 99 0| 0 0 | 58M 230k| 0 0 |6639 4178 | 0 0 > > > 1 2 0 98 0| 0 0 | 61M 243k| 0 0 |7117 4477 | 0 0 > > > 0 1 0 99 0| 0 0 | 61M 228k| 0 0 |6500 4078 | 0 0 > > > 0 1 0 99 0| 0 0 | 65M 234k| 0 0 |6595 3914 | 0 0 > > > 0 1 0 98 0| 0 0 | 64M 219k| 0 0 |6507 5755 | 0 0 > > > 1 1 0 99 0| 0 0 | 64M 233k| 0 0 |6869 4153 | 0 0 > > > 1 2 0 98 0| 0 0 | 63M 232k| 0 0 |6632 3907 | > > > 0 0 ^C > > > > Load average is high, but it looks like it's all just waiting on I/O. > > > > > jk@wombat:~$ w > > > 07:50:56 up 11:25, 1 user, load average: 221.35, 88.07, 55.02 > > > USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT > > > jk pts/0 10.194.133.42 06:54 0.00s 0.05s 0.00s w > > > jk@wombat:~$ > > > > > -- > > Jeff Layton <jlayton@xxxxxxxxxx> > > -- Jeff Layton <jlayton@xxxxxxxxxx>
