Re: messenger: performance drop, v2 vs v1

Mark Nelson <mark.a.nelson@xxxxxxxxx> · Wed, 29 May 2019 16:20:37 -0500

On 5/29/19 3:41 PM, Gregory Farnum wrote:
On Wed, May 29, 2019 at 12:36 PM Roman Penyaev <rpenyaev@xxxxxxx> wrote:

Hi all,

I did a quick protocol performance comparison using fio_ceph_messenger
engine having `37c70bd1a75f ("Merge pull request #28099 from
tchaikov/wip-blobhash")`
as a master.  I use default fio job and config:

    src/test/fio/ceph-messenger.fio

       iodepth=128

    src/test/fio/ceph-messenger.conf

       [global]
       ms_type=async+posix
       ms_crc_data=false
       ms_crc_header=false
       ms_dispatch_throttle_bytes=0
       debug_ms=0/0

Results:

    protocol v1:

      4k  IOPS=116k, BW=454MiB/s, Lat=1100.75usec
      8k  IOPS=104k, BW=816MiB/s, Lat=1224.83usec
     16k  IOPS=93.7k, BW=1463MiB/s, Lat=1366.15usec
     32k  IOPS=81.5k, BW=2548MiB/s, Lat=1568.80usec
     64k  IOPS=69.8k, BW=4366MiB/s, Lat=1831.76usec
    128k  IOPS=47.8k, BW=5973MiB/s, Lat=2677.71usec
    256k  IOPS=23.7k, BW=5917MiB/s, Lat=5406.42usec
    512k  IOPS=11.8k, BW=5912MiB/s, Lat=10823.24usec
      1m  IOPS=5792, BW=5793MiB/s, Lat=22092.82usec

    protocol v2:

      4k  IOPS=95.5k, BW=373MiB/s, Lat=1340.09usec
      8k  IOPS=85.3k, BW=666MiB/s, Lat=1499.54usec
     16k  IOPS=75.8k, BW=1184MiB/s, Lat=1688.65usec
     32k  IOPS=61.6k, BW=1924MiB/s, Lat=2078.29usec
     64k  IOPS=53.6k, BW=3349MiB/s, Lat=2388.17usec
    128k  IOPS=32.5k, BW=4059MiB/s, Lat=3940.99usec
    256k  IOPS=17.5k, BW=4376MiB/s, Lat=7310.90usec
    512k  IOPS=8718, BW=4359MiB/s, Lat=14679.53usec
      1m  IOPS=3785, BW=3785MiB/s, Lat=33811.59usec

     IOPS percentage change:

            v1                v2            % change

      4k  IOPS=116k        IOPS=95.5k         -17%
      8k  IOPS=104k        IOPS=85.3k         -17%
     16k  IOPS=93.7k       IOPS=75.8k         -19%
     32k  IOPS=81.5k       IOPS=61.6k         -24%
     64k  IOPS=69.8k       IOPS=53.6k         -23%
    128k  IOPS=47.8k       IOPS=32.5k         -32%
    256k  IOPS=23.7k       IOPS=17.5k         -26%
    512k  IOPS=11.8k       IOPS=8718          -25%
      1m  IOPS=5792        IOPS=3785          -35%

Is that expected? Does anyone have similar numbers?

I don't think it's expected, and it looks like those config options
*should* have done as expected, but maybe run with some debug logs and
make sure the ProtocolV2 isn't doing CRCs or something after all.

Sage mentioned at Cephalocon he was a bit nervous that we might see some 
regression but I don't recall what the reasoning was.  A perf or 
wallclock profile would tell us pretty quick if it's CRC.  The fact that 
it's worse with large IOs vs small IOs might be a clue.

PS. Am I mistaken or 'ms_msgr2_encrypt_messages' and
      'ms_msgr2_sign_messages' options are not used at all?

Hmm yeah, those should get removed (...or maybe implemented for this
fio stuff). They're replaced by ms_cluster_mode et al when running in
a cluster. (The defaults in master are "crc secure" for most stuff
communications but "secure crc" for anything involving the monitors.)
-Greg