Re: Bluestore crashing constantly with load on newly created cluster/host.

Tyler Bishop <tyler.bishop@xxxxxxxxxxxxxxxxx> · Mon, 27 Aug 2018 22:48:05 -0400

My host has 256GB of ram.  62GB used under most heavy io workload.
_____________________________________________

Tyler Bishop
EST 2007

O: 513-299-7108 x1000
M: 513-646-5809
http://BeyondHosting.net

This email is intended only for the recipient(s) above and/or otherwise authorized personnel. The information contained herein and attached is confidential and the property of Beyond Hosting. Any unauthorized copying, forwarding, printing, and/or disclosing any information related to this email is prohibited. If you received this message in error, please contact the sender and destroy all copies of this email and any attachment(s).

On Mon, Aug 27, 2018 at 10:36 PM Alfredo Daniel Rezinovsky <alfredo.rezinovsky@xxxxxxxxxxxxxxxxxxxxxxxx> wrote:

    I had blockdb in ssd, with 3 OSDs per host (8G ram) and the
      default 3G bluestore_cache_size_ssd
    I stopped having inconsistencies dropping the cache to 1G.

    On 27/08/18 23:32, Tyler Bishop wrote:

      Having a constant segfault issue under io load with
        my newly created bluestore deployment.

        https://pastebin.com/82YjXRm7

        Setup is 28GB SSD LVM for block.db and 6T spinner for
          data.  

        Config:

          [global]
          fsid =  REDACTED
          mon_initial_members = cephmon-1001, cephmon-1002,
            cephmon-1003
          mon_host = 10.20.142.5,10.20.142.6,10.20.142.7
          auth_cluster_required = cephx
          auth_service_required = cephx
          auth_client_required = cephx
          filestore_xattr_use_omap = true

          # Fixes issue where image is created with newer than
            supported features enabled.
          rbd_default_features = 3

          # Debug Tuning
          debug_lockdep = 0/0
          debug_context = 0/0
          debug_crush = 0/0
          debug_buffer = 0/0
          debug_timer = 0/0
          debug_filer = 0/0
          debug_objecter = 0/0
          debug_rados = 0/0
          debug_rbd = 0/0
          debug_journaler = 0/0
          debug_objectcatcher = 0/0
          debug_client = 0/0
          debug_osd = 0/0
          debug_optracker = 0/0
          debug_objclass = 0/0
          debug_filestore = 0/0
          debug_journal = 0/0
          debug_ms = 0/0
          debug_monc = 0/0
          debug_tp = 0/0
          debug_auth = 0/0
          debug_finisher = 0/0
          debug_heartbeatmap = 0/0
          debug_perfcounter = 0/0
          debug_asok = 0/0
          debug_throttle = 0/0
          debug_mon = 0/0
          debug_paxos = 0/0
          debug_rgw = 0/0

          [osd]
          osd_mkfs_type = xfs
          osd_mount_options_xfs =
            rw,noatime,,nodiratime,inode64,logbsize=256k,delaylog
          osd_mkfs_options_xfs = -f -i size=2048
          osd_journal_size = 10240
          filestore_queue_max_ops=1000
          filestore_queue_max_bytes = 1048576000
          filestore_max_sync_interval = 10
          filestore_merge_threshold = 500
          filestore_split_multiple = 100
          osd_op_shard_threads = 6
          journal_max_write_entries = 5000
          journal_max_write_bytes = 1048576000
          journal_queueu_max_ops = 3000
          journal_queue_max_bytes = 1048576000
          ms_dispatch_throttle_bytes = 1048576000
          objecter_inflight_op_bytes = 1048576000
          public network = 10.20.142.0/24
          cluster_network = 10.20.136.0/24
          osd_disk_thread_ioprio_priority = 7
          osd_disk_thread_ioprio_class = idle
          osd_max_backfills = 2
          osd_recovery_sleep = 0.10

          [client]
          rbd_cache = False
          rbd cache size = 33554432
          rbd cache target dirty = 16777216
          rbd cache max dirty = 25165824
          rbd cache max dirty age = 2
          rbd cache writethrough until flush = false

          --------

            2018-08-28 02:31:30.961954 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:319]
              [default] [JOB 19] Level-0 flush table #688: 6121532 bytes
              OK
            2018-08-28 02:31:30.962476 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:242]
              adding log 681 to recycle list

            2018-08-28 02:31:30.962495 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:31:30.961973)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:360]
              [default] Level-0 commit table #688 started
            2018-08-28 02:31:30.962501 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:31:30.962413)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:383]
              [default] Level-0 commit table #688: memtable #1 done
            2018-08-28 02:31:30.962505 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:31:30.962432)
              EVENT_LOG_v1 {"time_micros": 1535423490962423, "job": 19,
              "event": "flush_finished", "lsm_state": [1, 4, 1, 0, 0, 0,
              0], "immutable_memtables": 0}
            2018-08-28 02:31:30.962509 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:31:30.962458)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:132]
              [default] Level summary: base level 1 max bytes base
              268435456 files[1 4 1 0 0 0 0] max score 0.84

            2018-08-28 02:31:30.962517 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:388]
              [JOB 19] Try to delete WAL files size 258068015, prev
              total WAL file size 260608480, number of live WAL files 2.

            2018-08-28 02:32:06.102335 7f64b917b700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_write.cc:684]
              reusing log 681 from recycle list

            2018-08-28 02:32:06.102473 7f64b917b700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_write.cc:725]
              [default] New memtable created with log file: #689.
              Immutable memtables: 0.

            2018-08-28 02:32:06.102542 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:49]
              [JOB 20] Syncing log #687
            2018-08-28 02:32:06.103394 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:32:06.102527)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:1158]
              Calling FlushMemTableToOutputFile with column family
              [default], flush slots available 1, compaction slots
              allowed 1, compaction slots scheduled 1
            2018-08-28 02:32:06.103407 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:264]
              [default] [JOB 20] Flushing memtable with next log file:
              689

            2018-08-28 02:32:06.103435 7f64a895a700  4 rocksdb:
              EVENT_LOG_v1 {"time_micros": 1535423526103422, "job": 20,
              "event": "flush_started", "num_memtables": 1,
              "num_entries": 97689, "num_deletes": 21335,
              "memory_usage": 260069984}
            2018-08-28 02:32:06.103446 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:293]
              [default] [JOB 20] Level-0 flush table #690: started
            2018-08-28 02:32:06.155755 7f64a895a700  4 rocksdb:
              EVENT_LOG_v1 {"time_micros": 1535423526155726, "cf_name":
              "default", "job": 20, "event": "table_file_creation",
              "file_number": 690, "file_size": 6343137,
              "table_properties": {"data_size": 6153638, "index_size":
              65232, "filter_size": 123278, "raw_key_size": 2289031,
              "raw_average_key_size": 52, "raw_value_size": 5374531,
              "raw_average_value_size": 122, "num_data_blocks": 1047,
              "num_entries": 43785, "filter_policy_name":
              "rocksdb.BuiltinBloomFilter", "kDeletedKeys": "21429",
              "kMergeOperands": "220"}}
            2018-08-28 02:32:06.155776 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/flush_job.cc:319]
              [default] [JOB 20] Level-0 flush table #690: 6343137 bytes
              OK
            2018-08-28 02:32:06.156214 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:242]
              adding log 687 to recycle list

            2018-08-28 02:32:06.156225 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:32:06.155790)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:360]
              [default] Level-0 commit table #690 started
            2018-08-28 02:32:06.156229 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:32:06.156164)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/memtable_list.cc:383]
              [default] Level-0 commit table #690: memtable #1 done
            2018-08-28 02:32:06.156239 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:32:06.156178)
              EVENT_LOG_v1 {"time_micros": 1535423526156172, "job": 20,
              "event": "flush_finished", "lsm_state": [2, 4, 1, 0, 0, 0,
              0], "immutable_memtables": 0}
            2018-08-28 02:32:06.156244 7f64a895a700  4 rocksdb:
              (Original Log Time 2018/08/28-02:32:06.156199)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_compaction_flush.cc:132]
              [default] Level summary: base level 1 max bytes base
              268435456 files[2 4 1 0 0 0 0] max score 0.84

            2018-08-28 02:32:06.156252 7f64a895a700  4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_files.cc:388]
              [JOB 20] Try to delete WAL files size 257866117, prev
              total WAL file size 259275521, number of live WAL files 2.

              _____________________________________________

                      Tyler Bishop
                      EST 2007

                      O: 513-299-7108
                            x1000

                          M: 513-646-5809
                          http://BeyondHosting.net

                                  This email is
                                      intended only for the recipient(s)
                                      above and/or otherwise authorized
                                      personnel. The
                                      information contained herein and
                                      attached is confidential and the
                                      property of Beyond Hosting. Any
                                      unauthorized copying, forwarding,
                                      printing, and/or disclosing
                                      any information related to this
                                      email is prohibited. If you
                                      received this message in error,
                                      please contact the sender and
                                      destroy all copies of this email
                                      and any attachment(s).

      _______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

    -- 
Alfredo Daniel Rezinovsky
Director de Tecnologías de Información y Comunicaciones
Facultad de Ingeniería - Universidad Nacional de Cuyo

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com