Re: bluefs _allocate unable to allocate

[Igor Fedotov <ifedotov@xxxxxxx>]

On 10/6/2021 2:16 PM, José H. Freidhof wrote:
> Hi Igor,
>
> yes i have some osd settings set :-) here are my ceph config dump. those
> settings are from a redhat document for bluestore devices
> maybe it is that setting causing this problem? "advanced
>   mon_compact_on_trim    false"???

OMG!!!

No - mon_compact_on_trim has nothing to deal with bluestore.

Highly likely it's bluestore_rocksdb_options which hurts... 
Documentations tend to fall behind the best practices.... I would 
strongly discourage you from using non-default settings unless it's 
absolutely clear why this is necessary.

Even at the first glance the following settings (just a few ones I'm 
completely aware) are suboptimal/non-recommended:

rocksdb_perf

bluefs_sync_write

bluefs_csum_type


Not to mention bluestore_rocksdb_options which hasn't got much adoption 
so far and apparently greatly alters rocksdb behavior...


So I would suggest to revert rocksdb options back to default, run the 
compaction and if it succeeds monitor the OSD for a while. Then if it 
works fine - apply the same for others


Hope this helps,

Igor



> i will test it this afternoon... at the moment are everything semi
> prodcuctive and i need to repair one osd node.. because i think of this
> reason the osds crashed on the node and the osd container crashes with a
> dump while coming up now.
> need first to replicate all between all three nodes and then i can take
> offline the osd.2.and test your command. i will inform you later...
>
> root@cd88-ceph-osdh-01:/# ceph config dump
> WHO                                               MASK
>   LEVEL     OPTION                                       VALUE
>
>
>
>
>
>                                     RO
> global
>   advanced  leveldb_max_open_files                       131072
>
>
>
>
> global
>   advanced  mon_compact_on_trim                          false
>
>
>
>
> global
>   dev       ms_crc_data                                  false
>
>
>
>
> global
>   advanced  osd_deep_scrub_interval                      1209600.000000
>
>
>
>
> global
>   advanced  osd_max_scrubs                               16
>
>
>
>
> global
>   advanced  osd_scrub_load_threshold                     0.010000
>
>
>
>
> global
>   advanced  osd_scrub_max_interval                       1209600.000000
>
>
>
>
> global
>   advanced  osd_scrub_min_interval                       86400.000000
>
>
>
>
> global
>   advanced  perf                                         true
>
>
>
>
> global
>   advanced  rbd_readahead_disable_after_bytes            0
>
>
>
>
> global
>   advanced  rbd_readahead_max_bytes                      4194304
>
>
>
>
> global
>   advanced  rocksdb_perf                                 true
>
>
>
>
> global
>   advanced  throttler_perf_counter                       false
>
>
>
>
>    mon
> advanced  auth_allow_insecure_global_id_reclaim        false
>
>
>
>
>    mon
> advanced  cluster_network                              10.50.50.0/24
>
>
>
>
>
>                                     *
>    mon
> advanced  mon_osd_down_out_interval                    300
>
>
>
>
>    mon
> advanced  public_network                               10.50.50.0/24
>
>
>
>
>
>                                     *
>    mgr
> advanced  mgr/cephadm/container_init                   True
>
>
>
>
>
>                                    *
>    mgr
> advanced  mgr/cephadm/device_enhanced_scan             true
>
>
>
>
>
>                                    *
>    mgr
> advanced  mgr/cephadm/migration_current                2
>
>
>
>
>
>                                     *
>    mgr
> advanced  mgr/cephadm/warn_on_stray_daemons            false
>
>
>
>
>
>                                     *
>    mgr
> advanced  mgr/cephadm/warn_on_stray_hosts              false
>
>
>
>
>
>                                     *
>    osd
> advanced  bluefs_sync_write                            true
>
>
>
>
>    osd
> dev       bluestore_cache_autotune                     true
>
>
>
>
>    osd
> dev       bluestore_cache_kv_ratio                     0.200000
>
>
>
>
>    osd
> dev       bluestore_cache_meta_ratio                   0.800000
>
>
>
>
>    osd
> dev       bluestore_cache_size                         2147483648
>
>
>
>
>    osd
> dev       bluestore_cache_size_hdd                     2147483648
>
>
>
>
>    osd
> advanced  bluestore_csum_type                          none
>
>
>
>
>    osd
> dev       bluestore_extent_map_shard_max_size          200
>
>
>
>
>    osd
> dev       bluestore_extent_map_shard_min_size          50
>
>
>
>
>    osd
> dev       bluestore_extent_map_shard_target_size       100
>
>
>
>
>    osd
> advanced  bluestore_rocksdb_options
>   compression=kNoCompression,max_write_buffer_number=64,min_write_buffer_number_to_merge=32,recycle_log_file_num=64,compaction_style=kCompactionStyleLevel,write_buffer_size=4MB,target_file_size_base=4MB,max_background_compactions=64,level0_file_num_compaction_trigger=64,level0_slowdown_writes_trigger=128,level0_stop_writes_trigger=256,max_bytes_for_level_base=6GB,compaction_threads=32,flusher_threads=8,compaction_readahead_size=2MB
>   *
>    osd
> advanced  mon_osd_cache_size                           1024
>
>
>
>
>    osd
> dev       ms_crc_data                                  false
>
>
>
>
>    osd
> advanced  osd_map_share_max_epochs                     5
>
>
>
>
>    osd
> advanced  osd_max_backfills                            1
>
>
>
>
>    osd
> dev       osd_max_pg_log_entries                       10
>
>
>
>
>    osd
> dev       osd_memory_cache_min                         3000000000
>
>
>
>
>    osd                                             host:cd133-ceph-osdh-01
> basic     osd_memory_target                            5797322383
>
>
>
>
>    osd                                             host:cd133k-ceph-osdh-01
>   basic     osd_memory_target                            9402402385
>
>
>
>
>    osd                                             host:cd88-ceph-osdh-01
>   basic     osd_memory_target                            5797322096
>
>
>
>
>    osd
> advanced  osd_memory_target_autotune                   true
>
>
>
>
>    osd
> dev       osd_min_pg_log_entries                       10
>
>
>
>
>    osd
> advanced  osd_op_num_shards                            8
>
>
>
>
>
>                                     *
>    osd
> advanced  osd_op_num_threads_per_shard                 2
>
>
>
>
>
>                                     *
>    osd
> dev       osd_pg_log_dups_tracked                      10
>
>
>
>
>    osd
> dev       osd_pg_log_trim_min                          10
>
>
>
>
>    osd
> advanced  osd_recovery_max_active                      3
>
>
>
>
>    osd
> advanced  osd_recovery_max_single_start                1
>
>
>
>
>    osd
> advanced  osd_recovery_sleep                           0.000000
>
>
> Am Mi., 6. Okt. 2021 um 12:55 Uhr schrieb Igor Fedotov <ifedotov@xxxxxxx>:
>
> > Jose,
> >
> > In fact 48GB is a way too much for WAL drive - usually the write ahead log
> > tend to be 2-4 GBs.
> >
> > But in your case it's ~150GB, while DB itself is very small (146MB!!!):
> >
> > WAL         45 GiB      111 GiB     0 B         0 B         0 B
> > 154 GiB     2400
> >
> > DB          0 B         164 MiB     0 B         0 B         0 B
> > 146 MiB     30
> >
> >
> > which means that there are some issues with RocksDB's WAL processing,
> > which needs some troubleshooting...
> >
> > Curious if other OSDs are suffering from the same and whether you have any
> > custom settings for your OSD(s)?
> >
> > Additionally you might want to try the following command to compact this
> > specific OSD manually and check if this would normalize the DB layout - the
> > majority of data has to be at DB level not WAL. Please share the resulting
> > layout (reported by "ceph daemon osd.2 bluefs stats" command) after the
> > compaction is fulfiled and OSD is restarted.
> >
> > The compaction command to be applied on an offline OSD: "ceph-kvstore-tool
> > bluestore-kv <path-to-osd> compact"
> >
> > Even if the above works great please refrain from applying that compaction
> > to every OSD - let's see how that "compacted" OSD evolves.Would WAL grow
> > again or not?
> >
> > Thanks,
> >
> > Igor
> >
> >
> >
> >
> >
> >
> > On 10/6/2021 1:35 PM, José H. Freidhof wrote:
> >
> > Hello Igor,
> >
> > yes the volume is  nvme wal partitions for the bluestore devicegroups are
> > only 48gb each
> >
> > on each osd node are 1 nvme with 1tb splitted in 20 lvs with 48gb (WAL)
> > on each osd node are 4 ssd with 1tb splitted in 5 lvs with 175gb  (rock.db)
> > on each osd node are 20 hdd with 5.5tb with 1 lvs (block.db)
> >
> > each blustore have 1 partition nvme,ssd and hdd like described in the
> > documentation
> > https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/
> >
> > is this to small or can i adjust the max allocation on the wal nvme device
> > in the ceph configuration?
> > i know that the ssd and nvme are to small for those 5.5tb disk... its 1%
> > only ot the rotation disk.
> > i am new in ceph and still or always learning, but we are in a little
> > hurry because our other datastores are old and full.
> >
> > root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluestore bluefs device info
> > {
> >      "dev": {
> >          "device": "BDEV_WAL",
> >          "total": 48318377984,
> >          "free": 1044480,
> >          "bluefs_used": 48317333504
> >      },
> >      "dev": {
> >          "device": "BDEV_DB",
> >          "total": 187904811008,
> >          "free": 68757217280,
> >          "bluefs_used": 119147593728
> >      },
> >      "dev": {
> >          "device": "BDEV_SLOW",
> >          "total": 6001172414464,
> >          "free": 5624912359424,
> >          "bluefs_used": 0,
> >          "bluefs max available": 5624401231872
> >      }
> > }
> > root@cd88-ceph-osdh-01:/# ceph daemon osd.2 bluefs stats
> > 0 : device size 0xb3ffff000 : using 0xb3ff00000(45 GiB)
> > 1 : device size 0x2bbfffe000 : using 0x1bbeb00000(111 GiB)
> > 2 : device size 0x57541c00000 : using 0x579b592000(350 GiB)
> > RocksDBBlueFSVolumeSelector: wal_total:45902462976, db_total:178509578240,
> > slow_total:5701113793740, db_avail:103884521472
> > Usage matrix:
> > DEV/LEV     WAL         DB          SLOW        *           *
> > REAL        FILES
> > LOG         124 MiB     2.3 GiB     0 B         0 B         0 B
> > 7.5 MiB     1
> > WAL         45 GiB      111 GiB     0 B         0 B         0 B
> > 154 GiB     2400
> > DB          0 B         164 MiB     0 B         0 B         0 B
> > 146 MiB     30
> > SLOW        0 B         0 B         0 B         0 B         0 B         0
> > B         0
> > TOTALS      45 GiB      113 GiB     0 B         0 B         0 B         0
> > B         2431
> > MAXIMUMS:
> > LOG         124 MiB     2.3 GiB     0 B         0 B         0 B         17
> > MiB
> > WAL         45 GiB      149 GiB     0 B         0 B         0 B
> > 192 GiB
> > DB          0 B         762 MiB     0 B         0 B         0 B
> > 741 MiB
> > SLOW        0 B         0 B         0 B         0 B         0 B         0 B
> > TOTALS      45 GiB      150 GiB     0 B         0 B         0 B         0 B
> >
> > Am Mi., 6. Okt. 2021 um 11:45 Uhr schrieb Igor Fedotov <ifedotov@xxxxxxx>:
> >
> > > Hey Jose,
> > >
> > > it looks like your WAL volume is out of space which looks weird given
> > > its capacity = 48Gb.
> > >
> > > Could you please share the output of the following commands:
> > >
> > > ceph daemon osd.N bluestore bluefs device info
> > >
> > > ceph daemon osd.N bluefs stats
> > >
> > >
> > > Thanks,
> > >
> > > Igor
> > >
> > >
> > > On 10/6/2021 12:24 PM, José H. Freidhof wrote:
> > > > Hello together
> > > >
> > > > we have a running ceph pacific 16.2.5 cluster and i found this messages
> > > in
> > > > the service logs of the osd daemons.
> > > >
> > > > we have three osd nodes .. each node has 20osds as bluestore with
> > > > nvme/ssd/hdd
> > > >
> > > > is this a bug or maybe i have some settings wrong?
> > > >
> > > >
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:25.821+0000
> > > > 7f38eebd4700  1 bluefs _allocate unable to allocate 0x100000 on bdev 0,
> > > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> > > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:29.857+0000
> > > > 7f38eebd4700  1 bluefs _allocate unable to allocate 0x100000 on bdev 0,
> > > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> > > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.073+0000
> > > > 7f38eebd4700  1 bluefs _allocate unable to allocate 0x400000 on bdev 0,
> > > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> > > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.405+0000
> > > > 7f38eebd4700  1 bluefs _allocate unable to allocate 0x100000 on bdev 0,
> > > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> > > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.465+0000
> > > > 7f38eebd4700  1 bluefs _allocate unable to allocate 0x100000 on bdev 0,
> > > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> > > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.529+0000
> > > > 7f38eebd4700  1 bluefs _allocate unable to allocate 0x100000 on bdev 0,
> > > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> > > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> > > > 7f38eebd4700  4 rocksdb: [db_impl/db_impl_write.cc:1668] [L] New
> > > memtable
> > > > created with log file: #9588. Immutable memtables: 1.
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> > > > 7f38eebd4700  1 bluefs _allocate unable to allocate 0x100000 on bdev 0,
> > > > allocator name bluefs-wal, allocator type hybrid, capacity 0xb40000000,
> > > > block size 0x100000, free 0xff000, fragmentation 0, allocated 0x0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> > > > 7f3905c02700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.547575)
> > > > [db_impl/db_impl_compaction_flush.cc:2198] Calling
> > > > FlushMemTableToOutputFile with column family [L], flush slots available
> > > 1,
> > > > compaction slots available 1, flush slots scheduled 1, compaction slots
> > > > scheduled 0
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> > > > 7f3905c02700  4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing
> > > > memtable with next log file: 9587
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> > > > 7f3905c02700  4 rocksdb: [flush_job.cc:321] [L] [JOB 5709] Flushing
> > > > memtable with next log file: 9588
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> > > > 7f3905c02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850547916,
> > > > "job": 5709, "event": "flush_started", "num_memtables": 2,
> > > "num_entries":
> > > > 4146, "num_deletes": 0, "total_data_size": 127203926, "memory_usage":
> > > > 130479920, "flush_reason": "Write Buffer Full"}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.545+0000
> > > > 7f3905c02700  4 rocksdb: [flush_job.cc:350] [L] [JOB 5709] Level-0 flush
> > > > table #9589: started
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f3905c02700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850559292,
> > > > "cf_name": "L", "job": 5709, "event": "table_file_creation",
> > > "file_number":
> > > > 9589, "file_size": 3249934, "table_properties": {"data_size": 3247855,
> > > > "index_size": 1031, "index_partitions": 0, "top_level_index_size": 0,
> > > > "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
> > > > "filter_size": 197, "raw_key_size": 1088, "raw_average_key_size": 16,
> > > > "raw_value_size": 3246252, "raw_average_value_size": 47739,
> > > > "num_data_blocks": 36, "num_entries": 68, "num_deletions": 32,
> > > > "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0,
> > > > "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
> > > > "column_family_name": "L", "column_family_id": 10, "comparator":
> > > > "leveldb.BytewiseComparator", "merge_operator": "nullptr",
> > > > "prefix_extractor_name": "nullptr", "property_collectors": "[]",
> > > > "compression": "NoCompression", "compression_options": "window_bits=-14;
> > > > level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
> > > > enabled=0; ", "creation_time": 1633511730, "oldest_key_time":
> > > 1633511730,
> > > > "file_creation_time": 1633511850}}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f3905c02700  4 rocksdb: [flush_job.cc:401] [L] [JOB 5709] Level-0 flush
> > > > table #9589: 3249934 bytes OK
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f3905c02700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559362)
> > > > [memtable_list.cc:447] [L] Level-0 commit table #9589 started
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f3905c02700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559583)
> > > > [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #1 done
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f3905c02700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559586)
> > > > [memtable_list.cc:503] [L] Level-0 commit table #9589: memtable #2 done
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f3905c02700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559601)
> > > > EVENT_LOG_v1 {"time_micros": 1633511850559593, "job": 5709, "event":
> > > > "flush_finished", "output_compression": "NoCompression", "lsm_state":
> > > [8,
> > > > 1, 0, 0, 0, 0, 0], "immutable_memtables": 0}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f3905c02700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.559638)
> > > > [db_impl/db_impl_compaction_flush.cc:205] [L] Level summary: files[8 1
> > > 0 0
> > > > 0 0 0] max score 1.00
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f38fb3ed700  4 rocksdb: [compaction/compaction_job.cc:1676] [L] [JOB
> > > 5710]
> > > > Compacting 8@0 + 1@1 files to L1, score 1.00
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f38fb3ed700  4 rocksdb: [compaction/compaction_job.cc:1680] [L]
> > > Compaction
> > > > start summary: Base version 3090 Base level 0, inputs: [9589(3173KB)
> > > > 9586(4793KB) 9583(1876KB) 9580(194KB) 9576(6417KB) 9573(1078KB)
> > > 9570(405KB)
> > > > 9567(29KB)], [9564(1115KB)]
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.557+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850559956,
> > > > "job": 5710, "event": "compaction_started", "compaction_reason":
> > > > "LevelL0FilesNum", "files_L0": [9589, 9586, 9583, 9580, 9576, 9573,
> > > 9570,
> > > > 9567], "files_L1": [9564], "score": 1, "input_data_size": 19542092}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: [compaction/compaction_job.cc:1349] [L] [JOB
> > > 5710]
> > > > Generated table #9590: 36 keys, 3249524 bytes
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850582987,
> > > > "cf_name": "L", "job": 5710, "event": "table_file_creation",
> > > "file_number":
> > > > 9590, "file_size": 3249524, "table_properties": {"data_size": 3247449,
> > > > "index_size": 1031, "index_partitions": 0, "top_level_index_size": 0,
> > > > "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0,
> > > > "filter_size": 197, "raw_key_size": 576, "raw_average_key_size": 16,
> > > > "raw_value_size": 3246252, "raw_average_value_size": 90173,
> > > > "num_data_blocks": 36, "num_entries": 36, "num_deletions": 0,
> > > > "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0,
> > > > "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter",
> > > > "column_family_name": "L", "column_family_id": 10, "comparator":
> > > > "leveldb.BytewiseComparator", "merge_operator": "nullptr",
> > > > "prefix_extractor_name": "nullptr", "property_collectors": "[]",
> > > > "compression": "NoCompression", "compression_options": "window_bits=-14;
> > > > level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0;
> > > > enabled=0; ", "creation_time": 1633471854, "oldest_key_time": 0,
> > > > "file_creation_time": 1633511850}}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: [compaction/compaction_job.cc:1415] [L] [JOB
> > > 5710]
> > > > Compacted 8@0 + 1@1 files to L1 => 3249524 bytes
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.583469)
> > > > [compaction/compaction_job.cc:760] [L] compacted to: files[0 1 0 0 0 0
> > > 0]
> > > > max score 0.01, MB/sec: 846.1 rd, 140.7 wr, level 1, files in(8, 1)
> > > out(1)
> > > > MB in(17.5, 1.1) out(3.1), read-write-amplify(1.2) write-amplify(0.2)
> > > OK,
> > > > records in: 376, records dropped: 340 output_compression: NoCompression
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: (Original Log Time 2021/10/06-09:17:30.583498)
> > > > EVENT_LOG_v1 {"time_micros": 1633511850583485, "job": 5710, "event":
> > > > "compaction_finished", "compaction_time_micros": 23098,
> > > > "compaction_time_cpu_micros": 20039, "output_level": 1,
> > > "num_output_files":
> > > > 1, "total_output_size": 3249524, "num_input_records": 376,
> > > > "num_output_records": 36, "num_subcompactions": 1, "output_compression":
> > > > "NoCompression", "num_single_delete_mismatches": 0,
> > > > "num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583615,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9589}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583648,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9586}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583675,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9583}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583709,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9580}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583739,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9576}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583769,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9573}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583804,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9570}
> > > > cd88-ceph-osdh-01 bash[6283]: debug 2021-10-06T09:17:30.581+0000
> > > > 7f38fb3ed700  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1633511850583835,
> > > > "job": 5710, "event": "table_file_deletion", "file_number": 9567}
> > > > _______________________________________________
> > > > ceph-users mailing list -- ceph-users@xxxxxxx
> > > > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >
> > --
> >
> > Mit freundlichen Grüßen,
> >
> >   -
> >
> > José H. Freidhof
> >
> > Reyerhütterstrasse 130b
> > 41065 Mönchengladbach
> > eMail: harald.freidhof@xxxxxxxxx
> > mobil: +49 (0) 1523 – 717 7801
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx