Re: cephfs filesystem offline, ceph-mds core dumped... How to recover?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Actually looking again, I don't see core dumps (though I could have sworn I did before). But looking at my MDS servers, they appear to be in a constant cycle of trying to restart, and not making it, then trying again, etc. The logs look like what I sent in my previous email, over and over again.
I tried restarting all three of my ceph-mds daemons at the same time, and it still doesn't seem to help. 

I hope I can recover...

-erich

On 5/7/24 5:18 PM, Erich Weiler wrote:

Hi All,

We just noticed that our cephfs filesystem was unreachable and we saw this:

# ceph -s
   cluster:
     id:     58bde08a-d7ed-11ee-9098-506b4b4da440
     health: HEALTH_WARN
             1 filesystem is degraded
             insufficient standby MDS daemons available

   services:
    mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) 
     mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf
     mds: 2/2 daemons up
     osd: 46 osds: 46 up (since 3d), 46 in (since 8w)

   data:
     volumes: 0/1 healthy, 1 recovering
     pools:   4 pools, 1313 pgs
     objects: 309.96M objects, 558 TiB
     usage:   842 TiB used, 287 TiB / 1.1 PiB avail
     pgs:     1293 active+clean
              13   active+clean+scrubbing
              7    active+clean+scrubbing+deep

   io:
     client:   1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr

[root@pr-md-02 ~]# ceph health detail
HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available 
[WRN] FS_DEGRADED: 1 filesystem is degraded
     fs slugfs is degraded
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
     have 0; want 1 more


And filesystem status is:

# ceph fs status
slugfs - 31 clients
======
RANK      STATE               MDS               ACTIVITY     DNS    INOS   DIRS   CAPS  0    replay(laggy)  slugfs.pr-md-02.sbblqq                    0      0      0      0  1        active     slugfs.pr-md-01.xdtppo  Reqs:    0 /s  7958   7959   13.1k   442 
        POOL           TYPE     USED  AVAIL
  cephfs_metadata    metadata   144G  3168G
cephfs_md_and_data    data       0   65.4T
    cephfs_data        data     838T   130T
MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
The ceph-mds container is not running on pr-md-02.  All OSDs are online.  I see this in the system logs on that machine (last 1000 lines), I see some core dumping happening in there: 

# tail -1000 /var/log/messages
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL files size 318677, prev total WAL file size 318677, number of live WAL files 2. May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485736) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337363339' seq:72057594037927935, type:22 .. '7061786F73003130313337383931' seq:0, type:0; will stop at (end) May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12099] Compacting 1@0 + 1@6 files to L6, score -1.00 May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12098 Base level 0, inputs: [228436(200KB)], [228434(37MB)] May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677485763, "job": 12099, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228436], "files_L6": [228434], "score": -1, "input_data_size": 39197377, "oldest_snapshot_seqno": -1} May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12099] Generated table #228437: 25153 keys, 38148344 bytes, temperature: kUnknown May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677626467, "cf_name": "default", "job": 12099, "event": "table_file_creation", "file_number": 228437, "file_size": 38148344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37996789, "index_size": 87588, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 62917, "raw_key_size": 697744, "raw_average_key_size": 27, "raw_value_size": 37553381, "raw_average_value_size": 1492, "num_data_blocks": 3073, "num_entries": 25153, "num_filter_entries": 25153, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437, "seqno_to_time_mapping": "N/A"}} May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.626709) [db/compaction/compaction_job.cc:1663] [default] [JOB 12099] Compacted 1@0 + 1@6 files to L6 => 38148344 bytes May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629741) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4 +0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK, records in: 25817, records dropped: 664 output_compression: NoCompression May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros": 1715126677629753, "job": 12099, "event": "compaction_finished", "compaction_time_micros": 140760, "compaction_time_cpu_micros": 62440, "output_level": 6, "num_output_files": 1, "total_output_size": 38148344, "num_input_records": 25817, "num_output_records": 25153, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677629887, "job": 12099, "event": "table_file_deletion", "file_number": 228436} May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126677632998, "job": 12099, "event": "table_file_deletion", "file_number": 228434} May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.485696) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633030) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633038) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633042) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633045) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:04:37.633049) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:04:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s May  7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:04:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May  7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May  7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:51 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May  7 17:04:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May  7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:01 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.630854043 -0700 PDT m=+0.047521077 container create 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_CLEAN=True) May  7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope. 
May  7 17:05:03 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.671554205 -0700 PDT m=+0.088221217 container init 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 
May  7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167
May  7 17:05:03 pr-md-02 systemd[1]: libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679778085 -0700 PDT m=+0.096445105 container start 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>) May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.679881368 -0700 PDT m=+0.096548394 container attach 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.build-date=20240222) May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.680110752 -0700 PDT m=+0.096777772 container died 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07 17:05:03.605825339 -0700 PDT m=+0.022492359 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07 17:05:03.7398573 -0700 PDT m=+0.053387996 container remove 268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD) May  7 17:05:03 pr-md-02 systemd[1]: libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated successfully. May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.869704758 -0700 PDT m=+0.050589984 container create ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, GIT_BRANCH=HEAD) May  7 17:05:03 pr-md-02 systemd[1]: Started libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope. 
May  7 17:05:03 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.901342134 -0700 PDT m=+0.082227374 container init ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True) May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909537569 -0700 PDT m=+0.090422794 container start ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.909642002 -0700 PDT m=+0.090527227 container attach ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>) May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07 17:05:03.845778509 -0700 PDT m=+0.026663744 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:05:04 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated successfully. May  7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed data devices: 2 physical, 0 LVM May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> relative data size: 1.0 May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All data devices are unavailable May  7 17:05:05 pr-md-02 systemd[1]: libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May  7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07 17:05:05.365433795 -0700 PDT m=+1.546319027 container died ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD) May  7 17:05:05 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated successfully. May  7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07 17:05:05.428169761 -0700 PDT m=+0.055435773 container remove ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) May  7 17:05:05 pr-md-02 systemd[1]: libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated successfully. May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.002418626 -0700 PDT m=+0.040341047 container create d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope. 
May  7 17:05:06 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.032819591 -0700 PDT m=+0.070742046 container init d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) 
May  7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167
May  7 17:05:06 pr-md-02 systemd[1]: libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040138184 -0700 PDT m=+0.078060613 container start d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0) May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040286008 -0700 PDT m=+0.078208464 container attach d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, ceph=True) May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:06.040587551 -0700 PDT m=+0.078509982 container died d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07 17:05:05.987393175 -0700 PDT m=+0.025315605 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07 17:05:06.097478996 -0700 PDT m=+0.050797295 container remove d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True) May  7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated successfully. May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.228492423 -0700 PDT m=+0.042467581 container create 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May  7 17:05:06 pr-md-02 systemd[1]: Started libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope. 
May  7 17:05:06 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.263672711 -0700 PDT m=+0.077647878 container init 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.271891014 -0700 PDT m=+0.085866179 container start 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.27203062 -0700 PDT m=+0.086005784 container attach 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>) May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.213038456 -0700 PDT m=+0.027013622 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    "41": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "devices": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1" 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_size": "1920378863616", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "tags": {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            },
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        }
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    "43": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "devices": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1" 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_size": "1920378863616", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "tags": {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.crush_device_class": "", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            },
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" 
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        }
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    ]
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: }
May  7 17:05:06 pr-md-02 systemd[1]: libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07 17:05:06.67755909 -0700 PDT m=+0.491534254 container died 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May  7 17:05:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May  7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07 17:05:06.745980606 -0700 PDT m=+0.056378222 container remove 39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May  7 17:05:06 pr-md-02 systemd[1]: libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated successfully. May  7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:07 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated successfully. May  7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.344365868 -0700 PDT m=+0.042884110 container create 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope. 
May  7 17:05:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.374008883 -0700 PDT m=+0.072527132 container init 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 
May  7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167
May  7 17:05:07 pr-md-02 systemd[1]: libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.381560483 -0700 PDT m=+0.080078731 container start 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_BRANCH=HEAD) May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38167372 -0700 PDT m=+0.080191969 container attach 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.38196021 -0700 PDT m=+0.080478458 container died 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07 17:05:07.330060004 -0700 PDT m=+0.028578252 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07 17:05:07.443179465 -0700 PDT m=+0.051396095 container remove 748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May  7 17:05:07 pr-md-02 systemd[1]: libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated successfully. May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.581081662 -0700 PDT m=+0.046354470 container create bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May  7 17:05:07 pr-md-02 systemd[1]: Started libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope. 
May  7 17:05:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.613009264 -0700 PDT m=+0.078282065 container init bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620807072 -0700 PDT m=+0.086079873 container start bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD) May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.620921904 -0700 PDT m=+0.086194728 container attach bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07 17:05:07.560418202 -0700 PDT m=+0.025691007 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:05:07 May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data', 'cephfs_metadata'] May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated successfully. May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May  7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' 
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: {
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", 
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "osd_id": 43,
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "type": "bluestore" 
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:    },
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", 
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "osd_id": 41,
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "type": "bluestore" 
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:    }
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }
May  7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May  7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07 17:05:08.623675315 -0700 PDT m=+1.088948125 container died bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May  7 17:05:08 pr-md-02 systemd[1]: libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU time. May  7 17:05:08 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated successfully. May  7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07 17:05:08.684978208 -0700 PDT m=+0.052557214 container remove bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1) May  7 17:05:08 pr-md-02 systemd[1]: libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated successfully. May  7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist May  7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:12 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling 
May  7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root.
May  7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope: Deactivated successfully. May  7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May  7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:17 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s May  7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s May  7 17:05:22 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May  7 17:05:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May  7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:27 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May  7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May  7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s May  7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log [INF] : from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May  7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished May  7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May  7 17:05:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s May  7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May  7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May  7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s May  7 17:05:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May  7 17:05:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May  7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May  7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s May  7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:05:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:05:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:05:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:06:07 May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', '.mgr', 'cephfs_data', 'cephfs_metadata'] May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May  7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May  7 17:06:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.181584373 -0700 PDT m=+0.039979954 container create 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May  7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope. 
May  7 17:06:11 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.211049202 -0700 PDT m=+0.069444730 container init 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 
May  7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167
May  7 17:06:11 pr-md-02 systemd[1]: libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218033241 -0700 PDT m=+0.076428761 container start 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218168917 -0700 PDT m=+0.076564434 container attach 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.218487253 -0700 PDT m=+0.076882764 container died 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True) May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07 17:06:11.165130923 -0700 PDT m=+0.023526448 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07 17:06:11.276198352 -0700 PDT m=+0.051426463 container remove 64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True) May  7 17:06:11 pr-md-02 systemd[1]: libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated successfully. May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.398685928 -0700 PDT m=+0.038305972 container create 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:06:11 pr-md-02 systemd[1]: Started libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope. 
May  7 17:06:11 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.4284462 -0700 PDT m=+0.068066246 container init 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436328165 -0700 PDT m=+0.075948196 container start 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.436459593 -0700 PDT m=+0.076079634 container attach 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07 17:06:11.383297135 -0700 PDT m=+0.022917157 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May  7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296615 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data devices: 2 physical, 0 LVM May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative data size: 1.0 May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data devices are unavailable May  7 17:06:12 pr-md-02 systemd[1]: libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May  7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07 17:06:12.059347046 -0700 PDT m=+0.698967087 container died 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07 17:06:12.115085789 -0700 PDT m=+0.048338618 container remove 4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) May  7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated successfully. May  7 17:06:12 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated successfully. May  7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.66977777 -0700 PDT m=+0.041271914 container create 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>) May  7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope. 
May  7 17:06:12 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.698793806 -0700 PDT m=+0.070287951 container init 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) 
May  7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167
May  7 17:06:12 pr-md-02 systemd[1]: libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.705992776 -0700 PDT m=+0.077486921 container start 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>) May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706102938 -0700 PDT m=+0.077597093 container attach 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD) May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.706421977 -0700 PDT m=+0.077916136 container died 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS) May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07 17:06:12.65486152 -0700 PDT m=+0.026355674 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07 17:06:12.760218425 -0700 PDT m=+0.047515050 container remove 24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May  7 17:06:12 pr-md-02 systemd[1]: libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated successfully. May  7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.883756999 -0700 PDT m=+0.038722013 container create ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1) May  7 17:06:12 pr-md-02 systemd[1]: Started libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope. 
May  7 17:06:12 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.915479032 -0700 PDT m=+0.070443976 container init ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image) May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922714281 -0700 PDT m=+0.077679227 container start ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.922813112 -0700 PDT m=+0.077778070 container attach ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2) May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07 17:06:12.867928578 -0700 PDT m=+0.022893523 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated successfully. 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    "41": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "tags": {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            },
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "type": "block", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        }
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    "43": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [ 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size": "1920378863616", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "tags": {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.cluster_name": "ceph", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.crush_device_class": "", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            },
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "type": "block", May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" 
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        }
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    ]
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: }
May  7 17:06:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:13 pr-md-02 systemd[1]: libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May  7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07 17:06:13.298397993 -0700 PDT m=+0.453362947 container died ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>) May  7 17:06:13 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated successfully. May  7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07 17:06:13.360359344 -0700 PDT m=+0.052612579 container remove ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD, ceph=True) May  7 17:06:13 pr-md-02 systemd[1]: libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated successfully. May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.944018291 -0700 PDT m=+0.043243095 container create b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) May  7 17:06:13 pr-md-02 systemd[1]: Started libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope. 
May  7 17:06:13 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.972707239 -0700 PDT m=+0.071931967 container init b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.vendor=CentOS) 
May  7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167
May  7 17:06:13 pr-md-02 systemd[1]: libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979610545 -0700 PDT m=+0.078835287 container start b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979741103 -0700 PDT m=+0.078965840 container attach b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.979972502 -0700 PDT m=+0.079197239 container died b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) May  7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07 17:06:13.927504129 -0700 PDT m=+0.026728865 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07 17:06:14.033457546 -0700 PDT m=+0.047525301 container remove b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.build-date=20240222) May  7 17:06:14 pr-md-02 systemd[1]: libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated successfully. May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.146017051 -0700 PDT m=+0.040230170 container create 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True) May  7 17:06:14 pr-md-02 systemd[1]: Started libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope. 
May  7 17:06:14 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.175282354 -0700 PDT m=+0.069495491 container init 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182850934 -0700 PDT m=+0.077064054 container start 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.182936358 -0700 PDT m=+0.077149475 container attach 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) May  7 17:06:14 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated successfully. May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07 17:06:14.129225093 -0700 PDT m=+0.023438214 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail 
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: {
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", 
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "osd_id": 43,
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "type": "bluestore" 
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:    },
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", 
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "osd_id": 41,
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "type": "bluestore" 
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:    }
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }
May  7 17:06:15 pr-md-02 systemd[1]: libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May  7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07 17:06:15.160506671 -0700 PDT m=+1.054719792 container died 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May  7 17:06:15 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated successfully. May  7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07 17:06:15.220598278 -0700 PDT m=+0.052865840 container remove 0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) May  7 17:06:15 pr-md-02 systemd[1]: libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated successfully. May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist May  7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log [DBG] : from='client.296625 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch May  7 17:06:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225254: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225255: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:06:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:06:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225256: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:26] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May  7 17:06:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225257: 1313 pgs: 7 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225258: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225259: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225260: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225261: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:36] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May  7 17:06:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225262: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225263: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225264: 1313 pgs: 6 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1294 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225265: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225266: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:46] "GET /metrics HTTP/1.1" 200 73531 "" "Prometheus/2.43.0" May  7 17:06:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225267: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225268: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225269: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225270: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:06:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:06:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225271: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:06:56] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May  7 17:06:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225272: 1313 pgs: 5 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:06:58 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:06:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225273: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228438. Immutable memtables: 0. May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.708981) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12100] Flushing memtable with next log file: 228438 May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819709068, "job": 12100, "event": "flush_started", "num_memtables": 1, "num_entries": 3587, "num_deletes": 921, "total_data_size": 2910092, "memory_usage": 2992096, "flush_reason": "Manual Compaction"} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12100] Level-0 flush table #228439: started May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819721834, "cf_name": "default", "job": 12100, "event": "table_file_creation", "file_number": 228439, "file_size": 1847607, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141701983, "largest_seqno": 141705565, "table_properties": {"data_size": 1834448, "index_size": 6860, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 5253, "raw_key_size": 55412, "raw_average_key_size": 26, "raw_value_size": 1798552, "raw_average_value_size": 857, "num_data_blocks": 265, "num_entries": 2097, "num_filter_entries": 2097, "num_deletions": 921, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126677, "oldest_key_time": 1715126677, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228439, "seqno_to_time_mapping": "N/A"}} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12100] Flush lasted 12882 microseconds, and 5541 cpu microseconds. May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721870) [db/flush_job.cc:967] [default] [JOB 12100] Level-0 flush table #228439: 1847607 bytes OK May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.721889) [db/memtable_list.cc:519] [default] Level-0 commit table #228439 started May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724483) [db/memtable_list.cc:722] [default] Level-0 commit table #228439: memtable #1 done May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724523) EVENT_LOG_v1 {"time_micros": 1715126819724510, "job": 12100, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.724561) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12100] Try to delete WAL files size 2890782, prev total WAL file size 2890782, number of live WAL files 2. May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228435.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725271) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '7061786F73003130313337383930' seq:72057594037927935, type:22 .. '7061786F73003130313338313432' seq:0, type:0; will stop at (end) May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12101] Compacting 1@0 + 1@6 files to L6, score -1.00 May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12100 Base level 0, inputs: [228439(1804KB)], [228437(36MB)] May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819725328, "job": 12101, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228439], "files_L6": [228437], "score": -1, "input_data_size": 39995951, "oldest_snapshot_seqno": -1} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12101] Generated table #228440: 25379 keys, 38251292 bytes, temperature: kUnknown May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819875843, "cf_name": "default", "job": 12101, "event": "table_file_creation", "file_number": 228440, "file_size": 38251292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 38096937, "index_size": 89812, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702456, "raw_average_key_size": 27, "raw_value_size": 37648320, "raw_average_value_size": 1483, "num_data_blocks": 3163, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228440, "seqno_to_time_mapping": "N/A"}} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.876095) [db/compaction/compaction_job.cc:1663] [default] [JOB 12101] Compacted 1@0 + 1@6 files to L6 => 38251292 bytes May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878629) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 265.6 rd, 254.0 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(1.8, 36.4 +0.0 blob) out(36.5 +0.0 blob), read-write-amplify(42.4) write-amplify(20.7) OK, records in: 27250, records dropped: 1871 output_compression: NoCompression May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.878669) EVENT_LOG_v1 {"time_micros": 1715126819878658, "job": 12101, "event": "compaction_finished", "compaction_time_micros": 150577, "compaction_time_cpu_micros": 64114, "output_level": 6, "num_output_files": 1, "total_output_size": 38251292, "num_input_records": 27250, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228439.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819878937, "job": 12101, "event": "table_file_deletion", "file_number": 228439} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228437.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126819882652, "job": 12101, "event": "table_file_deletion", "file_number": 228437} May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.725225) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882737) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882741) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882743) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882744) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:06:59 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:06:59.882746) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:07:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225274: 1313 pgs: 7 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225275: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:03 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225276: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:06] "GET /metrics HTTP/1.1" 200 73533 "" "Prometheus/2.43.0" May  7 17:07:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225277: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May  7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:07:07 May  7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 
May  7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_data', 'cephfs_md_and_data', '.mgr', 'cephfs_metadata'] May  7 17:07:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May  7 17:07:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May  7 17:07:08 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225278: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225279: 1313 pgs: 5 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225280: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:13 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May  7 17:07:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May  7 17:07:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225281: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 e5614: 46 total, 46 up, 46 in May  7 17:07:15 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:16] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:07:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225283: 1313 pgs: 6 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev d807ba79-465d-4d89-ab03-9bff6275dd10 does not exist May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9e26e8f6-fba9-425c-923b-ba9da32693a7 does not exist May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 1f435daa-e8f0-4964-b7fb-07ea14b95e9f does not exist May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:17 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.694558647 -0700 PDT m=+0.039788036 container create 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1) May  7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope. 
May  7 17:07:17 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.722272344 -0700 PDT m=+0.067501679 container init 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, ceph=True) 
May  7 17:07:17 pr-md-02 stoic_mayer[2565752]: 167 167
May  7 17:07:17 pr-md-02 systemd[1]: libpod-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May  7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728419093 -0700 PDT m=+0.073648421 container start 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May  7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728522035 -0700 PDT m=+0.073751360 container attach 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.728769341 -0700 PDT m=+0.073998662 container died 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=HEAD) May  7 17:07:17 pr-md-02 podman[2565736]: 2024-05-07 17:07:17.678321769 -0700 PDT m=+0.023551090 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:07:17 pr-md-02 podman[2565757]: 2024-05-07 17:07:17.78259101 -0700 PDT m=+0.046708660 container remove 877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=stoic_mayer, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:07:17 pr-md-02 systemd[1]: libpod-conmon-877f3f2f39644710f6a08499d230d3af827df70612c20d5e409ec36b554c4839.scope: Deactivated successfully. May  7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.90851263 -0700 PDT m=+0.039669496 container create 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) May  7 17:07:17 pr-md-02 systemd[1]: Started libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope. 
May  7 17:07:17 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.936966849 -0700 PDT m=+0.068123675 container init 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May  7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943702066 -0700 PDT m=+0.074858891 container start 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May  7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.943794994 -0700 PDT m=+0.074951827 container attach 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, ceph=True, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May  7 17:07:17 pr-md-02 podman[2565778]: 2024-05-07 17:07:17.892929482 -0700 PDT m=+0.024086307 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch May  7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:07:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch May  7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> passed data devices: 2 physical, 0 LVM May  7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> relative data size: 1.0 May  7 17:07:18 pr-md-02 intelligent_taussig[2565794]: --> All data devices are unavailable May  7 17:07:18 pr-md-02 systemd[1]: libpod-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May  7 17:07:18 pr-md-02 podman[2565778]: 2024-05-07 17:07:18.556739188 -0700 PDT m=+0.687896012 container died 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) May  7 17:07:18 pr-md-02 podman[2565822]: 2024-05-07 17:07:18.614711286 -0700 PDT m=+0.047414226 container remove 263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350 (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=intelligent_taussig, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git) May  7 17:07:18 pr-md-02 systemd[1]: libpod-conmon-263083cabee2763b55db565f8fc223d2af29c999d42e972cc67055e08986f350.scope: Deactivated successfully. May  7 17:07:18 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-2f63fd9ffd2763a7ee3f75befaf99b27a75325cd025cc2e74e72d7bd150afc42-merged.mount: Deactivated successfully. May  7 17:07:18 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225284: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.171860525 -0700 PDT m=+0.039261474 container create eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2) May  7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope. 
May  7 17:07:19 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.199519439 -0700 PDT m=+0.066920407 container init eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1) 
May  7 17:07:19 pr-md-02 xenodochial_wu[2565974]: 167 167
May  7 17:07:19 pr-md-02 systemd[1]: libpod-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May  7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.205924433 -0700 PDT m=+0.073325392 container start eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True) May  7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.2060646 -0700 PDT m=+0.073465559 container attach eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) May  7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.206298065 -0700 PDT m=+0.073699031 container died eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1) May  7 17:07:19 pr-md-02 podman[2565959]: 2024-05-07 17:07:19.158156321 -0700 PDT m=+0.025557273 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:07:19 pr-md-02 podman[2565979]: 2024-05-07 17:07:19.25994464 -0700 PDT m=+0.046894141 container remove eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=xenodochial_wu, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) May  7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-eb695a8059499edec16f6205fa8eabfecd83baf7f28b29860ac05430368bb96e.scope: Deactivated successfully. May  7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.388083595 -0700 PDT m=+0.043236647 container create a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS) May  7 17:07:19 pr-md-02 systemd[1]: Started libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope. 
May  7 17:07:19 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.417053374 -0700 PDT m=+0.072206356 container init a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222) May  7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423715752 -0700 PDT m=+0.078868733 container start a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) May  7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.423810643 -0700 PDT m=+0.078963634 container attach a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May  7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.371099332 -0700 PDT m=+0.026252317 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-91f5899d00d5794e67bbdd45fa72d22c74b0b977efec0238aa842ac272b635ce-merged.mount: Deactivated successfully. 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: {
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:    "41": [
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:        {
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "devices": [
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme0n1"
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            ],
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_size": "1920378863616", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_tags": "ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "path": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "tags": {
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1",
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "41",
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block",
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0"
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            },
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "type": "block", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc" 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:        }
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:    ],
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:    "43": [
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:        {
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "devices": [
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "/dev/nvme1n1"
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            ],
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_size": "1920378863616", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_tags": "ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "path": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "tags": {
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_device": "/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cephx_lockbox_secret": "AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.cluster_name": "ceph", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.crush_device_class": "", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.encrypted": "1",
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osd_id": "43",
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.osdspec_affinity": "md_encrypted_nvme_disks", 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.type": "block",
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: "ceph.vdo": "0"
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            },
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "type": "block", May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:            "vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a" 
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:        }
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]:    ]
May  7 17:07:19 pr-md-02 agitated_mayer[2566015]: }
May  7 17:07:19 pr-md-02 systemd[1]: libpod-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May  7 17:07:19 pr-md-02 podman[2566000]: 2024-05-07 17:07:19.801695942 -0700 PDT m=+0.456848932 container died a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image) May  7 17:07:19 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-68bc380ea9ec1cff282d08f862c7b661ba0571a20622225a5a3994e5fc88985d-merged.mount: Deactivated successfully. May  7 17:07:19 pr-md-02 podman[2566028]: 2024-05-07 17:07:19.854983202 -0700 PDT m=+0.045986655 container remove a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=agitated_mayer, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222) May  7 17:07:19 pr-md-02 systemd[1]: libpod-conmon-a9fae2fc56591b62da497f4015a6e693da36864c0ee3f0259fc0d5a2e796184d.scope: Deactivated successfully. May  7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.420915201 -0700 PDT m=+0.041065865 container create f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) May  7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope. 
May  7 17:07:20 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.45143613 -0700 PDT m=+0.071586769 container init f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_CLEAN=True, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 
May  7 17:07:20 pr-md-02 practical_wu[2566180]: 167 167
May  7 17:07:20 pr-md-02 systemd[1]: libpod-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May  7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457513493 -0700 PDT m=+0.077664122 container start f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True) May  7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457638676 -0700 PDT m=+0.077789308 container attach f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) May  7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.457913249 -0700 PDT m=+0.078063874 container died f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) May  7 17:07:20 pr-md-02 podman[2566165]: 2024-05-07 17:07:20.405495878 -0700 PDT m=+0.025646501 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:07:20 pr-md-02 podman[2566185]: 2024-05-07 17:07:20.515553442 -0700 PDT m=+0.045837844 container remove f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=practical_wu, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) May  7 17:07:20 pr-md-02 systemd[1]: libpod-conmon-f1af63ba0752e42553bce21029758749d47f2dccd9f6503a004e15a9c340bb0a.scope: Deactivated successfully. May  7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.631476902 -0700 PDT m=+0.043759539 container create c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0) May  7 17:07:20 pr-md-02 systemd[1]: Started libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope. 
May  7 17:07:20 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.660235834 -0700 PDT m=+0.072518401 container init c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True) May  7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.666915701 -0700 PDT m=+0.079198260 container start c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD) May  7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.667002215 -0700 PDT m=+0.079284787 container attach c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) May  7 17:07:20 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-dbb2679257a24186947b435237af458658d281389b531640688ba39cfa7cac79-merged.mount: Deactivated successfully. May  7 17:07:20 pr-md-02 podman[2566207]: 2024-05-07 17:07:20.6134279 -0700 PDT m=+0.025710470 image pull 5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8 May  7 17:07:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225285: 1313 pgs: 3 active+clean+scrubbing+deep, 8 active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail 
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]: {
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]: "50a79ade-482b-4c7e-84ae-0dac068ed570": { May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT", 
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "osd_id": 43,
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570", 
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "type": "bluestore"
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:    },
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]: "88070adf-ee15-406a-b8e9-98a61e294c8b": { May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440", May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO", 
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "osd_id": 41,
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b", 
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:        "type": "bluestore"
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]:    }
May  7 17:07:21 pr-md-02 quirky_clarke[2566222]: }
May  7 17:07:21 pr-md-02 systemd[1]: libpod-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May  7 17:07:21 pr-md-02 podman[2566207]: 2024-05-07 17:07:21.655826716 -0700 PDT m=+1.068109291 container died c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0) May  7 17:07:21 pr-md-02 systemd[1]: var-lib-containers-storage-overlay-e573f9cf164334d9b913d82ba760fa7f00e98646ea067755c99a3842969d30d8-merged.mount: Deactivated successfully. May  7 17:07:21 pr-md-02 podman[2566303]: 2024-05-07 17:07:21.711419562 -0700 PDT m=+0.047850993 container remove c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f (image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quirky_clarke, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1) May  7 17:07:21 pr-md-02 systemd[1]: libpod-conmon-c8952c7d94b5440b33b6562bed1ec270208bbfa94a25d32c81278e71b821495f.scope: Deactivated successfully. May  7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 57f8febe-215e-403c-821d-95da6e3c5073 does not exist May  7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 2637522f-3bd6-453c-976c-f7da78d12399 does not exist May  7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev b0cc7927-d150-4870-8ec7-6ef7d847ffd2 does not exist May  7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 9b053170-587d-42ee-929e-25112b3abba1 does not exist May  7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 352cfbfb-4b0a-420c-959b-e42615f9913d does not exist May  7 17:07:21 pr-md-02 ceph-mgr[5311]: [progress WARNING root] complete: ev 144dbb62-1eeb-49ba-b928-eee4a1756251 does not exist May  7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:07:22 pr-md-02 ceph-mon[5348]: from='mgr.274299 ' entity='mgr.pr-md-02.emffhz' May  7 17:07:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225286: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:23 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:07:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:07:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225287: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:26 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:26] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:07:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225288: 1313 pgs: 5 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1298 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:28 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:29 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225289: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:31 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225290: 1313 pgs: 6 active+clean+scrubbing+deep, 11 active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:33 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225291: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:33 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:35 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225292: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:36 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:36] "GET /metrics HTTP/1.1" 200 73532 "" "Prometheus/2.43.0" May  7 17:07:37 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225293: 1313 pgs: 4 active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:38 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:39 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225294: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:41 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225295: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:43 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225296: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:43 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:45 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225297: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:46 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:46] "GET /metrics HTTP/1.1" 200 73538 "" "Prometheus/2.43.0" May  7 17:07:47 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225298: 1313 pgs: 7 active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:48 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:49 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225299: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:51 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225300: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:53 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225301: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:53 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] scanning for idle connections.. May  7 17:07:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util] cleaning up connections: [] May  7 17:07:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225302: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:56 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:07:56] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May  7 17:07:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225303: 1313 pgs: 8 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:07:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:07:59 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:07:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225304: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225305: 1313 pgs: 8 active+clean+scrubbing+deep, 13 active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5614 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:08:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225306: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:04 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:08:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225307: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub starts May  7 17:08:06 pr-md-02 ceph-osd[6259]: log_channel(cluster) log [DBG] : purged_snaps scrub ok May  7 17:08:06 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:06] "GET /metrics HTTP/1.1" 200 73534 "" "Prometheus/2.43.0" May  7 17:08:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225308: 1313 pgs: 6 active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 e5615: 46 total, 46 up, 46 in May  7 17:08:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:08:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] MirrorSnapshotScheduleHandler: load_schedules May  7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Optimize plan auto_2024-05-08_00:08:07 May  7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode upmap, max misplaced 0.050000 
May  7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools ['cephfs_md_and_data', 'cephfs_data', '.mgr', 'cephfs_metadata'] May  7 17:08:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] prepared 0/10 changes May  7 17:08:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root] TrashPurgeScheduleHandler: load_schedules May  7 17:08:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225310: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:09 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:08:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag not found in client metadata May  7 17:08:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225311: 1313 pgs: 3 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 1308 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:08:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225312: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root] _maybe_adjust May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_metadata won't scale due to overlapping roots: {-15, -1} May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool cephfs_md_and_data won't scale due to overlapping roots: {-15, -1, -2} May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] Please See: https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 1 contains an overlapping root -1... skipping scaling May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 7 contains an overlapping root -15... skipping scaling May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 8 contains an overlapping root -2... skipping scaling May  7 17:08:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING root] pool 9 contains an overlapping root -2... skipping scaling May  7 17:08:14 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]: system:0 May  7 17:08:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225313: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:16 pr-md-02 ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]: ::ffff:10.50.1.74 - - [08/May/2024:00:08:16] "GET /metrics HTTP/1.1" 200 73536 "" "Prometheus/2.43.0" May  7 17:08:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log [DBG] : pgmap v225314: 1313 pgs: 4 active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1299 active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail May  7 17:08:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd e5615 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840 kv_alloc: 318767104 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #228441. Immutable memtables: 0. May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.105603) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:856] [default] [JOB 12102] Flushing memtable with next log file: 228441 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898105644, "job": 12102, "event": "flush_started", "num_memtables": 1, "num_entries": 2046, "num_deletes": 617, "total_data_size": 1704550, "memory_usage": 1744008, "flush_reason": "Manual Compaction"} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:885] [default] [JOB 12102] Level-0 flush table #228442: started May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898113765, "cf_name": "default", "job": 12102, "event": "table_file_creation", "file_number": 228442, "file_size": 847046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 141705570, "largest_seqno": 141707611, "table_properties": {"data_size": 839933, "index_size": 2991, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 3077, "raw_key_size": 32187, "raw_average_key_size": 26, "raw_value_size": 819966, "raw_average_value_size": 670, "num_data_blocks": 115, "num_entries": 1223, "num_filter_entries": 1223, "num_deletions": 617, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715126820, "oldest_key_time": 1715126820, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228442, "seqno_to_time_mapping": "N/A"}} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 12102] Flush lasted 8207 microseconds, and 2745 cpu microseconds. May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113810) [db/flush_job.cc:967] [default] [JOB 12102] Level-0 flush table #228442: 847046 bytes OK May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.113830) [db/memtable_list.cc:519] [default] Level-0 commit table #228442 started May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116583) [db/memtable_list.cc:722] [default] Level-0 commit table #228442: memtable #1 done May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116634) EVENT_LOG_v1 {"time_micros": 1715126898116612, "job": 12102, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0, 0, 1], "immutable_memtables": 0} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116650) [db/db_impl/db_impl_compaction_flush.cc:299] [default] Level summary: base level 6 level multiplier 10.00 max bytes base 268435456 files[1 0 0 0 0 0 1] max score 0.25 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/db_impl/db_impl_files.cc:463] [JOB 12102] Try to delete WAL files size 1693223, prev total WAL file size 1693550, number of live WAL files 2. May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228438.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.117035) [db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual compaction from level-0 to level-6 from '6D6772737461740032393030383037' seq:72057594037927935, type:22 .. '6D6772737461740032393031303630' seq:0, type:0; will stop at (end) May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1995] [default] [JOB 12103] Compacting 1@0 + 1@6 files to L6, score -1.00 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:2001] [default]: Compaction start summary: Base version 12102 Base level 0, inputs: [228442(827KB)], [228440(36MB)] May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898117062, "job": 12103, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [228442], "files_L6": [228440], "score": -1, "input_data_size": 39098338, "oldest_snapshot_seqno": -1} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [db/compaction/compaction_job.cc:1588] [default] [JOB 12103] Generated table #228443: 25379 keys, 37258556 bytes, temperature: kUnknown May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898320688, "cf_name": "default", "job": 12103, "event": "table_file_creation", "file_number": 228443, "file_size": 37258556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 37107602, "index_size": 86411, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 63493, "raw_key_size": 702042, "raw_average_key_size": 27, "raw_value_size": 36662378, "raw_average_value_size": 1444, "num_data_blocks": 3038, "num_entries": 25379, "num_filter_entries": 25379, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709312729, "oldest_key_time": 0, "file_creation_time": 1715126898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id": "LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228443, "seqno_to_time_mapping": "N/A"}} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.321010) [db/compaction/compaction_job.cc:1663] [default] [JOB 12103] Compacted 1@0 + 1@6 files to L6 => 37258556 bytes May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323275) [db/compaction/compaction_job.cc:865] [default] compacted to: base level 6 level multiplier 10.00 max bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 191.9 rd, 182.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB in(0.8, 36.5 +0.0 blob) out(35.5 +0.0 blob), read-write-amplify(90.1) write-amplify(44.0) OK, records in: 26602, records dropped: 1223 output_compression: NoCompression May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.323329) EVENT_LOG_v1 {"time_micros": 1715126898323308, "job": 12103, "event": "compaction_finished", "compaction_time_micros": 203764, "compaction_time_cpu_micros": 65018, "output_level": 6, "num_output_files": 1, "total_output_size": 37258556, "num_input_records": 26602, "num_output_records": 25379, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228442.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898323660, "job": 12103, "event": "table_file_deletion", "file_number": 228442} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-pr-md-02/store.db/228440.sst immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715126898326958, "job": 12103, "event": "table_file_deletion", "file_number": 228440} May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.116998) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326990) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.326994) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327002) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327006) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting May  7 17:08:18 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time 2024/05/08-00:08:18.327008) [db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual compaction starting
I have not updated any packages or anything, I'm not sure how this came about.  I did set the following two items earlier today, although I'm not sure if it would make any difference? 

ceph config set mds mds_recall_max_caps 3000
ceph config set mds mds_cache_memory_limit 16000000000

I tried to restart the MDS daemons alike so:

ceph orch daemon restart mds.slugfs.pr-md-02.sbblqq

And they seem to start for a moment then stop again.
I then tried to restart the one remaining one that was online, then it seemed to die as well, and now I see this: 

# ceph fs status
slugfs - 0 clients
======
RANK  STATE            MDS            ACTIVITY   DNS    INOS   DIRS   CAPS
  0    replay  slugfs.pr-md-01.xdtppo               0      0      0      0
  1    failed
        POOL           TYPE     USED  AVAIL
  cephfs_metadata    metadata   144G  3168G
cephfs_md_and_data    data       0   65.4T
    cephfs_data        data     838T   130T
MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) 

But the ceph-mds on pr-md-01 doesn't exist now.

Anyone know how to recover from this?  I'm kind of panicking.  :(

Thanks much,
erich

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux