Re: cephfs filesystem offline, ceph-mds core dumped... How to recover?

Eugen Block <eblock@xxxxxx> · Wed, 08 May 2024 19:48:24 +0000

The stack trace looks exactly like the one in this thread [1]. But I'm  
not sure how to apply the workaround, maybe stop all client IO and see  
if the MDS recovers? And then mount the clients with the option  
nowsync? Hopefully Xiubo (or someone else) will chime in here.

[1]  
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/OEU6PWVR5CQSVE7LRXKEW5X3AJPAYHBC/

Zitat von Erich Weiler <weiler@xxxxxxxxxxxx>:

My supervisor Jorge replied to Eugen but I'm copying the list with  
the log file, attached.

-erich

On 5/8/24 12:24 AM, Eugen Block wrote:
Hi Erich, I don't see anything useful in the syslog output. Can you  
instead paste only the mds log? On the mds host you can run

cephadm logs --name mds.slugfs.pr-md-02.sbblqq

or

journalctl -u ceph-{FSID}@mds.slugfs.pr-md-02.sbblqq

Zitat von Erich Weiler <weiler@xxxxxxxxxxxx>:

OK, we also tried reducing our three MDS daemons to only one:

# ceph orch apply mds slugfs --placement="1 pr-md-02.prism"

But it still seems stuck:

# ceph fs status
slugfs - 0 clients
======
RANK      STATE               MDS            ACTIVITY   DNS     
INOS DIRS   CAPS
 0    replay(laggy)  slugfs.pr-md-02.sbblqq               0      0 0      0
 1        failed
       POOL           TYPE     USED  AVAIL
 cephfs_metadata    metadata   144G  3168G
cephfs_md_and_data    data       0   65.4T
   cephfs_data        data     838T   130T

Logs from pr-md-02.prism below:

# tail -f /var/log/messages
May  7 20:25:53 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 20:25:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231554: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:25:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:03:25:56] "GET /metrics  
HTTP/1.1" 200 73369 "" "Prometheus/2.43.0"
May  7 20:25:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231555: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:25:57 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:25:58 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:25:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231556: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231557: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:02 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231558: 1313 pgs: 5 active+clean+scrubbing+deep, 6  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:03 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231559: 1313 pgs: 5 active+clean+scrubbing+deep, 6  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 9abdbcfe-a5ee-45ca-b32b-0a9b75a5b2f8 does not exist
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 6bbf1d1b-469f-4971-a217-ba32c62c9328 does not exist
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 411ec1ac-56f3-4b60-b025-eaf2996e2705 does not exist
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:06 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:03:26:06] "GET /metrics  
HTTP/1.1" 200 73369 "" "Prometheus/2.43.0"
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]:  
dispatch
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.080404999 -0700 PDT m=+0.038892155 container create  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
ceph=True)
May  7 20:26:07 pr-md-02 systemd[1]: Started  
libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope.
May  7 20:26:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.109800855 -0700 PDT m=+0.068287973 container init  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_BRANCH=HEAD)
May  7 20:26:07 pr-md-02 charming_shaw[2767109]: 167 167
May  7 20:26:07 pr-md-02 systemd[1]:  
libpod-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated  
successfully.
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.116328625 -0700 PDT m=+0.074815739 container start  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
io.buildah.version=1.29.1)
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.116466001 -0700 PDT m=+0.074953116 container attach  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git,  
ceph=True)
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.116702031 -0700 PDT m=+0.075189148 container died  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git,  
org.label-schema.license=GPLv2)
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.065184403 -0700 PDT m=+0.023671520 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:07 pr-md-02 podman[2767114]: 2024-05-07  
20:26:07.17137253 -0700 PDT m=+0.048040950 container remove  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image,  
org.label-schema.build-date=20240222)
May  7 20:26:07 pr-md-02 systemd[1]:  
libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated  
successfully.
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.294528483 -0700 PDT m=+0.038700338 container create  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 20:26:07 pr-md-02 systemd[1]: Started  
libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope.
May  7 20:26:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.326042029 -0700 PDT m=+0.070213877 container init  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
org.label-schema.schema-version=1.0)
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.33345637 -0700 PDT m=+0.077628218 container start  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True,  
org.label-schema.vendor=CentOS)
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.333546448 -0700 PDT m=+0.077718304 container attach  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD,  
org.label-schema.vendor=CentOS)
May  7 20:26:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231560: 1313 pgs: 5 active+clean+scrubbing+deep, 6  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.280097752 -0700 PDT m=+0.024269612 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:07 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> passed data  
devices: 2 physical, 0 LVM
May  7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> relative  
data size: 1.0
May  7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> All data  
devices are unavailable
May  7 20:26:07 pr-md-02 systemd[1]:  
libpod-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated  
successfully.
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.965949287 -0700 PDT m=+0.710121150 container died  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222,  
ceph=True)
May  7 20:26:08 pr-md-02 podman[2767179]: 2024-05-07  
20:26:08.023360488 -0700 PDT m=+0.050079180 container remove  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 20:26:08 pr-md-02 systemd[1]:  
libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-75dbf75fb6c31dcc473ffd4ca180f13e739046bb45c7161fb98d3db9afd17a40-merged.mount: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.57429109 -0700 PDT m=+0.039861869 container create  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:08 pr-md-02 systemd[1]: Started  
libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope.
May  7 20:26:08 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.604945119 -0700 PDT m=+0.070515899 container init  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:08 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:08 pr-md-02 inspiring_elion[2767333]: 167 167
May  7 20:26:08 pr-md-02 systemd[1]:  
libpod-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.612050835 -0700 PDT m=+0.077621696 container start  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD,  
RELEASE=HEAD)
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.612246287 -0700 PDT m=+0.077817071 container attach  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_CLEAN=True,  
ceph=True)
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.612503203 -0700 PDT m=+0.078073975 container died  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD,  
org.label-schema.license=GPLv2)
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.558251332 -0700 PDT m=+0.023822115 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:08 pr-md-02 podman[2767338]: 2024-05-07  
20:26:08.670336725 -0700 PDT m=+0.047749623 container remove  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base  
Image)
May  7 20:26:08 pr-md-02 systemd[1]:  
libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
MirrorSnapshotScheduleHandler: load_schedules
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.800520351 -0700 PDT m=+0.040282530 container create  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:08 pr-md-02 systemd[1]: Started  
libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope.
May  7 20:26:08 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.834540782 -0700 PDT m=+0.074302964 container init  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.8432442 -0700 PDT m=+0.083006378 container start  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.843360003 -0700 PDT m=+0.083122206 container attach  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.783755475 -0700 PDT m=+0.023517683 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:09 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-047f14812498e50e518bc4b80f672140ffa1509b5d508aa134fcb20a609a3189-merged.mount: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    "41": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "devices": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme0n1"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            ],
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_size": "1920378863616",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "tags": {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.block_device":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cephx_lockbox_secret":  
"AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_name": "ceph",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.crush_device_class": "",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid":  
"88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "41",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            },
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        }
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    ],
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    "43": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "devices": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme1n1"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            ],
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_size": "1920378863616",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "tags": {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.block_device":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cephx_lockbox_secret":  
"AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_name": "ceph",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.crush_device_class": "",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid":  
"50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "43",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            },
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        }
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    ]
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: }
May  7 20:26:09 pr-md-02 systemd[1]:  
libpod-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 podman[2767359]: 2024-05-07  
20:26:09.219197894 -0700 PDT m=+0.458960105 container died  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 20:26:09 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-f96acd560a7d6f711091ba778252072d08b84d701640411f20ac2c0a30d0d0a0-merged.mount: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 podman[2767388]: 2024-05-07  
20:26:09.276980938 -0700 PDT m=+0.050710570 container remove  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1,  
ceph=True)
May  7 20:26:09 pr-md-02 systemd[1]:  
libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
TrashPurgeScheduleHandler: load_schedules
May  7 20:26:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231561: 1313 pgs: 5 active+clean+scrubbing+deep, 5  
active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.236422812 -0700 PDT m=+0.037690490 container create  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
org.label-schema.schema-version=1.0)
May  7 20:26:10 pr-md-02 systemd[1]: Started  
libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope.
May  7 20:26:10 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.266363875 -0700 PDT m=+0.067631554 container init  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 20:26:10 pr-md-02 objective_goldstine[2767540]: 167 167
May  7 20:26:10 pr-md-02 systemd[1]:  
libpod-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated  
successfully.
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.273111289 -0700 PDT m=+0.074378967 container start  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_CLEAN=True)
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.273212161 -0700 PDT m=+0.074479839 container attach  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2,  
io.buildah.version=1.29.1)
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.273506844 -0700 PDT m=+0.074774521 container died  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
RELEASE=HEAD)
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.222032919 -0700 PDT m=+0.023300596 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:10 pr-md-02 podman[2767545]: 2024-05-07  
20:26:10.329094187 -0700 PDT m=+0.049135572 container remove  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1,  
GIT_CLEAN=True)
May  7 20:26:10 pr-md-02 systemd[1]:  
libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated  
successfully.
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.441995116 -0700 PDT m=+0.040292335 container create  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:10 pr-md-02 systemd[1]: Started  
libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope.
May  7 20:26:10 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.475459241 -0700 PDT m=+0.073756409 container init  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.482648828 -0700 PDT m=+0.080945996 container start  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.482737614 -0700 PDT m=+0.081034798 container attach  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True,  
GIT_CLEAN=True)
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.426366966 -0700 PDT m=+0.024664165 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:11 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-30d82c4f42c9c8e9838529715b3cb7fd1874f71c37de5570564e1346385ea590-merged.mount: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231562: 1313 pgs: 5 active+clean+scrubbing+deep, 5  
active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]: {
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:  
"50a79ade-482b-4c7e-84ae-0dac068ed570": {
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:        "osd_id": 43,
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"type": "bluestore"
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:    },
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:  
"88070adf-ee15-406a-b8e9-98a61e294c8b": {
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:        "osd_id": 41,
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"type": "bluestore"
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:    }
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]: }
May  7 20:26:11 pr-md-02 systemd[1]:  
libpod-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 podman[2767566]: 2024-05-07  
20:26:11.468853634 -0700 PDT m=+1.067150811 container died  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_BRANCH=HEAD)
May  7 20:26:11 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-8b793e8aa6dce35c6bd88479a14748d9d2eeba53693b0716213fb533d5bd9a2b-merged.mount: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 podman[2767662]: 2024-05-07  
20:26:11.526340084 -0700 PDT m=+0.050031409 container remove  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_BRANCH=HEAD)
May  7 20:26:11 pr-md-02 systemd[1]:  
libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev a64027d0-50d9-4df4-84e0-2d6fc7c28d01 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 9c614f97-f8ac-4106-bdc4-22a735263ae7 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 087c2c0d-6427-4c79-8604-8e8697a30ad3 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 92b105d6-1f55-4259-8118-6a5ff6cbf800 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 37ac8996-faa0-47bc-8584-d323f77d7b3f does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev f8b7279c-f34c-4451-9400-f78a5674f672 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Optimize plan auto_2024-05-08_03:26:11
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode  
upmap, max misplaced 0.050000
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
pools ['cephfs_metadata', '.mgr', 'cephfs_data',  
'cephfs_md_and_data']
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
prepared 0/10 changes
May  7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:12 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231563: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:13 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231564: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon) e5  
handle_command mon_command({"prefix": "status"} v 0) v1
May  7 20:26:15 pr-md-02 ceph-mon[5348]: log_channel(audit) log  
[DBG] : from='client.? 10.50.1.74:0/2874616819'  
entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
May  7 20:26:16 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:03:26:16] "GET /metrics  
HTTP/1.1" 200 73366 "" "Prometheus/2.43.0"
May  7 20:26:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231565: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:17 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:18 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(audit) log  
[DBG] : from='client.289994 -' entity='client.admin'  
cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231566: 1313 pgs: 3 active+clean+scrubbing+deep, 8  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root]  
_maybe_adjust
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_metadata won't scale due to overlapping roots:  
{-15, -1}
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_data won't scale due to overlapping roots: {-15,  
-1, -2}
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_md_and_data won't scale due to overlapping  
roots: {-15, -1, -2}
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 1 contains an overlapping root -1... skipping scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 7 contains an overlapping root -15... skipping scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 8 contains an overlapping root -2... skipping scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 9 contains an overlapping root -2... skipping scaling
May  7 20:26:21 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231567: 1313 pgs: 3 active+clean+scrubbing+deep, 8  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/db_impl/db_impl_write.cc:2098] [default] New memtable created  
with log file: #228921. Immutable memtables: 0.
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.115346)  
[db/db_impl/db_impl_compaction_flush.cc:2832] Calling  
FlushMemTableToOutputFile with column family [default], flush  
slots available 1, compaction slots available 1, flush slots  
scheduled 1, compaction slots scheduled 0
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/flush_job.cc:856] [default] [JOB 12422] Flushing memtable with  
next log file: 228921
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782115376, "job": 12422, "event":  
"flush_started", "num_memtables": 1, "num_entries": 3185,  
"num_deletes": 724, "total_data_size": 2772294, "memory_usage":  
2834928, "flush_reason": "Manual Compaction"}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/flush_job.cc:885] [default] [JOB 12422] Level-0 flush table  
#228922: started
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782128750, "cf_name": "default", "job":  
12422, "event": "table_file_creation", "file_number": 228922,  
"file_size": 1762231, "file_checksum": "",  
"file_checksum_func_name": "Unknown", "smallest_seqno": 142008149,  
"largest_seqno": 142011329, "table_properties": {"data_size":  
1750535, "index_size": 6357, "index_partitions": 0,  
"top_level_index_size": 0, "index_key_is_user_key": 1,  
"index_value_is_delta_encoded": 1, "filter_size": 4293,  
"raw_key_size": 43713, "raw_average_key_size": 25,  
"raw_value_size": 1720798, "raw_average_value_size": 1017,  
"num_data_blocks": 243, "num_entries": 1692, "num_filter_entries":  
1692, "num_deletions": 724, "num_merge_operands": 0,  
"num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0,  
"filter_policy": "bloomfilter", "column_family_name": "default",  
"column_family_id": 0, "comparator": "leveldb.BytewiseComparator",  
"merge_operator": "", "prefix_extractor_name": "nullptr",  
"property_collectors": "[]", "compression": "NoCompression",  
"compression_options": "window_bits=-14; level=32767; strategy=0;  
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0;  
max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ",  
"creation_time": 1715138637, "oldest_key_time": 1715138637,  
"file_creation_time": 1715138782,  
"slow_compression_estimated_data_size": 0,  
"fast_compression_estimated_data_size": 0, "db_id":  
"9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id":  
"LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228922,  
"seqno_to_time_mapping": "N/A"}}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/flush_job.cc:1019] [default] [JOB 12422] Flush lasted 13451  
microseconds, and 5016 cpu microseconds.
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.128796) [db/flush_job.cc:967] [default]  
[JOB 12422] Level-0 flush table #228922: 1762231 bytes OK
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.128813) [db/memtable_list.cc:519]  
[default] Level-0 commit table #228922 started
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.131803) [db/memtable_list.cc:722]  
[default] Level-0 commit table #228922: memtable #1 done
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.131858) EVENT_LOG_v1 {"time_micros":  
1715138782131834, "job": 12422, "event": "flush_finished",  
"output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0,  
0, 0, 1], "immutable_memtables": 0}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.131894)  
[db/db_impl/db_impl_compaction_flush.cc:299] [default] Level  
summary: base level 6 level multiplier 10.00 max bytes base  
268435456 files[1 0 0 0 0 0 1] max score 0.25
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/db_impl/db_impl_files.cc:463] [JOB 12422] Try to delete WAL  
files size 2755983, prev total WAL file size 2755983, number of  
live WAL files 2.
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228918.log immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.132654)  
[db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual  
compaction from level-0 to level-6 from  
'7061786F73003130313539323235' seq:72057594037927935, type:22 ..  
'7061786F73003130313539343737' seq:0, type:0; will stop at (end)
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1995] [default] [JOB 12423]  
Compacting 1@0 + 1@6 files to L6, score -1.00
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:2001] [default]: Compaction start  
summary: Base version 12422 Base level 0, inputs:  
[228922(1720KB)], [228920(30MB)]
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782132682, "job": 12423, "event":  
"compaction_started", "compaction_reason": "ManualCompaction",  
"files_L0": [228922], "files_L6": [228920], "score": -1,  
"input_data_size": 33301420, "oldest_snapshot_seqno": -1}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1588] [default] [JOB 12423]  
Generated table #228923: 24931 keys, 32057163 bytes, temperature:  
kUnknown
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782250508, "cf_name": "default", "job":  
12423, "event": "table_file_creation", "file_number": 228923,  
"file_size": 32057163, "file_checksum": "",  
"file_checksum_func_name": "Unknown", "smallest_seqno": 0,  
"largest_seqno": 0, "table_properties": {"data_size": 31912742,  
"index_size": 81030, "index_partitions": 0,  
"top_level_index_size": 0, "index_key_is_user_key": 1,  
"index_value_is_delta_encoded": 1, "filter_size": 62341,  
"raw_key_size": 693222, "raw_average_key_size": 27,  
"raw_value_size": 31478263, "raw_average_value_size": 1262,  
"num_data_blocks": 2833, "num_entries": 24931,  
"num_filter_entries": 24931, "num_deletions": 0,  
"num_merge_operands": 0, "num_range_deletions": 0,  
"format_version": 0, "fixed_key_len": 0, "filter_policy":  
"bloomfilter", "column_family_name": "default",  
"column_family_id": 0, "comparator": "leveldb.BytewiseComparator",  
"merge_operator": "", "prefix_extractor_name": "nullptr",  
"property_collectors": "[]", "compression": "NoCompression",  
"compression_options": "window_bits=-14; level=32767; strategy=0;  
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0;  
max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ",  
"creation_time": 1709312729, "oldest_key_time": 0,  
"file_creation_time": 1715138782,  
"slow_compression_estimated_data_size": 0,  
"fast_compression_estimated_data_size": 0, "db_id":  
"9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id":  
"LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228923,  
"seqno_to_time_mapping": "N/A"}}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.250767)  
[db/compaction/compaction_job.cc:1663] [default] [JOB 12423]  
Compacted 1@0 + 1@6 files to L6 => 32057163 bytes
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.253022)  
[db/compaction/compaction_job.cc:865] [default] compacted to: base  
level 6 level multiplier 10.00 max bytes base 268435456 files[0 0  
0 0 0 0 1] max score 0.00, MB/sec: 282.5 rd, 271.9 wr, level 6,  
files in(1, 1) out(1 +0 blob) MB in(1.7, 30.1 +0.0 blob) out(30.6  
+0.0 blob), read-write-amplify(37.1) write-amplify(18.2) OK,  
records in: 26401, records dropped: 1470 output_compression:  
NoCompression
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.253076) EVENT_LOG_v1 {"time_micros":  
1715138782253053, "job": 12423, "event": "compaction_finished",  
"compaction_time_micros": 117880, "compaction_time_cpu_micros":  
56129, "output_level": 6, "num_output_files": 1,  
"total_output_size": 32057163, "num_input_records": 26401,  
"num_output_records": 24931, "num_subcompactions": 1,  
"output_compression": "NoCompression",  
"num_single_delete_mismatches": 0,  
"num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0,  
0, 1]}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228922.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782253451, "job": 12423, "event":  
"table_file_deletion", "file_number": 228922}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228920.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782256177, "job": 12423, "event":  
"table_file_deletion", "file_number": 228920}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.132618)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.256209)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.256215)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.256218)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.256221)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-03:26:22.256225)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:23 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231568: 1313 pgs: 3 active+clean+scrubbing+deep, 8  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:23 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata

On 5/7/24 6:05 PM, Erich Weiler wrote:
Actually looking again, I don't see core dumps (though I could  
have sworn I did before).  But looking at my MDS servers, they  
appear to be in a constant cycle of trying to restart, and not  
making it, then trying again, etc.  The logs look like what I  
sent in my previous email, over and over again.

I tried restarting all three of my ceph-mds daemons at the same  
time, and it still doesn't seem to help.

I hope I can recover...

-erich

On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,

We just noticed that our cephfs filesystem was unreachable and  
we saw this:

# ceph -s
   cluster:
     id:     58bde08a-d7ed-11ee-9098-506b4b4da440
     health: HEALTH_WARN
             1 filesystem is degraded
             insufficient standby MDS daemons available

   services:
     mon: 5 daemons, quorum  
pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d)
     mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf
     mds: 2/2 daemons up
     osd: 46 osds: 46 up (since 3d), 46 in (since 8w)

   data:
     volumes: 0/1 healthy, 1 recovering
     pools:   4 pools, 1313 pgs
     objects: 309.96M objects, 558 TiB
     usage:   842 TiB used, 287 TiB / 1.1 PiB avail
     pgs:     1293 active+clean
              13   active+clean+scrubbing
              7    active+clean+scrubbing+deep

   io:
     client:   1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr

[root@pr-md-02 ~]# ceph health detail
HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS  
daemons available
[WRN] FS_DEGRADED: 1 filesystem is degraded
     fs slugfs is degraded
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons  
available
     have 0; want 1 more

And filesystem status is:

# ceph fs status
slugfs - 31 clients
======
RANK      STATE               MDS               ACTIVITY     DNS  
   INOS    DIRS   CAPS
  0    replay(laggy)  slugfs.pr-md-02.sbblqq                     
0      0       0      0
  1        active     slugfs.pr-md-01.xdtppo  Reqs:    0 /s   
7958   7959    13.1k   442
        POOL           TYPE     USED  AVAIL
  cephfs_metadata    metadata   144G  3168G
cephfs_md_and_data    data       0   65.4T
    cephfs_data        data     838T   130T
MDS version: ceph version 18.2.1  
(7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)

The ceph-mds container is not running on pr-md-02.  All OSDs are  
online.   I see this in the system logs on that machine (last  
1000 lines), I see some core dumping happening in there:

# tail -1000 /var/log/messages
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL  
files size 318677, prev total WAL file size 318677, number of  
live WAL files 2.
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio  
0.250000
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.485736)  
[db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual  
compaction from level-0 to level-6 from  
'7061786F73003130313337363339' seq:72057594037927935, type:22 ..  
'7061786F73003130313337383931' seq:0, type:0; will stop at (end)
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1995] [default] [JOB 12099]  
Compacting 1@0 + 1@6 files to L6, score -1.00
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:2001] [default]: Compaction  
start summary: Base version 12098 Base level 0, inputs:  
[228436(200KB)], [228434(37MB)]
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677485763, "job": 12099, "event":  
"compaction_started", "compaction_reason": "ManualCompaction",  
"files_L0": [228436], "files_L6": [228434], "score": -1,  
"input_data_size": 39197377, "oldest_snapshot_seqno": -1}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1588] [default] [JOB 12099]  
Generated table #228437: 25153 keys, 38148344 bytes,  
temperature: kUnknown
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677626467, "cf_name": "default", "job":  
12099, "event": "table_file_creation", "file_number": 228437,  
"file_size": 38148344, "file_checksum": "",  
"file_checksum_func_name": "Unknown", "smallest_seqno": 0,  
"largest_seqno": 0, "table_properties": {"data_size": 37996789,  
"index_size": 87588, "index_partitions": 0,  
"top_level_index_size": 0, "index_key_is_user_key": 1,  
"index_value_is_delta_encoded": 1, "filter_size": 62917,  
"raw_key_size": 697744, "raw_average_key_size": 27,  
"raw_value_size": 37553381, "raw_average_value_size": 1492,  
"num_data_blocks": 3073, "num_entries": 25153,  
"num_filter_entries": 25153, "num_deletions": 0,  
"num_merge_operands": 0, "num_range_deletions": 0,  
"format_version": 0, "fixed_key_len": 0, "filter_policy":  
"bloomfilter", "column_family_name": "default",  
"column_family_id": 0, "comparator":  
"leveldb.BytewiseComparator", "merge_operator": "",  
"prefix_extractor_name": "nullptr", "property_collectors": "[]",  
"compression": "NoCompression", "compression_options":  
"window_bits=-14; level=32767; strategy=0; max_dict_bytes=0;  
zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0;  
use_zstd_dict_trainer=1; ", "creation_time": 1709312729,  
"oldest_key_time": 0, "file_creation_time": 1715126677,  
"slow_compression_estimated_data_size": 0,  
"fast_compression_estimated_data_size": 0, "db_id":  
"9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id":  
"LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437,  
"seqno_to_time_mapping": "N/A"}}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.626709)  
[db/compaction/compaction_job.cc:1663] [default] [JOB 12099]  
Compacted 1@0 + 1@6 files to L6 => 38148344 bytes
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.629741)  
[db/compaction/compaction_job.cc:865] [default] compacted to:  
base level 6 level multiplier 10.00 max bytes base 268435456  
files[0 0 0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr,  
level 6, files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0  
blob) out(36.4 +0.0 blob), read-write-amplify(376.1)  
write-amplify(185.5) OK, records in: 25817, records dropped: 664  
output_compression: NoCompression
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros":  
1715126677629753, "job": 12099, "event": "compaction_finished",  
"compaction_time_micros": 140760, "compaction_time_cpu_micros":  
62440, "output_level": 6, "num_output_files": 1,  
"total_output_size": 38148344, "num_input_records": 25817,  
"num_output_records": 25153, "num_subcompactions": 1,  
"output_compression": "NoCompression",  
"num_single_delete_mismatches": 0,  
"num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0,  
0, 1]}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio  
0.250000
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677629887, "job": 12099, "event":  
"table_file_deletion", "file_number": 228436}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio  
0.250000
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677632998, "job": 12099, "event":  
"table_file_deletion", "file_number": 228434}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.485696)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633030)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633038)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633042)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633045)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633049)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:38 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225202: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s
May  7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225203: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:42 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225204: 1313 pgs: 9  
active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225205: 1313 pgs: 9  
active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:46 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics  
HTTP/1.1" 200 73533 "" "Prometheus/2.43.0"
May  7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225206: 1313 pgs: 9  
active+clean+scrubbing+deep, 12 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:47 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225207: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s
May  7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225208: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:51 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:04:52 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225209: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225210: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:04:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics  
HTTP/1.1" 200 73531 "" "Prometheus/2.43.0"
May  7 17:04:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225211: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:57 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225212: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s
May  7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225213: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:01 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:02 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225214: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]:  
dispatch
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.630854043 -0700 PDT m=+0.047521077 container create  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 systemd[1]: Started  
libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope.
May  7 17:05:03 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.671554205 -0700 PDT m=+0.088221217 container init  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167
May  7 17:05:03 pr-md-02 systemd[1]:  
libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated  
successfully.
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.679778085 -0700 PDT m=+0.096445105 container start  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.679881368 -0700 PDT m=+0.096548394 container attach  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True,  
org.label-schema.build-date=20240222)
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.680110752 -0700 PDT m=+0.096777772 container died  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.605825339 -0700 PDT m=+0.022492359 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07  
17:05:03.7398573 -0700 PDT m=+0.053387996 container remove  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
RELEASE=HEAD)
May  7 17:05:03 pr-md-02 systemd[1]:  
libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated  
successfully.
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.869704758 -0700 PDT m=+0.050589984 container create  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1,  
GIT_BRANCH=HEAD)
May  7 17:05:03 pr-md-02 systemd[1]: Started  
libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope.
May  7 17:05:03 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.901342134 -0700 PDT m=+0.082227374 container init  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.909537569 -0700 PDT m=+0.090422794 container start  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.909642002 -0700 PDT m=+0.090527227 container attach  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.845778509 -0700 PDT m=+0.026663744 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:04 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated  
successfully.
May  7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225215: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: -->  
passed data devices: 2 physical, 0 LVM
May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: -->  
relative data size: 1.0
May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All  
data devices are unavailable
May  7 17:05:05 pr-md-02 systemd[1]:  
libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated  
successfully.
May  7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07  
17:05:05.365433795 -0700 PDT m=+1.546319027 container died  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0,  
RELEASE=HEAD)
May  7 17:05:05 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated  
successfully.
May  7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07  
17:05:05.428169761 -0700 PDT m=+0.055435773 container remove  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS,  
org.label-schema.license=GPLv2)
May  7 17:05:05 pr-md-02 systemd[1]:  
libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.002418626 -0700 PDT m=+0.040341047 container create  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:06 pr-md-02 systemd[1]: Started  
libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope.
May  7 17:05:06 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.032819591 -0700 PDT m=+0.070742046 container init  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1,  
GIT_CLEAN=True)
May  7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.040138184 -0700 PDT m=+0.078060613 container start  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git,  
org.label-schema.schema-version=1.0)
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.040286008 -0700 PDT m=+0.078208464 container attach  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1,  
ceph=True)
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.040587551 -0700 PDT m=+0.078509982 container died  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0,  
GIT_BRANCH=HEAD)
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:05.987393175 -0700 PDT m=+0.025315605 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07  
17:05:06.097478996 -0700 PDT m=+0.050797295 container remove  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0,  
GIT_CLEAN=True)
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.228492423 -0700 PDT m=+0.042467581 container create  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 17:05:06 pr-md-02 systemd[1]: Started  
libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope.
May  7 17:05:06 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.263672711 -0700 PDT m=+0.077647878 container init  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.271891014 -0700 PDT m=+0.085866179 container start  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222,  
org.label-schema.vendor=CentOS)
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.27203062 -0700 PDT m=+0.086005784 container attach  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.213038456 -0700 PDT m=+0.027013622 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    "41": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "devices": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_size": "1920378863616",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "tags": {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.block_device":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cephx_lockbox_secret":  
"AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cluster_name": "ceph",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.crush_device_class": "",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid":  
"88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            },
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        }
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    "43": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "devices": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_size": "1920378863616",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "tags": {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.block_device":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cephx_lockbox_secret":  
"AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cluster_name": "ceph",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.crush_device_class": "",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid":  
"50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            },
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        }
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    ]
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: }
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.67755909 -0700 PDT m=+0.491534254 container died  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1,  
io.buildah.version=1.29.1)
May  7 17:05:06 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics  
HTTP/1.1" 200 73531 "" "Prometheus/2.43.0"
May  7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07  
17:05:06.745980606 -0700 PDT m=+0.056378222 container remove  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225216: 1313 pgs: 7  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:07 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated  
successfully.
May  7 17:05:07 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.344365868 -0700 PDT m=+0.042884110 container create  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:07 pr-md-02 systemd[1]: Started  
libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope.
May  7 17:05:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.374008883 -0700 PDT m=+0.072527132 container init  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167
May  7 17:05:07 pr-md-02 systemd[1]:  
libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated  
successfully.
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.381560483 -0700 PDT m=+0.080078731 container start  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True,  
GIT_BRANCH=HEAD)
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.38167372 -0700 PDT m=+0.080191969 container attach  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True,  
org.label-schema.schema-version=1.0)
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.38196021 -0700 PDT m=+0.080478458 container died  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.330060004 -0700 PDT m=+0.028578252 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07  
17:05:07.443179465 -0700 PDT m=+0.051396095 container remove  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:05:07 pr-md-02 systemd[1]:  
libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated  
successfully.
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
MirrorSnapshotScheduleHandler: load_schedules
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.581081662 -0700 PDT m=+0.046354470 container create  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 17:05:07 pr-md-02 systemd[1]: Started  
libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope.
May  7 17:05:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.613009264 -0700 PDT m=+0.078282065 container init  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.620807072 -0700 PDT m=+0.086079873 container start  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD,  
RELEASE=HEAD)
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.620921904 -0700 PDT m=+0.086194728 container attach  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.560418202 -0700 PDT m=+0.025691007 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Optimize plan auto_2024-05-08_00:05:07
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Mode upmap, max misplaced 0.050000
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data',  
'cephfs_metadata']
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
prepared 0/10 changes
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:08 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
TrashPurgeScheduleHandler: load_schedules
May  7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: {
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"50a79ade-482b-4c7e-84ae-0dac068ed570": {
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"osd_id": 43,
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"type": "bluestore"
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:    },
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"88070adf-ee15-406a-b8e9-98a61e294c8b": {
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"osd_id": 41,
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"type": "bluestore"
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:    }
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }
May  7 17:05:08 pr-md-02 systemd[1]:  
libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07  
17:05:08.623675315 -0700 PDT m=+1.088948125 container died  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
GIT_BRANCH=HEAD)
May  7 17:05:08 pr-md-02 systemd[1]:  
libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU  
time.
May  7 17:05:08 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07  
17:05:08.684978208 -0700 PDT m=+0.052557214 container remove  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 17:05:08 pr-md-02 systemd[1]:  
libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225217: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist
May  7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225218: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:05:12 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225219: 1313 pgs: 6  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:12 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO  
root] _maybe_adjust
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_metadata won't scale due to overlapping roots:  
{-15, -1}
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_data won't scale due to overlapping roots:  
{-15, -1, -2}
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_md_and_data won't scale due to overlapping  
roots: {-15, -1, -2}
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 1 contains an overlapping root -1... skipping scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 7 contains an overlapping root -15... skipping scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 8 contains an overlapping root -2... skipping scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 9 contains an overlapping root -2... skipping scaling
May  7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root.
May  7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope:  
Deactivated successfully.
May  7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225220: 1313 pgs: 6  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:16 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics  
HTTP/1.1" 200 73531 "" "Prometheus/2.43.0"
May  7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225221: 1313 pgs: 6  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:17 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:17 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225222: 1313 pgs: 6  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225223: 1313 pgs: 6  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:22 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5612 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225224: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s
May  7 17:05:22 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:22 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 e5613: 46 total, 46 up, 46 in
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225226: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:26 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225227: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:27 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:27 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225228: 1313 pgs: 7  
active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log  
[INF] : from='admin socket' entity='admin socket' cmd='smart'  
args=[json]: dispatch
May  7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225229: 1313 pgs: 7  
active+clean+scrubbing+deep, 10 active+clean+scrubbing, 1296  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log  
[INF] : from='admin socket' entity='admin socket' cmd=smart  
args=[json]: finished
May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:32 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225230: 1313 pgs: 4  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:33 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225231: 1313 pgs: 4  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s
May  7 17:05:36 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225232: 1313 pgs: 4  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s
May  7 17:05:37 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:38 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225233: 1313 pgs: 6  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s
May  7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225234: 1313 pgs: 6  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s
May  7 17:05:42 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225235: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s
May  7 17:05:43 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225236: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s
May  7 17:05:46 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics  
HTTP/1.1" 200 73534 "" "Prometheus/2.43.0"
May  7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225237: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s
May  7 17:05:47 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:48 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225238: 1313 pgs: 6  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB  
avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s
May  7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225239: 1313 pgs: 6  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1293  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:05:52 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225240: 1313 pgs: 5  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:05:53 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225241: 1313 pgs: 5  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:05:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225242: 1313 pgs: 5  
active+clean+scrubbing+deep, 16 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:05:57 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:05:58 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225243: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225244: 1313 pgs: 6  
active+clean+scrubbing+deep, 15 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:02 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225245: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:03 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225246: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:06 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225247: 1313 pgs: 7  
active+clean+scrubbing+deep, 14 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:07 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
MirrorSnapshotScheduleHandler: load_schedules
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Optimize plan auto_2024-05-08_00:06:07
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Mode upmap, max misplaced 0.050000
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
pools ['cephfs_md_and_data', '.mgr', 'cephfs_data',  
'cephfs_metadata']
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
prepared 0/10 changes
May  7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
TrashPurgeScheduleHandler: load_schedules
May  7 17:06:08 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225248: 1313 pgs: 4  
active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225249: 1313 pgs: 4  
active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1292  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.181584373 -0700 PDT m=+0.039979954 container create  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD,  
ceph=True)
May  7 17:06:11 pr-md-02 systemd[1]: Started  
libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope.
May  7 17:06:11 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.211049202 -0700 PDT m=+0.069444730 container init  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS,  
org.label-schema.license=GPLv2)
May  7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167
May  7 17:06:11 pr-md-02 systemd[1]:  
libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated  
successfully.
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.218033241 -0700 PDT m=+0.076428761 container start  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
org.label-schema.schema-version=1.0)
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.218168917 -0700 PDT m=+0.076564434 container attach  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.218487253 -0700 PDT m=+0.076882764 container died  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
ceph=True)
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.165130923 -0700 PDT m=+0.023526448 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07  
17:06:11.276198352 -0700 PDT m=+0.051426463 container remove  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD,  
ceph=True)
May  7 17:06:11 pr-md-02 systemd[1]:  
libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated  
successfully.
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.398685928 -0700 PDT m=+0.038305972 container create  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:11 pr-md-02 systemd[1]: Started  
libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope.
May  7 17:06:11 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.4284462 -0700 PDT m=+0.068066246 container init  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS,  
org.label-schema.schema-version=1.0)
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.436328165 -0700 PDT m=+0.075948196 container start  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.436459593 -0700 PDT m=+0.076079634 container attach  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.383297135 -0700 PDT m=+0.022917157 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]:  
dispatch
May  7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log  
[DBG] : from='client.296615 -' entity='client.admin'  
cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch
May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed  
data devices: 2 physical, 0 LVM
May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative  
data size: 1.0
May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data  
devices are unavailable
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07  
17:06:12.059347046 -0700 PDT m=+0.698967087 container died  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07  
17:06:12.115085789 -0700 PDT m=+0.048338618 container remove  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 ceph-mon[5348]:  
mon.pr-md-02@1(peon).osd e5613 _set_new_cache_sizes  
cache_size:1020054731 inc_alloc: 339738624 full_alloc: 356515840  
kv_alloc: 318767104
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.66977777 -0700 PDT m=+0.041271914 container create  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:06:12 pr-md-02 systemd[1]: Started  
libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope.
May  7 17:06:12 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.698793806 -0700 PDT m=+0.070287951 container init  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1,  
io.buildah.version=1.29.1)
May  7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.705992776 -0700 PDT m=+0.077486921 container start  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.706102938 -0700 PDT m=+0.077597093 container attach  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
RELEASE=HEAD)
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.706421977 -0700 PDT m=+0.077916136 container died  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222,  
org.label-schema.vendor=CentOS)
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.65486152 -0700 PDT m=+0.026355674 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07  
17:06:12.760218425 -0700 PDT m=+0.047515050 container remove  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base  
Image)
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225250: 1313 pgs: 5  
active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.883756999 -0700 PDT m=+0.038722013 container create  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 17:06:12 pr-md-02 systemd[1]: Started  
libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope.
May  7 17:06:12 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.915479032 -0700 PDT m=+0.070443976 container init  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base  
Image)
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.922714281 -0700 PDT m=+0.077679227 container start  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.922813112 -0700 PDT m=+0.077778070 container attach  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
org.label-schema.license=GPLv2)
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.867928578 -0700 PDT m=+0.022893523 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:13 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    "41": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name":  
"osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size":  
"1920378863616",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags":  
"ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid":  
"MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "tags": {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_device":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cephx_lockbox_secret":  
"AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_name": "ceph",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.crush_device_class": "",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.encrypted": "1",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            },
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name":  
"ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        }
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    "43": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name":  
"osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size":  
"1920378863616",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags":  
"ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid":  
"zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "tags": {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_device":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cephx_lockbox_secret":  
"AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_name": "ceph",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.crush_device_class": "",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.encrypted": "1",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            },
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name":  
"ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        }
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    ]
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: }
May  7 17:06:13 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:06:13 pr-md-02 systemd[1]:  
libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07  
17:06:13.298397993 -0700 PDT m=+0.453362947 container died  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:06:13 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07  
17:06:13.360359344 -0700 PDT m=+0.052612579 container remove  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
ceph=True)
May  7 17:06:13 pr-md-02 systemd[1]:  
libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO  
root] _maybe_adjust
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_metadata won't scale due to overlapping roots:  
{-15, -1}
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_data won't scale due to overlapping roots:  
{-15, -1, -2}
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_md_and_data won't scale due to overlapping  
roots: {-15, -1, -2}
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 1 contains an overlapping root -1... skipping scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 7 contains an overlapping root -15... skipping scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 8 contains an overlapping root -2... skipping scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 9 contains an overlapping root -2... skipping scaling
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.944018291 -0700 PDT m=+0.043243095 container create  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image,  
org.label-schema.vendor=CentOS)
May  7 17:06:13 pr-md-02 systemd[1]: Started  
libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope.
May  7 17:06:13 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.972707239 -0700 PDT m=+0.071931967 container init  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD,  
org.label-schema.vendor=CentOS)
May  7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167
May  7 17:06:13 pr-md-02 systemd[1]:  
libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.979610545 -0700 PDT m=+0.078835287 container start  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.979741103 -0700 PDT m=+0.078965840 container attach  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git,  
RELEASE=HEAD)
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.979972502 -0700 PDT m=+0.079197239 container died  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD,  
org.label-schema.vendor=CentOS)
May  7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.927504129 -0700 PDT m=+0.026728865 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07  
17:06:14.033457546 -0700 PDT m=+0.047525301 container remove  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD,  
org.label-schema.build-date=20240222)
May  7 17:06:14 pr-md-02 systemd[1]:  
libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated  
successfully.
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.146017051 -0700 PDT m=+0.040230170 container create  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:06:14 pr-md-02 systemd[1]: Started  
libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope.
May  7 17:06:14 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.175282354 -0700 PDT m=+0.069495491 container init  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2,  
org.label-schema.vendor=CentOS)
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.182850934 -0700 PDT m=+0.077064054 container start  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2,  
ceph=True)
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.182936358 -0700 PDT m=+0.077149475 container attach  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
org.label-schema.license=GPLv2)
May  7 17:06:14 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated  
successfully.
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.129225093 -0700 PDT m=+0.023438214 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster)  
log [DBG] : pgmap v225251: 1313 pgs: 5  
active+clean+scrubbing+deep, 17 active+clean+scrubbing, 1291  
active+clean; 558 TiB data, 842 TiB used, 287 TiB / 1.1 PiB avail
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: {
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:  
"50a79ade-482b-4c7e-84ae-0dac068ed570": {
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:  
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:         
"device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:         
"osd_id": 43,
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:  
"osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed
[Text gekürzt]

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx