Re: cephfs filesystem offline, ceph-mds core dumped... How to recover?

Eugen Block <eblock@xxxxxx> · Wed, 08 May 2024 07:24:13 +0000

Hi Erich, I don't see anything useful in the syslog output. Can you  
instead paste only the mds log? On the mds host you can run

cephadm logs --name mds.slugfs.pr-md-02.sbblqq

or

journalctl -u ceph-{FSID}@mds.slugfs.pr-md-02.sbblqq

Zitat von Erich Weiler <weiler@xxxxxxxxxxxx>:

OK, we also tried reducing our three MDS daemons to only one:

# ceph orch apply mds slugfs --placement="1 pr-md-02.prism"

But it still seems stuck:

# ceph fs status
slugfs - 0 clients
======
RANK      STATE               MDS            ACTIVITY   DNS    INOS  
DIRS   CAPS
 0    replay(laggy)  slugfs.pr-md-02.sbblqq               0      0 0      0
 1        failed
       POOL           TYPE     USED  AVAIL
 cephfs_metadata    metadata   144G  3168G
cephfs_md_and_data    data       0   65.4T
   cephfs_data        data     838T   130T

Logs from pr-md-02.prism below:

# tail -f /var/log/messages
May  7 20:25:53 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 20:25:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 20:25:55 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231554: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:25:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:03:25:56] "GET /metrics HTTP/1.1"  
200 73369 "" "Prometheus/2.43.0"
May  7 20:25:57 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231555: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:25:57 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:25:58 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:25:59 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231556: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:01 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231557: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:02 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:03 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231558: 1313 pgs: 5 active+clean+scrubbing+deep, 6  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:03 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:05 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231559: 1313 pgs: 5 active+clean+scrubbing+deep, 6  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:05 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 9abdbcfe-a5ee-45ca-b32b-0a9b75a5b2f8 does not exist
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 6bbf1d1b-469f-4971-a217-ba32c62c9328 does not exist
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 411ec1ac-56f3-4b60-b025-eaf2996e2705 does not exist
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:06 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:06 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:03:26:06] "GET /metrics HTTP/1.1"  
200 73369 "" "Prometheus/2.43.0"
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix":  
"auth get", "entity": "client.admin"}]: dispatch
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:06 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz' cmd=[{"prefix":  
"auth get", "entity": "client.bootstrap-osd"}]: dispatch
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.080404999 -0700 PDT m=+0.038892155 container create  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
ceph=True)
May  7 20:26:07 pr-md-02 systemd[1]: Started  
libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope.
May  7 20:26:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.109800855 -0700 PDT m=+0.068287973 container init  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_BRANCH=HEAD)
May  7 20:26:07 pr-md-02 charming_shaw[2767109]: 167 167
May  7 20:26:07 pr-md-02 systemd[1]:  
libpod-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated  
successfully.
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.116328625 -0700 PDT m=+0.074815739 container start  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
io.buildah.version=1.29.1)
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.116466001 -0700 PDT m=+0.074953116 container attach  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git,  
ceph=True)
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.116702031 -0700 PDT m=+0.075189148 container died  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git,  
org.label-schema.license=GPLv2)
May  7 20:26:07 pr-md-02 podman[2767093]: 2024-05-07  
20:26:07.065184403 -0700 PDT m=+0.023671520 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:07 pr-md-02 podman[2767114]: 2024-05-07  
20:26:07.17137253 -0700 PDT m=+0.048040950 container remove  
4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=charming_shaw, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image,  
org.label-schema.build-date=20240222)
May  7 20:26:07 pr-md-02 systemd[1]:  
libpod-conmon-4062d42d5e85a4d5321e061d5c8fb0a7f026b2563cc9288ed3f3b5878a014648.scope: Deactivated  
successfully.
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.294528483 -0700 PDT m=+0.038700338 container create  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 20:26:07 pr-md-02 systemd[1]: Started  
libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope.
May  7 20:26:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.326042029 -0700 PDT m=+0.070213877 container init  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
org.label-schema.schema-version=1.0)
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.33345637 -0700 PDT m=+0.077628218 container start  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True,  
org.label-schema.vendor=CentOS)
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.333546448 -0700 PDT m=+0.077718304 container attach  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=HEAD,  
org.label-schema.vendor=CentOS)
May  7 20:26:07 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231560: 1313 pgs: 5 active+clean+scrubbing+deep, 6  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.280097752 -0700 PDT m=+0.024269612 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:07 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> passed data  
devices: 2 physical, 0 LVM
May  7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> relative data  
size: 1.0
May  7 20:26:07 pr-md-02 youthful_bartik[2767151]: --> All data  
devices are unavailable
May  7 20:26:07 pr-md-02 systemd[1]:  
libpod-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated  
successfully.
May  7 20:26:07 pr-md-02 podman[2767136]: 2024-05-07  
20:26:07.965949287 -0700 PDT m=+0.710121150 container died  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222,  
ceph=True)
May  7 20:26:08 pr-md-02 podman[2767179]: 2024-05-07  
20:26:08.023360488 -0700 PDT m=+0.050079180 container remove  
38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=youthful_bartik, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 20:26:08 pr-md-02 systemd[1]:  
libpod-conmon-38fd6ea3060f4d7209ab226823553da7a054c648a8c2d23862656777936fc2bd.scope: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-75dbf75fb6c31dcc473ffd4ca180f13e739046bb45c7161fb98d3db9afd17a40-merged.mount: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.57429109 -0700 PDT m=+0.039861869 container create  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:08 pr-md-02 systemd[1]: Started  
libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope.
May  7 20:26:08 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.604945119 -0700 PDT m=+0.070515899 container init  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:08 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:08 pr-md-02 inspiring_elion[2767333]: 167 167
May  7 20:26:08 pr-md-02 systemd[1]:  
libpod-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.612050835 -0700 PDT m=+0.077621696 container start  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD,  
RELEASE=HEAD)
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.612246287 -0700 PDT m=+0.077817071 container attach  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_CLEAN=True,  
ceph=True)
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.612503203 -0700 PDT m=+0.078073975 container died  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, RELEASE=HEAD,  
org.label-schema.license=GPLv2)
May  7 20:26:08 pr-md-02 podman[2767317]: 2024-05-07  
20:26:08.558251332 -0700 PDT m=+0.023822115 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:08 pr-md-02 podman[2767338]: 2024-05-07  
20:26:08.670336725 -0700 PDT m=+0.047749623 container remove  
417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=inspiring_elion, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base  
Image)
May  7 20:26:08 pr-md-02 systemd[1]:  
libpod-conmon-417e3474c993b03f8f8da8cf58884c715bbaaabd2e2f29e9d111274425369595.scope: Deactivated  
successfully.
May  7 20:26:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
MirrorSnapshotScheduleHandler: load_schedules
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.800520351 -0700 PDT m=+0.040282530 container create  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:08 pr-md-02 systemd[1]: Started  
libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope.
May  7 20:26:08 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.834540782 -0700 PDT m=+0.074302964 container init  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.8432442 -0700 PDT m=+0.083006378 container start  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.843360003 -0700 PDT m=+0.083122206 container attach  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 20:26:08 pr-md-02 podman[2767359]: 2024-05-07  
20:26:08.783755475 -0700 PDT m=+0.023517683 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:09 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-047f14812498e50e518bc4b80f672140ffa1509b5d508aa134fcb20a609a3189-merged.mount: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    "41": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "devices": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme0n1"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            ],
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_size": "1920378863616",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "name":  
"osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "tags": {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.block_device":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid":  
"MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cephx_lockbox_secret":  
"AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_name": "ceph",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.crush_device_class": "",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid":  
"88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "41",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            },
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        }
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    ],
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    "43": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "devices": [
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "/dev/nvme1n1"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            ],
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_size": "1920378863616",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "name":  
"osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "tags": {
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.block_device":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.block_uuid":  
"zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cephx_lockbox_secret":  
"AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.cluster_name": "ceph",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.crush_device_class": "",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.encrypted": "1",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_fsid":  
"50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.osd_id": "43",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: "ceph.vdo": "0"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            },
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:            "type": "block",
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:             
"vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a"
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:        }
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]:    ]
May  7 20:26:09 pr-md-02 trusting_noyce[2767376]: }
May  7 20:26:09 pr-md-02 systemd[1]:  
libpod-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 podman[2767359]: 2024-05-07  
20:26:09.219197894 -0700 PDT m=+0.458960105 container died  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 20:26:09 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-f96acd560a7d6f711091ba778252072d08b84d701640411f20ac2c0a30d0d0a0-merged.mount: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 podman[2767388]: 2024-05-07  
20:26:09.276980938 -0700 PDT m=+0.050710570 container remove  
386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=trusting_noyce, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1,  
ceph=True)
May  7 20:26:09 pr-md-02 systemd[1]:  
libpod-conmon-386153d03a151f356c7d5ddf3ef82765b84d6e067b21e78223abfc5b9c78c3b6.scope: Deactivated  
successfully.
May  7 20:26:09 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
TrashPurgeScheduleHandler: load_schedules
May  7 20:26:09 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231561: 1313 pgs: 5 active+clean+scrubbing+deep, 5  
active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.236422812 -0700 PDT m=+0.037690490 container create  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
org.label-schema.schema-version=1.0)
May  7 20:26:10 pr-md-02 systemd[1]: Started  
libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope.
May  7 20:26:10 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.266363875 -0700 PDT m=+0.067631554 container init  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 20:26:10 pr-md-02 objective_goldstine[2767540]: 167 167
May  7 20:26:10 pr-md-02 systemd[1]:  
libpod-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated  
successfully.
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.273111289 -0700 PDT m=+0.074378967 container start  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_CLEAN=True)
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.273212161 -0700 PDT m=+0.074479839 container attach  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, org.label-schema.license=GPLv2,  
io.buildah.version=1.29.1)
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.273506844 -0700 PDT m=+0.074774521 container died  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
RELEASE=HEAD)
May  7 20:26:10 pr-md-02 podman[2767524]: 2024-05-07  
20:26:10.222032919 -0700 PDT m=+0.023300596 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:10 pr-md-02 podman[2767545]: 2024-05-07  
20:26:10.329094187 -0700 PDT m=+0.049135572 container remove  
3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=objective_goldstine, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1,  
GIT_CLEAN=True)
May  7 20:26:10 pr-md-02 systemd[1]:  
libpod-conmon-3d3712aa0a440c0b37589d22c68e97f90f37fd98c8ead84b68e5f44b48e888e7.scope: Deactivated  
successfully.
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.441995116 -0700 PDT m=+0.040292335 container create  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 20:26:10 pr-md-02 systemd[1]: Started  
libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope.
May  7 20:26:10 pr-md-02 systemd[1]: Started libcrun container.
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.475459241 -0700 PDT m=+0.073756409 container init  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, io.buildah.version=1.29.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.482648828 -0700 PDT m=+0.080945996 container start  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.482737614 -0700 PDT m=+0.081034798 container attach  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True,  
GIT_CLEAN=True)
May  7 20:26:10 pr-md-02 podman[2767566]: 2024-05-07  
20:26:10.426366966 -0700 PDT m=+0.024664165 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 20:26:11 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-30d82c4f42c9c8e9838529715b3cb7fd1874f71c37de5570564e1346385ea590-merged.mount: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231562: 1313 pgs: 5 active+clean+scrubbing+deep, 5  
active+clean+scrubbing, 1303 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]: {
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:  
"50a79ade-482b-4c7e-84ae-0dac068ed570": {
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:        "osd_id": 43,
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"osd_uuid": "50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:        "type":  
"bluestore"
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:    },
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:  
"88070adf-ee15-406a-b8e9-98a61e294c8b": {
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:        "osd_id": 41,
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:         
"osd_uuid": "88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:        "type":  
"bluestore"
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]:    }
May  7 20:26:11 pr-md-02 recursing_burnell[2767581]: }
May  7 20:26:11 pr-md-02 systemd[1]:  
libpod-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 podman[2767566]: 2024-05-07  
20:26:11.468853634 -0700 PDT m=+1.067150811 container died  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_CLEAN=True, io.buildah.version=1.29.1, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_BRANCH=HEAD)
May  7 20:26:11 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-8b793e8aa6dce35c6bd88479a14748d9d2eeba53693b0716213fb533d5bd9a2b-merged.mount: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 podman[2767662]: 2024-05-07  
20:26:11.526340084 -0700 PDT m=+0.050031409 container remove  
c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=recursing_burnell, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_BRANCH=HEAD)
May  7 20:26:11 pr-md-02 systemd[1]:  
libpod-conmon-c6f1361a91356f9c6abc9cee862864759140f38a7fe817262cbe5da0725a5824.scope: Deactivated  
successfully.
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev a64027d0-50d9-4df4-84e0-2d6fc7c28d01 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 9c614f97-f8ac-4106-bdc4-22a735263ae7 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 087c2c0d-6427-4c79-8604-8e8697a30ad3 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 92b105d6-1f55-4259-8118-6a5ff6cbf800 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 37ac8996-faa0-47bc-8584-d323f77d7b3f does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev f8b7279c-f34c-4451-9400-f78a5674f672 does not exist
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Optimize plan auto_2024-05-08_03:26:11
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode  
upmap, max misplaced 0.050000
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root] pools  
['cephfs_metadata', '.mgr', 'cephfs_data', 'cephfs_md_and_data']
May  7 20:26:11 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
prepared 0/10 changes
May  7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 20:26:12 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:13 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231563: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:13 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:15 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231564: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:15 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon) e5  
handle_command mon_command({"prefix": "status"} v 0) v1
May  7 20:26:15 pr-md-02 ceph-mon[5348]: log_channel(audit) log  
[DBG] : from='client.? 10.50.1.74:0/2874616819'  
entity='client.admin' cmd=[{"prefix": "status"}]: dispatch
May  7 20:26:16 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:03:26:16] "GET /metrics HTTP/1.1"  
200 73366 "" "Prometheus/2.43.0"
May  7 20:26:17 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231565: 1313 pgs: 4 active+clean+scrubbing+deep, 7  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:17 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:18 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(audit) log  
[DBG] : from='client.289994 -' entity='client.admin' cmd=[{"prefix":  
"fs status", "target": ["mon-mgr", ""]}]: dispatch
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231566: 1313 pgs: 3 active+clean+scrubbing+deep, 8  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root]  
_maybe_adjust
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_metadata won't scale due to overlapping roots:  
{-15, -1}
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_data won't scale due to overlapping roots: {-15,  
-1, -2}
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_md_and_data won't scale due to overlapping roots:  
{-15, -1, -2}
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 1 contains an overlapping root -1... skipping scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 7 contains an overlapping root -15... skipping scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 8 contains an overlapping root -2... skipping scaling
May  7 20:26:19 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 9 contains an overlapping root -2... skipping scaling
May  7 20:26:21 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231567: 1313 pgs: 3 active+clean+scrubbing+deep, 8  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/db_impl/db_impl_write.cc:2098] [default] New memtable created  
with log file: #228921. Immutable memtables: 0.
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.115346)  
[db/db_impl/db_impl_compaction_flush.cc:2832] Calling  
FlushMemTableToOutputFile with column family [default], flush slots  
available 1, compaction slots available 1, flush slots scheduled 1,  
compaction slots scheduled 0
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/flush_job.cc:856] [default] [JOB 12422] Flushing memtable with  
next log file: 228921
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782115376, "job": 12422, "event":  
"flush_started", "num_memtables": 1, "num_entries": 3185,  
"num_deletes": 724, "total_data_size": 2772294, "memory_usage":  
2834928, "flush_reason": "Manual Compaction"}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/flush_job.cc:885] [default] [JOB 12422] Level-0 flush table  
#228922: started
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782128750, "cf_name": "default", "job":  
12422, "event": "table_file_creation", "file_number": 228922,  
"file_size": 1762231, "file_checksum": "",  
"file_checksum_func_name": "Unknown", "smallest_seqno": 142008149,  
"largest_seqno": 142011329, "table_properties": {"data_size":  
1750535, "index_size": 6357, "index_partitions": 0,  
"top_level_index_size": 0, "index_key_is_user_key": 1,  
"index_value_is_delta_encoded": 1, "filter_size": 4293,  
"raw_key_size": 43713, "raw_average_key_size": 25, "raw_value_size":  
1720798, "raw_average_value_size": 1017, "num_data_blocks": 243,  
"num_entries": 1692, "num_filter_entries": 1692, "num_deletions":  
724, "num_merge_operands": 0, "num_range_deletions": 0,  
"format_version": 0, "fixed_key_len": 0, "filter_policy":  
"bloomfilter", "column_family_name": "default", "column_family_id":  
0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "",  
"prefix_extractor_name": "nullptr", "property_collectors": "[]",  
"compression": "NoCompression", "compression_options":  
"window_bits=-14; level=32767; strategy=0; max_dict_bytes=0;  
zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0;  
use_zstd_dict_trainer=1; ", "creation_time": 1715138637,  
"oldest_key_time": 1715138637, "file_creation_time": 1715138782,  
"slow_compression_estimated_data_size": 0,  
"fast_compression_estimated_data_size": 0, "db_id":  
"9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id":  
"LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228922,  
"seqno_to_time_mapping": "N/A"}}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/flush_job.cc:1019] [default] [JOB 12422] Flush lasted 13451  
microseconds, and 5016 cpu microseconds.
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.128796) [db/flush_job.cc:967] [default] [JOB  
12422] Level-0 flush table #228922: 1762231 bytes OK
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.128813) [db/memtable_list.cc:519] [default]  
Level-0 commit table #228922 started
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.131803) [db/memtable_list.cc:722] [default]  
Level-0 commit table #228922: memtable #1 done
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.131858) EVENT_LOG_v1 {"time_micros":  
1715138782131834, "job": 12422, "event": "flush_finished",  
"output_compression": "NoCompression", "lsm_state": [1, 0, 0, 0, 0,  
0, 1], "immutable_memtables": 0}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.131894)  
[db/db_impl/db_impl_compaction_flush.cc:299] [default] Level  
summary: base level 6 level multiplier 10.00 max bytes base  
268435456 files[1 0 0 0 0 0 1] max score 0.25
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/db_impl/db_impl_files.cc:463] [JOB 12422] Try to delete WAL  
files size 2755983, prev total WAL file size 2755983, number of live  
WAL files 2.
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228918.log immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.132654)  
[db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual  
compaction from level-0 to level-6 from  
'7061786F73003130313539323235' seq:72057594037927935, type:22 ..  
'7061786F73003130313539343737' seq:0, type:0; will stop at (end)
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1995] [default] [JOB 12423]  
Compacting 1@0 + 1@6 files to L6, score -1.00
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:2001] [default]: Compaction start  
summary: Base version 12422 Base level 0, inputs: [228922(1720KB)],  
[228920(30MB)]
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782132682, "job": 12423, "event":  
"compaction_started", "compaction_reason": "ManualCompaction",  
"files_L0": [228922], "files_L6": [228920], "score": -1,  
"input_data_size": 33301420, "oldest_snapshot_seqno": -1}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1588] [default] [JOB 12423]  
Generated table #228923: 24931 keys, 32057163 bytes, temperature:  
kUnknown
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782250508, "cf_name": "default", "job":  
12423, "event": "table_file_creation", "file_number": 228923,  
"file_size": 32057163, "file_checksum": "",  
"file_checksum_func_name": "Unknown", "smallest_seqno": 0,  
"largest_seqno": 0, "table_properties": {"data_size": 31912742,  
"index_size": 81030, "index_partitions": 0, "top_level_index_size":  
0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1,  
"filter_size": 62341, "raw_key_size": 693222,  
"raw_average_key_size": 27, "raw_value_size": 31478263,  
"raw_average_value_size": 1262, "num_data_blocks": 2833,  
"num_entries": 24931, "num_filter_entries": 24931, "num_deletions":  
0, "num_merge_operands": 0, "num_range_deletions": 0,  
"format_version": 0, "fixed_key_len": 0, "filter_policy":  
"bloomfilter", "column_family_name": "default", "column_family_id":  
0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "",  
"prefix_extractor_name": "nullptr", "property_collectors": "[]",  
"compression": "NoCompression", "compression_options":  
"window_bits=-14; level=32767; strategy=0; max_dict_bytes=0;  
zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0;  
use_zstd_dict_trainer=1; ", "creation_time": 1709312729,  
"oldest_key_time": 0, "file_creation_time": 1715138782,  
"slow_compression_estimated_data_size": 0,  
"fast_compression_estimated_data_size": 0, "db_id":  
"9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id":  
"LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228923,  
"seqno_to_time_mapping": "N/A"}}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.250767) [db/compaction/compaction_job.cc:1663]  
[default] [JOB 12423] Compacted 1@0 + 1@6 files to L6 => 32057163  
bytes
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.253022) [db/compaction/compaction_job.cc:865]  
[default] compacted to: base level 6 level multiplier 10.00 max  
bytes base 268435456 files[0 0 0 0 0 0 1] max score 0.00, MB/sec:  
282.5 rd, 271.9 wr, level 6, files in(1, 1) out(1 +0 blob) MB  
in(1.7, 30.1 +0.0 blob) out(30.6 +0.0 blob),  
read-write-amplify(37.1) write-amplify(18.2) OK, records in: 26401,  
records dropped: 1470 output_compression: NoCompression
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.253076) EVENT_LOG_v1 {"time_micros":  
1715138782253053, "job": 12423, "event": "compaction_finished",  
"compaction_time_micros": 117880, "compaction_time_cpu_micros":  
56129, "output_level": 6, "num_output_files": 1,  
"total_output_size": 32057163, "num_input_records": 26401,  
"num_output_records": 24931, "num_subcompactions": 1,  
"output_compression": "NoCompression",  
"num_single_delete_mismatches": 0, "num_single_delete_fallthrough":  
0, "lsm_state": [0, 0, 0, 0, 0, 0, 1]}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228922.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782253451, "job": 12423, "event":  
"table_file_deletion", "file_number": 228922}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228920.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715138782256177, "job": 12423, "event":  
"table_file_deletion", "file_number": 228920}
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.132618)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.256209)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.256215)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.256218)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.256221)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log Time  
2024/05/08-03:26:22.256225)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 20:26:22 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 20:26:23 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v231568: 1313 pgs: 3 active+clean+scrubbing+deep, 8  
active+clean+scrubbing, 1302 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 20:26:23 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5678 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 20:26:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root] cmdtag  
not found in client metadata

On 5/7/24 6:05 PM, Erich Weiler wrote:
Actually looking again, I don't see core dumps (though I could have  
sworn I did before).  But looking at my MDS servers, they appear to  
be in a constant cycle of trying to restart, and not making it,  
then trying again, etc.  The logs look like what I sent in my  
previous email, over and over again.

I tried restarting all three of my ceph-mds daemons at the same  
time, and it still doesn't seem to help.

I hope I can recover...

-erich

On 5/7/24 5:18 PM, Erich Weiler wrote:
Hi All,

We just noticed that our cephfs filesystem was unreachable and we saw this:

# ceph -s
   cluster:
     id:     58bde08a-d7ed-11ee-9098-506b4b4da440
     health: HEALTH_WARN
             1 filesystem is degraded
             insufficient standby MDS daemons available

   services:
     mon: 5 daemons, quorum  
pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d)
     mgr: pr-md-02.emffhz(active, since 5d), standbys: pr-md-01.jemmdf
     mds: 2/2 daemons up
     osd: 46 osds: 46 up (since 3d), 46 in (since 8w)

   data:
     volumes: 0/1 healthy, 1 recovering
     pools:   4 pools, 1313 pgs
     objects: 309.96M objects, 558 TiB
     usage:   842 TiB used, 287 TiB / 1.1 PiB avail
     pgs:     1293 active+clean
              13   active+clean+scrubbing
              7    active+clean+scrubbing+deep

   io:
     client:   1.5 KiB/s rd, 65 KiB/s wr, 1 op/s rd, 2 op/s wr

[root@pr-md-02 ~]# ceph health detail
HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS  
daemons available
[WRN] FS_DEGRADED: 1 filesystem is degraded
     fs slugfs is degraded
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
     have 0; want 1 more

And filesystem status is:

# ceph fs status
slugfs - 31 clients
======
RANK      STATE               MDS               ACTIVITY     DNS    
 INOS    DIRS   CAPS
  0    replay(laggy)  slugfs.pr-md-02.sbblqq                    0   
    0       0      0
  1        active     slugfs.pr-md-01.xdtppo  Reqs:    0 /s  7958   
 7959    13.1k   442
        POOL           TYPE     USED  AVAIL
  cephfs_metadata    metadata   144G  3168G
cephfs_md_and_data    data       0   65.4T
    cephfs_data        data     838T   130T
MDS version: ceph version 18.2.1  
(7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)

The ceph-mds container is not running on pr-md-02.  All OSDs are  
online.   I see this in the system logs on that machine (last 1000  
lines), I see some core dumping happening in there:

# tail -1000 /var/log/messages
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/db_impl/db_impl_files.cc:463] [JOB 12098] Try to delete WAL  
files size 318677, prev total WAL file size 318677, number of live  
WAL files 2.
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228432.log immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.485736)  
[db/db_impl/db_impl_compaction_flush.cc:3165] [default] Manual  
compaction from level-0 to level-6 from  
'7061786F73003130313337363339' seq:72057594037927935, type:22 ..  
'7061786F73003130313337383931' seq:0, type:0; will stop at (end)
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1995] [default] [JOB 12099]  
Compacting 1@0 + 1@6 files to L6, score -1.00
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:2001] [default]: Compaction start  
summary: Base version 12098 Base level 0, inputs: [228436(200KB)],  
[228434(37MB)]
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677485763, "job": 12099, "event":  
"compaction_started", "compaction_reason": "ManualCompaction",  
"files_L0": [228436], "files_L6": [228434], "score": -1,  
"input_data_size": 39197377, "oldest_snapshot_seqno": -1}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[db/compaction/compaction_job.cc:1588] [default] [JOB 12099]  
Generated table #228437: 25153 keys, 38148344 bytes, temperature:  
kUnknown
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677626467, "cf_name": "default", "job":  
12099, "event": "table_file_creation", "file_number": 228437,  
"file_size": 38148344, "file_checksum": "",  
"file_checksum_func_name": "Unknown", "smallest_seqno": 0,  
"largest_seqno": 0, "table_properties": {"data_size": 37996789,  
"index_size": 87588, "index_partitions": 0,  
"top_level_index_size": 0, "index_key_is_user_key": 1,  
"index_value_is_delta_encoded": 1, "filter_size": 62917,  
"raw_key_size": 697744, "raw_average_key_size": 27,  
"raw_value_size": 37553381, "raw_average_value_size": 1492,  
"num_data_blocks": 3073, "num_entries": 25153,  
"num_filter_entries": 25153, "num_deletions": 0,  
"num_merge_operands": 0, "num_range_deletions": 0,  
"format_version": 0, "fixed_key_len": 0, "filter_policy":  
"bloomfilter", "column_family_name": "default",  
"column_family_id": 0, "comparator": "leveldb.BytewiseComparator",  
"merge_operator": "", "prefix_extractor_name": "nullptr",  
"property_collectors": "[]", "compression": "NoCompression",  
"compression_options": "window_bits=-14; level=32767; strategy=0;  
max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0;  
max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ",  
"creation_time": 1709312729, "oldest_key_time": 0,  
"file_creation_time": 1715126677,  
"slow_compression_estimated_data_size": 0,  
"fast_compression_estimated_data_size": 0, "db_id":  
"9cddaea3-96ea-4679-bc89-ab4fe1d01ddc", "db_session_id":  
"LVB8G4XNY4SWBL1V508Z", "orig_file_number": 228437,  
"seqno_to_time_mapping": "N/A"}}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.626709)  
[db/compaction/compaction_job.cc:1663] [default] [JOB 12099]  
Compacted 1@0 + 1@6 files to L6 => 38148344 bytes
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.629741)  
[db/compaction/compaction_job.cc:865] [default] compacted to: base  
level 6 level multiplier 10.00 max bytes base 268435456 files[0 0  
0 0 0 0 1] max score 0.00, MB/sec: 278.5 rd, 271.0 wr, level 6,  
files in(1, 1) out(1 +0 blob) MB in(0.2, 37.2 +0.0 blob) out(36.4  
+0.0 blob), read-write-amplify(376.1) write-amplify(185.5) OK,  
records in: 25817, records dropped: 664 output_compression:  
NoCompression
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.629762) EVENT_LOG_v1 {"time_micros":  
1715126677629753, "job": 12099, "event": "compaction_finished",  
"compaction_time_micros": 140760, "compaction_time_cpu_micros":  
62440, "output_level": 6, "num_output_files": 1,  
"total_output_size": 38148344, "num_input_records": 25817,  
"num_output_records": 25153, "num_subcompactions": 1,  
"output_compression": "NoCompression",  
"num_single_delete_mismatches": 0,  
"num_single_delete_fallthrough": 0, "lsm_state": [0, 0, 0, 0, 0,  
0, 1]}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228436.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677629887, "job": 12099, "event":  
"table_file_deletion", "file_number": 228436}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb:  
[file/delete_scheduler.cc:74] Deleted file  
/var/lib/ceph/mon/ceph-pr-md-02/store.db/228434.sst immediately,  
rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: EVENT_LOG_v1  
{"time_micros": 1715126677632998, "job": 12099, "event":  
"table_file_deletion", "file_number": 228434}
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.485696)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633030)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633038)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633042)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633045)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:37 pr-md-02 ceph-mon[5348]: rocksdb: (Original Log  
Time 2024/05/08-00:04:37.633049)  
[db/db_impl/db_impl_compaction_flush.cc:1903] [default] Manual  
compaction starting
May  7 17:04:38 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:04:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225202: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s
May  7 17:04:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225203: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:04:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225204: 1313 pgs: 9 active+clean+scrubbing+deep, 12  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225205: 1313 pgs: 9 active+clean+scrubbing+deep, 12  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:46 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:46 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:04:46] "GET /metrics  
HTTP/1.1" 200 73533 "" "Prometheus/2.43.0"
May  7 17:04:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225206: 1313 pgs: 9 active+clean+scrubbing+deep, 12  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:04:47 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:04:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225207: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s
May  7 17:04:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225208: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:51 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:04:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:04:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225209: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225210: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:04:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:04:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:04:56] "GET /metrics  
HTTP/1.1" 200 73531 "" "Prometheus/2.43.0"
May  7 17:04:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:04:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225211: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:04:57 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:04:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:04:58 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:04:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225212: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s
May  7 17:05:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225213: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:01 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:01 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225214: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 15bb48f0-f983-4282-b9c0-19dd7e9bf550 does not exist
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 4e81461d-8388-45ab-91e8-338e82465841 does not exist
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev e28e821d-f2eb-42d1-b9f8-a99c53e67866 does not exist
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:03 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]:  
dispatch
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.630854043 -0700 PDT m=+0.047521077 container create  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 systemd[1]: Started  
libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope.
May  7 17:05:03 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.671554205 -0700 PDT m=+0.088221217 container init  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 jovial_solomon[2564016]: 167 167
May  7 17:05:03 pr-md-02 systemd[1]:  
libpod-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated  
successfully.
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.679778085 -0700 PDT m=+0.096445105 container start  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.679881368 -0700 PDT m=+0.096548394 container attach  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, GIT_CLEAN=True,  
org.label-schema.build-date=20240222)
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.680110752 -0700 PDT m=+0.096777772 container died  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 podman[2564001]: 2024-05-07  
17:05:03.605825339 -0700 PDT m=+0.022492359 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:03 pr-md-02 podman[2564022]: 2024-05-07  
17:05:03.7398573 -0700 PDT m=+0.053387996 container remove  
268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=jovial_solomon, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
RELEASE=HEAD)
May  7 17:05:03 pr-md-02 systemd[1]:  
libpod-conmon-268bc85e5ba7c5b788c4460fe5b242e07a211d4808c84158ad0d2b3e178b9f4b.scope: Deactivated  
successfully.
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.869704758 -0700 PDT m=+0.050589984 container create  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1,  
GIT_BRANCH=HEAD)
May  7 17:05:03 pr-md-02 systemd[1]: Started  
libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope.
May  7 17:05:03 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.901342134 -0700 PDT m=+0.082227374 container init  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, RELEASE=HEAD,  
GIT_CLEAN=True)
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.909537569 -0700 PDT m=+0.090422794 container start  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.909642002 -0700 PDT m=+0.090527227 container attach  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:05:03 pr-md-02 podman[2564043]: 2024-05-07  
17:05:03.845778509 -0700 PDT m=+0.026663744 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:04 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-ddfb8ca9d0c69b0b759623d12dc51317ef66a0ae06c62015987327a7630ca3b2-merged.mount: Deactivated  
successfully.
May  7 17:05:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225215: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> passed  
data devices: 2 physical, 0 LVM
May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: -->  
relative data size: 1.0
May  7 17:05:05 pr-md-02 friendly_brahmagupta[2564058]: --> All  
data devices are unavailable
May  7 17:05:05 pr-md-02 systemd[1]:  
libpod-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated  
successfully.
May  7 17:05:05 pr-md-02 podman[2564043]: 2024-05-07  
17:05:05.365433795 -0700 PDT m=+1.546319027 container died  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0,  
RELEASE=HEAD)
May  7 17:05:05 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-dfc8fb25041eaa38ebd863e2dc2eea81db94bca7926c18bf08635dd8897a3d08-merged.mount: Deactivated  
successfully.
May  7 17:05:05 pr-md-02 podman[2564087]: 2024-05-07  
17:05:05.428169761 -0700 PDT m=+0.055435773 container remove  
ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=friendly_brahmagupta, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS,  
org.label-schema.license=GPLv2)
May  7 17:05:05 pr-md-02 systemd[1]:  
libpod-conmon-ec12311fc2494eb8b7529762d64377e27f0c6ff34d92242ba7776aa29f35125b.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.002418626 -0700 PDT m=+0.040341047 container create  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_CLEAN=True, RELEASE=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:06 pr-md-02 systemd[1]: Started  
libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope.
May  7 17:05:06 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.032819591 -0700 PDT m=+0.070742046 container init  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1,  
GIT_CLEAN=True)
May  7 17:05:06 pr-md-02 modest_austin[2564240]: 167 167
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.040138184 -0700 PDT m=+0.078060613 container start  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git,  
org.label-schema.schema-version=1.0)
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.040286008 -0700 PDT m=+0.078208464 container attach  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1,  
ceph=True)
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:06.040587551 -0700 PDT m=+0.078509982 container died  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0,  
GIT_BRANCH=HEAD)
May  7 17:05:06 pr-md-02 podman[2564224]: 2024-05-07  
17:05:05.987393175 -0700 PDT m=+0.025315605 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:06 pr-md-02 podman[2564245]: 2024-05-07  
17:05:06.097478996 -0700 PDT m=+0.050797295 container remove  
d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=modest_austin, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0,  
GIT_CLEAN=True)
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-conmon-d98b8a9e74376baf21cdc6e0712a8035e63483f1d678949a86b48646e7f35c38.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.228492423 -0700 PDT m=+0.042467581 container create  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 17:05:06 pr-md-02 systemd[1]: Started  
libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope.
May  7 17:05:06 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.263672711 -0700 PDT m=+0.077647878 container init  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.271891014 -0700 PDT m=+0.085866179 container start  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222,  
org.label-schema.vendor=CentOS)
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.27203062 -0700 PDT m=+0.086005784 container attach  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.213038456 -0700 PDT m=+0.027013622 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    "41": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "devices": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme0n1"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_size": "1920378863616",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "name":  
"osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "tags": {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.block_device":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid":  
"MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cephx_lockbox_secret":  
"AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.crush_device_class": "",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid":  
"88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "41",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            },
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"vg_name": "ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        }
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    "43": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "devices": [
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "/dev/nvme1n1"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            ],
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_size": "1920378863616",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_tags":  
"ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"lv_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "name":  
"osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "tags": {
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.block_device":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.block_uuid":  
"zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cephx_lockbox_secret":  
"AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.cluster_name": "ceph",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.crush_device_class": "",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.encrypted": "1",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_fsid":  
"50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.osd_id": "43",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: "ceph.vdo": "0"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            },
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:            "type": "block",
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:             
"vg_name": "ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a"
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:        }
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]:    ]
May  7 17:05:06 pr-md-02 goofy_tesla[2564282]: }
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 podman[2564266]: 2024-05-07  
17:05:06.67755909 -0700 PDT m=+0.491534254 container died  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1,  
io.buildah.version=1.29.1)
May  7 17:05:06 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:06] "GET /metrics  
HTTP/1.1" 200 73531 "" "Prometheus/2.43.0"
May  7 17:05:06 pr-md-02 podman[2564295]: 2024-05-07  
17:05:06.745980606 -0700 PDT m=+0.056378222 container remove  
39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_tesla, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:05:06 pr-md-02 systemd[1]:  
libpod-conmon-39499009a5ee7478a058cad0a646616d3d0b62fed532ddd1b2bbdb44128c30eb.scope: Deactivated  
successfully.
May  7 17:05:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225216: 1313 pgs: 7 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:07 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-522d9498e68f9764cedb1b2ff926f774718727cdd52974d491a188f93a2c19f6-merged.mount: Deactivated  
successfully.
May  7 17:05:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.344365868 -0700 PDT m=+0.042884110 container create  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:05:07 pr-md-02 systemd[1]: Started  
libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope.
May  7 17:05:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.374008883 -0700 PDT m=+0.072527132 container init  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 17:05:07 pr-md-02 goofy_pare[2564446]: 167 167
May  7 17:05:07 pr-md-02 systemd[1]:  
libpod-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated  
successfully.
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.381560483 -0700 PDT m=+0.080078731 container start  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True,  
GIT_BRANCH=HEAD)
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.38167372 -0700 PDT m=+0.080191969 container attach  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, RELEASE=HEAD, GIT_CLEAN=True,  
org.label-schema.schema-version=1.0)
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.38196021 -0700 PDT m=+0.080478458 container died  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:05:07 pr-md-02 podman[2564431]: 2024-05-07  
17:05:07.330060004 -0700 PDT m=+0.028578252 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:07 pr-md-02 podman[2564451]: 2024-05-07  
17:05:07.443179465 -0700 PDT m=+0.051396095 container remove  
748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=goofy_pare, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, RELEASE=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:05:07 pr-md-02 systemd[1]:  
libpod-conmon-748094f58d93b27e8acc845f1a3e6003c9e1b03fb568f6806ffb4cd17ebe21dd.scope: Deactivated  
successfully.
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
MirrorSnapshotScheduleHandler: load_schedules
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.581081662 -0700 PDT m=+0.046354470 container create  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 17:05:07 pr-md-02 systemd[1]: Started  
libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope.
May  7 17:05:07 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.613009264 -0700 PDT m=+0.078282065 container init  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.620807072 -0700 PDT m=+0.086079873 container start  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD,  
RELEASE=HEAD)
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.620921904 -0700 PDT m=+0.086194728 container attach  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 17:05:07 pr-md-02 podman[2564472]: 2024-05-07  
17:05:07.560418202 -0700 PDT m=+0.025691007 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Optimize plan auto_2024-05-08_00:05:07
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode  
upmap, max misplaced 0.050000
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
pools ['.mgr', 'cephfs_md_and_data', 'cephfs_data',  
'cephfs_metadata']
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
prepared 0/10 changes
May  7 17:05:07 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:08 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-a2a58f86faa3a65838beaf8bcb779039975420b572508bf98328122f6618f26b-merged.mount: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
TrashPurgeScheduleHandler: load_schedules
May  7 17:05:08 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: {
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"50a79ade-482b-4c7e-84ae-0dac068ed570": {
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "osd_id": 43,
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid":  
"50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"type": "bluestore"
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:    },
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"88070adf-ee15-406a-b8e9-98a61e294c8b": {
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:  
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:        "osd_id": 41,
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: "osd_uuid":  
"88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:         
"type": "bluestore"
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]:    }
May  7 17:05:08 pr-md-02 quizzical_antonelli[2564488]: }
May  7 17:05:08 pr-md-02 systemd[1]:  
libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 podman[2564472]: 2024-05-07  
17:05:08.623675315 -0700 PDT m=+1.088948125 container died  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
GIT_BRANCH=HEAD)
May  7 17:05:08 pr-md-02 systemd[1]:  
libpod-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Consumed 1.008s CPU  
time.
May  7 17:05:08 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-b2fb74061f0c1e8a4fa96b8bf733317d61899748c44f6e0fce8e5c3cf2d425cf-merged.mount: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 podman[2564569]: 2024-05-07  
17:05:08.684978208 -0700 PDT m=+0.052557214 container remove  
bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=quizzical_antonelli, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 17:05:08 pr-md-02 systemd[1]:  
libpod-conmon-bafe02cbdc9fdd28ea041a0798aa53b45fbcc252e2e615ac26c5049a1df47575.scope: Deactivated  
successfully.
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225217: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev dae056fc-5916-4778-a1c3-877b296e165d does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 0dcc2973-fa02-4241-98c4-2d39053a690c does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 0027459d-bcc8-4411-92f1-52408996a6c0 does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev b9b9bbe1-65bc-4826-812e-501f98bd0988 does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev cbb7bfae-1607-408d-a5d6-e7517465e941 does not exist
May  7 17:05:08 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev a1e40167-575e-4690-b910-1d9135a7bcdf does not exist
May  7 17:05:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:09 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225218: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:05:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225219: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:12 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root]  
_maybe_adjust
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_metadata won't scale due to overlapping roots:  
{-15, -1}
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_data won't scale due to overlapping roots: {-15,  
-1, -2}
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_md_and_data won't scale due to overlapping  
roots: {-15, -1, -2}
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 1 contains an overlapping root -1... skipping scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 7 contains an overlapping root -15... skipping scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 8 contains an overlapping root -2... skipping scaling
May  7 17:05:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 9 contains an overlapping root -2... skipping scaling
May  7 17:05:14 pr-md-02 systemd[1]: Started Session c1518 of User root.
May  7 17:05:14 pr-md-02 systemd[1]: session-c1518.scope:  
Deactivated successfully.
May  7 17:05:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225220: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:16 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:16] "GET /metrics  
HTTP/1.1" 200 73531 "" "Prometheus/2.43.0"
May  7 17:05:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225221: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:17 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:18 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225222: 1313 pgs: 6 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:18 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:20 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225223: 1313 pgs: 6 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 766 B/s rd, 33 KiB/s wr, 2 op/s
May  7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5612 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:22 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225224: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s
May  7 17:05:22 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:22 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 e5613: 46 total, 46 up, 46 in
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:24 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225226: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:26 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:26] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:05:26 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225227: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:27 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:27 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:28 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225228: 1313 pgs: 7 active+clean+scrubbing+deep, 10  
active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:29 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:30 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:30 pr-md-02 ceph-mon[5348]: log_channel(audit) log  
[INF] : from='admin socket' entity='admin socket' cmd='smart'  
args=[json]: dispatch
May  7 17:05:30 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225229: 1313 pgs: 7 active+clean+scrubbing+deep, 10  
active+clean+scrubbing, 1296 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 920 B/s rd, 39 KiB/s wr, 2 op/s
May  7 17:05:31 pr-md-02 ceph-mon[5348]: log_channel(audit) log  
[INF] : from='admin socket' entity='admin socket' cmd=smart  
args=[json]: finished
May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:31 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:31 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:32 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:32 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225230: 1313 pgs: 4 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.8 KiB/s rd, 85 KiB/s wr, 5 op/s
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:32 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:32 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:33 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:33 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd='smart' args=[json]: dispatch
May  7 17:05:33 pr-md-02 ceph-mon[5348]: from='admin socket'  
entity='admin socket' cmd=smart args=[json]: finished
May  7 17:05:34 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225231: 1313 pgs: 4 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s
May  7 17:05:36 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:36] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:05:36 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225232: 1313 pgs: 4 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 71 KiB/s wr, 4 op/s
May  7 17:05:37 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:38 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:38 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225233: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s
May  7 17:05:39 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:40 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225234: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s
May  7 17:05:42 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:42 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225235: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 3.0 KiB/s rd, 130 KiB/s wr, 9 op/s
May  7 17:05:43 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:44 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225236: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s
May  7 17:05:46 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:46] "GET /metrics  
HTTP/1.1" 200 73534 "" "Prometheus/2.43.0"
May  7 17:05:46 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225237: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s
May  7 17:05:47 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:48 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:48 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225238: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail; 1.5 KiB/s rd, 60 KiB/s wr, 4 op/s
May  7 17:05:50 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225239: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1293 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:05:52 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:52 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225240: 1313 pgs: 5 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:05:53 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
scanning for idle connections..
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: [volumes INFO mgr_util]  
cleaning up connections: []
May  7 17:05:54 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225241: 1313 pgs: 5 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:05:56 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:05:56] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:05:56 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225242: 1313 pgs: 5 active+clean+scrubbing+deep, 16  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:05:57 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:05:58 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:05:58 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225243: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:00 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225244: 1313 pgs: 6 active+clean+scrubbing+deep, 15  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:02 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:06:02 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225245: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:03 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:06:04 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225246: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:06 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:06:06] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:06:06 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225247: 1313 pgs: 7 active+clean+scrubbing+deep, 14  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:07 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
MirrorSnapshotScheduleHandler: load_schedules
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
Optimize plan auto_2024-05-08_00:06:07
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] Mode  
upmap, max misplaced 0.050000
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root] do_upmap
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
pools ['cephfs_md_and_data', '.mgr', 'cephfs_data',  
'cephfs_metadata']
May  7 17:06:07 pr-md-02 ceph-mgr[5311]: [balancer INFO root]  
prepared 0/10 changes
May  7 17:06:08 pr-md-02 ceph-mgr[5311]: [rbd_support INFO root]  
TrashPurgeScheduleHandler: load_schedules
May  7 17:06:08 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:06:08 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225248: 1313 pgs: 4 active+clean+scrubbing+deep, 17  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:09 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev d098a1b7-eb6e-446f-98a4-e1427262dc3c does not exist
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 9289daf9-2f8a-4915-8058-687f1fc5eb11 does not exist
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev ca963837-183c-4bc7-990d-e76dc3a7a4b0 does not exist
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: [stats WARNING root]  
cmdtag not found in client metadata
May  7 17:06:10 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225249: 1313 pgs: 4 active+clean+scrubbing+deep, 17  
active+clean+scrubbing, 1292 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.181584373 -0700 PDT m=+0.039979954 container create  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, RELEASE=HEAD,  
ceph=True)
May  7 17:06:11 pr-md-02 systemd[1]: Started  
libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope.
May  7 17:06:11 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.211049202 -0700 PDT m=+0.069444730 container init  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS,  
org.label-schema.license=GPLv2)
May  7 17:06:11 pr-md-02 nice_joliot[2564893]: 167 167
May  7 17:06:11 pr-md-02 systemd[1]:  
libpod-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated  
successfully.
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.218033241 -0700 PDT m=+0.076428761 container start  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
org.label-schema.schema-version=1.0)
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.218168917 -0700 PDT m=+0.076564434 container attach  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, io.buildah.version=1.29.1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.218487253 -0700 PDT m=+0.076882764 container died  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
ceph=True)
May  7 17:06:11 pr-md-02 podman[2564878]: 2024-05-07  
17:06:11.165130923 -0700 PDT m=+0.023526448 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:11 pr-md-02 podman[2564898]: 2024-05-07  
17:06:11.276198352 -0700 PDT m=+0.051426463 container remove  
64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nice_joliot, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=HEAD,  
ceph=True)
May  7 17:06:11 pr-md-02 systemd[1]:  
libpod-conmon-64db7aaf822e2ea641ac851bb7f28b14a32618ac3311196f0121008ea7dece3c.scope: Deactivated  
successfully.
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.398685928 -0700 PDT m=+0.038305972 container create  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:11 pr-md-02 systemd[1]: Started  
libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope.
May  7 17:06:11 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.4284462 -0700 PDT m=+0.068066246 container init  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS,  
org.label-schema.schema-version=1.0)
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.436328165 -0700 PDT m=+0.075948196 container start  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.436459593 -0700 PDT m=+0.076079634 container attach  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:11 pr-md-02 podman[2564919]: 2024-05-07  
17:06:11.383297135 -0700 PDT m=+0.022917157 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:06:11 pr-md-02 ceph-mon[5348]: from='mgr.274299  
10.50.1.75:0/136806890' entity='mgr.pr-md-02.emffhz'  
cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]:  
dispatch
May  7 17:06:11 pr-md-02 ceph-mgr[5311]: log_channel(audit) log  
[DBG] : from='client.296615 -' entity='client.admin'  
cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch
May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> passed data  
devices: 2 physical, 0 LVM
May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> relative  
data size: 1.0
May  7 17:06:12 pr-md-02 sleepy_roentgen[2564934]: --> All data  
devices are unavailable
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 podman[2564919]: 2024-05-07  
17:06:12.059347046 -0700 PDT m=+0.698967087 container died  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, org.label-schema.build-date=20240222,  
GIT_REPO=https://github.com/ceph/ceph-container.git)
May  7 17:06:12 pr-md-02 podman[2564962]: 2024-05-07  
17:06:12.115085789 -0700 PDT m=+0.048338618 container remove  
4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=sleepy_roentgen, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_CLEAN=True)
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-conmon-4ec999d23a0ee13ce614ae8ad5b3fd2a3acf360affe394b2507986452079dae8.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-5f35e0c5e28b64ec23044b996e488378fe0b7b5bd084aec04d3a8efbfd932b2a-merged.mount: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.66977777 -0700 PDT m=+0.041271914 container create  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:06:12 pr-md-02 systemd[1]: Started  
libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope.
May  7 17:06:12 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.698793806 -0700 PDT m=+0.070287951 container init  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1,  
io.buildah.version=1.29.1)
May  7 17:06:12 pr-md-02 strange_wescoff[2565115]: 167 167
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.705992776 -0700 PDT m=+0.077486921 container start  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.706102938 -0700 PDT m=+0.077597093 container attach  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.29.1, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image,  
RELEASE=HEAD)
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.706421977 -0700 PDT m=+0.077916136 container died  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.build-date=20240222,  
org.label-schema.vendor=CentOS)
May  7 17:06:12 pr-md-02 podman[2565099]: 2024-05-07  
17:06:12.65486152 -0700 PDT m=+0.026355674 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:12 pr-md-02 podman[2565120]: 2024-05-07  
17:06:12.760218425 -0700 PDT m=+0.047515050 container remove  
24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=strange_wescoff, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base  
Image)
May  7 17:06:12 pr-md-02 systemd[1]:  
libpod-conmon-24dd6de0e98fc7afcbfb71e44efe052816b6c5d19d57680deab9e553722d92b1.scope: Deactivated  
successfully.
May  7 17:06:12 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225250: 1313 pgs: 5 active+clean+scrubbing+deep, 17  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.883756999 -0700 PDT m=+0.038722013 container create  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2,  
CEPH_POINT_RELEASE=-18.2.1)
May  7 17:06:12 pr-md-02 systemd[1]: Started  
libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope.
May  7 17:06:12 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.915479032 -0700 PDT m=+0.070443976 container init  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base  
Image)
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.922714281 -0700 PDT m=+0.077679227 container start  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd)
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.922813112 -0700 PDT m=+0.077778070 container attach  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>,  
org.label-schema.license=GPLv2)
May  7 17:06:12 pr-md-02 podman[2565141]: 2024-05-07  
17:06:12.867928578 -0700 PDT m=+0.022893523 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:13 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-dbb003174f50c19fbf330fa5c795cae4a0f3b395ec57674376442616981290d1-merged.mount: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    "41": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme0n1"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name":  
"osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size":  
"1920378863616",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags":  
"ceph.block_device=/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.block_uuid=MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO,ceph.cephx_lockbox_secret=AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=88070adf-ee15-406a-b8e9-98a61e294c8b,ceph.osd_id=41,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid":  
"MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"name": "osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"path":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "tags": {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_device":  
"/dev/ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc/osd-block-88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_uuid": "MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cephx_lockbox_secret":  
"AQBJFuJlxpZUJBAAHM4wPn5k5ExaR9mY0pS+YA==",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_name": "ceph",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.crush_device_class": "",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osd_fsid": "88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "41",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            },
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name":  
"ceph-eac1ebc0-c00f-4558-8c72-741fc8f81abc"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        }
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    "43": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "devices": [
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "/dev/nvme1n1"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            ],
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_name":  
"osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_size":  
"1920378863616",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_tags":  
"ceph.block_device=/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.block_uuid=zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT,ceph.cephx_lockbox_secret=AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==,ceph.cluster_fsid=58bde08a-d7ed-11ee-9098-506b4b4da440,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=1,ceph.osd_fsid=50a79ade-482b-4c7e-84ae-0dac068ed570,ceph.osd_id=43,ceph.osdspec_affinity=md_encrypted_nvme_disks,ceph.type=block,ceph.vdo=0",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "lv_uuid":  
"zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"name": "osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"path":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            "tags": {
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_device":  
"/dev/ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a/osd-block-50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.block_uuid": "zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cephx_lockbox_secret":  
"AQBYFuJlTtj/DBAAt2wy+qfx2XCHFuwJZKwp6A==",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.cluster_name": "ceph",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.crush_device_class": "",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.encrypted": "1",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osd_fsid": "50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.osd_id": "43",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:  
"ceph.osdspec_affinity": "md_encrypted_nvme_disks",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "ceph.vdo": "0"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:            },
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:             
"type": "block",
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: "vg_name":  
"ceph-0b5aeb1a-7dea-4260-9af9-f2bff16d642a"
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:        }
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]:    ]
May  7 17:06:13 pr-md-02 affectionate_saha[2565157]: }
May  7 17:06:13 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:06:13 pr-md-02 systemd[1]:  
libpod-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 podman[2565141]: 2024-05-07  
17:06:13.298397993 -0700 PDT m=+0.453362947 container died  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.29.1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux  
<gabrioux@xxxxxxxxxx>)
May  7 17:06:13 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-8550ac4bd2edf3f53d2004d34dad39cc51be76947b4935e496819469f7103f92-merged.mount: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 podman[2565169]: 2024-05-07  
17:06:13.360359344 -0700 PDT m=+0.052612579 container remove  
ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=affectionate_saha, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=-18.2.1, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=HEAD,  
ceph=True)
May  7 17:06:13 pr-md-02 systemd[1]:  
libpod-conmon-ff87d15729739f66dfd552fbf0c2add7e0b158e09eba8b15660c6b9946c40ac4.scope: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler INFO root]  
_maybe_adjust
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_metadata won't scale due to overlapping roots:  
{-15, -1}
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_data won't scale due to overlapping roots: {-15,  
-1, -2}
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool cephfs_md_and_data won't scale due to overlapping  
roots: {-15, -1, -2}
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] Please See:  
https://docs.ceph.com/en/latest/rados/operations/placement-groups/#automated-scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 1 contains an overlapping root -1... skipping scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 7 contains an overlapping root -15... skipping scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 8 contains an overlapping root -2... skipping scaling
May  7 17:06:13 pr-md-02 ceph-mgr[5311]: [pg_autoscaler WARNING  
root] pool 9 contains an overlapping root -2... skipping scaling
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.944018291 -0700 PDT m=+0.043243095 container create  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image,  
org.label-schema.vendor=CentOS)
May  7 17:06:13 pr-md-02 systemd[1]: Started  
libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope.
May  7 17:06:13 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.972707239 -0700 PDT m=+0.071931967 container init  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD,  
org.label-schema.vendor=CentOS)
May  7 17:06:13 pr-md-02 nifty_knuth[2565321]: 167 167
May  7 17:06:13 pr-md-02 systemd[1]:  
libpod-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated  
successfully.
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.979610545 -0700 PDT m=+0.078835287 container start  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, RELEASE=HEAD, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.979741103 -0700 PDT m=+0.078965840 container attach  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, io.buildah.version=1.29.1, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=https://github.com/ceph/ceph-container.git,  
RELEASE=HEAD)
May  7 17:06:13 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.979972502 -0700 PDT m=+0.079197239 container died  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.29.1, ceph=True, GIT_BRANCH=HEAD,  
org.label-schema.vendor=CentOS)
May  7 17:06:14 pr-md-02 podman[2565305]: 2024-05-07  
17:06:13.927504129 -0700 PDT m=+0.026728865 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:14 pr-md-02 podman[2565326]: 2024-05-07  
17:06:14.033457546 -0700 PDT m=+0.047525301 container remove  
b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=nifty_knuth, io.buildah.version=1.29.1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.1, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD,  
org.label-schema.build-date=20240222)
May  7 17:06:14 pr-md-02 systemd[1]:  
libpod-conmon-b1570bd0cfaf1fa55ec97cd999849d15eb4c1a6b693696e342bb5598d5f12cd6.scope: Deactivated  
successfully.
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.146017051 -0700 PDT m=+0.040230170 container create  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.29.1, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git,  
GIT_CLEAN=True)
May  7 17:06:14 pr-md-02 systemd[1]: Started  
libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope.
May  7 17:06:14 pr-md-02 systemd[1]: Started libcrun container.
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.175282354 -0700 PDT m=+0.069495491 container init  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2,  
org.label-schema.vendor=CentOS)
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.182850934 -0700 PDT m=+0.077064054 container start  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_CLEAN=True, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=HEAD, io.buildah.version=1.29.1, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2,  
ceph=True)
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.182936358 -0700 PDT m=+0.077149475 container attach  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=-18.2.1, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.29.1, ceph=True, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd,  
org.label-schema.license=GPLv2)
May  7 17:06:14 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-c68f47e6650f22ede040907bfad1eaa0c96a7a2a19ef648cb1312a9ceed9a65b-merged.mount: Deactivated  
successfully.
May  7 17:06:14 pr-md-02 podman[2565347]: 2024-05-07  
17:06:14.129225093 -0700 PDT m=+0.023438214 image pull  
5be31c24972a920012b90a9769e8313e2490c82aee752aefd49af9cf4c0f3fcf  
quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8
May  7 17:06:14 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225251: 1313 pgs: 5 active+clean+scrubbing+deep, 17  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: {
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:  
"50a79ade-482b-4c7e-84ae-0dac068ed570": {
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:  
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:         
"device": "/dev/mapper/zN3gTU-nGwQ-DfIA-dzWZ-e7QI-OcmL-SZcgqT",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "osd_id": 43,
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid":  
"50a79ade-482b-4c7e-84ae-0dac068ed570",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:         
"type": "bluestore"
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:    },
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:  
"88070adf-ee15-406a-b8e9-98a61e294c8b": {
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:  
"ceph_fsid": "58bde08a-d7ed-11ee-9098-506b4b4da440",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:         
"device": "/dev/mapper/MwSZ0h-81BS-XH33-wWY4-vk23-2TIJ-DW9XgO",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:        "osd_id": 41,
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: "osd_uuid":  
"88070adf-ee15-406a-b8e9-98a61e294c8b",
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:         
"type": "bluestore"
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]:    }
May  7 17:06:15 pr-md-02 flamboyant_poincare[2565363]: }
May  7 17:06:15 pr-md-02 systemd[1]:  
libpod-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated  
successfully.
May  7 17:06:15 pr-md-02 podman[2565347]: 2024-05-07  
17:06:15.160506671 -0700 PDT m=+1.054719792 container died  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, RELEASE=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.29.1, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.1, org.label-schema.name=CentOS Stream 8 Base Image,  
GIT_BRANCH=HEAD)
May  7 17:06:15 pr-md-02 systemd[1]:  
var-lib-containers-storage-overlay-b053c5cb16b427d8a3ee8d426e731633a8d8b6337b3c37ef50f558eadad5bd38-merged.mount: Deactivated  
successfully.
May  7 17:06:15 pr-md-02 podman[2565444]: 2024-05-07  
17:06:15.220598278 -0700 PDT m=+0.052865840 container remove  
0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b  
(image=quay.io/ceph/ceph@sha256:632158bcaab42d1034c1fa4a9a4d90a2d910f8d87f7b8d97a9e58af7a17db9e8, name=flamboyant_poincare, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=-18.2.1, maintainer=Guillaume Abrioux <gabrioux@xxxxxxxxxx>, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.29.1, org.label-schema.schema-version=1.0,  
org.label-schema.build-date=20240222)
May  7 17:06:15 pr-md-02 systemd[1]:  
libpod-conmon-0dd718175492b3950e94f88f74b359bfb5caee9547a97927df1c979b5c46289b.scope: Deactivated  
successfully.
May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 0ef17f53-4942-4371-9efc-a1a6b11d0994 does not exist
May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 8e261f50-3ae9-48c5-9371-cad82ccaaa9b does not exist
May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 54b062b8-01ba-45e7-8809-24bb3b472492 does not exist
May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 953528e8-a621-4cb5-9c5d-ed9bc75deada does not exist
May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev 1c93f903-df81-4e2a-be93-75dc1a37dfb1 does not exist
May  7 17:06:15 pr-md-02 ceph-mgr[5311]: [progress WARNING root]  
complete: ev bcd9783c-8512-42cf-b704-cce9f827345e does not exist
May  7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:06:15 pr-md-02 ceph-mon[5348]: from='mgr.274299 '  
entity='mgr.pr-md-02.emffhz'
May  7 17:06:15 pr-md-02 ceph-mgr[5311]: log_channel(audit) log  
[DBG] : from='client.296625 -' entity='client.admin'  
cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch
May  7 17:06:16 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-mgr-pr-md-02-emffhz[5306]:  
::ffff:10.50.1.74 - - [08/May/2024:00:06:16] "GET /metrics  
HTTP/1.1" 200 73532 "" "Prometheus/2.43.0"
May  7 17:06:16 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225252: 1313 pgs: 5 active+clean+scrubbing+deep, 17  
active+clean+scrubbing, 1291 active+clean; 558 TiB data, 842 TiB  
used, 287 TiB / 1.1 PiB avail
May  7 17:06:17 pr-md-02 ceph-mon[5348]: mon.pr-md-02@1(peon).osd  
e5613 _set_new_cache_sizes cache_size:1020054731 inc_alloc:  
339738624 full_alloc: 356515840 kv_alloc: 318767104
May  7 17:06:18 pr-md-02  
ceph-58bde08a-d7ed-11ee-9098-506b4b4da440-ceph-exporter-pr-md-02[5226]:  
system:0
May  7 17:06:18 pr-md-02 ceph-mgr[5311]: log_channel(cluster) log  
[DBG] : pgmap v225253: 1313 pgs: 6 active+clean+scrubbing+deep, 14  
active+clean+scrubbing
[Text gekürzt]

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx