Good morning K,
the "freshly done" host, where it happened last got:
* 21x 8TB TOSHIBA MG06ACA800E (Spinning)
* No block.db devices (just removed the 2 cache SSDs by syncing the disks
out, wiping them and adding them back without block.db)
* 1x Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz
* 256GB ECC RAM
* 2x 10GBit Network (802.3ad encap3+4 lacp-fast bonding)
# free -g
total used free shared buff/cache
available
Mem: 251 87 2 0 161
162
Swap: 15 0 15
We had this problem with one of the 21 OSDs, but I expect it to happen
random some time in the future. Cluster got 212 OSDs and 2-3 of them get at
least marked down once per day. Sometime they get marked down >3 times, so
systemd hast to restart the OSD process.
Cheers
Boris
Am Di., 22. März 2022 um 07:48 Uhr schrieb Konstantin Shalygin <
k0ste@xxxxxxxx>:
> Hi,
>
> What is actual hardware (CPU, spinners, NVMe, network)?
> This is HDD with block.db on NVMe?
> How much PG per osd?
> How much obj per PG?
>
>
> k
> Sent from my iPhone
>
> > On 20 Mar 2022, at 19:59, Boris Behrens <bb@xxxxxxxxx> wrote:
> >
> > So,
> > I have tried to remove the OSDs, wipe the disks and sync them back in
> > without block.db SSD. (Still in progress, 212 spinning disks take time to
> > out and in again)
> > And I just experienced them same behavior on one OSD on a host where all
> > disks got synced in new. This disk was marked as in yesterday and is
> still
> > backfilling.
> >
> > Right before the OSD get marked as down by other OSDs I observe a ton of
> > these log entries:
> > 2022-03-20T11:54:40.759+0000 7ff9eef5d700 1 heartbeat_map is_healthy
> > 'OSD::osd_op_tp thread 0x7ff9d3c8a700' had timed out after 15
> > ...
> > 2022-03-20T11:55:02.370+0000 7ff9ee75c700 1 heartbeat_map is_healthy
> > 'OSD::osd_op_tp thread 0x7ff9d3c8a700' had timed out after 15
> > 2022-03-20T11:55:03.290+0000 7ff9d3c8a700 1 heartbeat_map reset_timeout
> > 'OSD::osd_op_tp thread 0x7ff9d3c8a700' had timed out after 15
> > ..
> > 2022-03-20T11:55:03.390+0000 7ff9df4a1700 0 log_channel(cluster) log
> [WRN]
> > : Monitor daemon marked osd.48 down, but it is still running
> > 2022-03-20T11:55:03.390+0000 7ff9df4a1700 0 log_channel(cluster) log
> [DBG]
> > : map e514383 wrongly marked me down at e514383
> > 2022-03-20T11:55:03.390+0000 7ff9df4a1700 -1 osd.48 514383
> > _committed_osd_maps marked down 6 > osd_max_markdown_count 5 in last
> > 600.000000 seconds, shutting down
> >
> > There are 21 OSDs without cache SSD in the host.
> > All disks are attached to a single Broadcom / LSI SAS3008 SAS controller.
> > 256GB ECC-RAM / 40 CPU cores.
> >
> > What else can I do to find the problem?
> >
> >> Am Di., 8. März 2022 um 12:25 Uhr schrieb Francois Legrand <
> >> fleg@xxxxxxxxxxxxxx>:
> >>
> >> Hi,
> >>
> >> The last 2 osd I recreated were on december 30 and february 8.
> >>
> >> I totally agree that ssd cache are a terrible spof. I think that's an
> >> option if you use 1 ssd/nvme for 1 or 2 osd, but the cost is then very
> >> high. Using 1 ssd for 10 osd increase the risk for almost no gain
> because
> >> the ssd is 10 times faster but has 10 times more access !
> >> Indeed, we did some benches with nvme for the wal db (1 nvme for ~10
> >> osds), and the gain was not tremendous, so we decided not use them !
> >> F.
> >>
> >>
> >> Le 08/03/2022 à 11:57, Boris Behrens a écrit :
> >>
> >> Hi Francois,
> >>
> >> thanks for the reminder. We offline compacted all of the OSDs when we
> >> reinstalled the hosts with the new OS.
> >> But actually reinstalling them was never on my list.
> >>
> >> I could try that and in the same go I can remove all the cache SSDs
> (when
> >> one SSD share the cache for 10 OSDs this is a horrible SPOF) and reuse
> the
> >> SSDs as OSDs for the smaller pools in a RGW (like log and meta).
> >>
> >> How long ago did you recreate the earliest OSD?
> >>
> >> Cheers
> >> Boris
> >>
> >> Am Di., 8. März 2022 um 10:03 Uhr schrieb Francois Legrand <
> >> fleg@xxxxxxxxxxxxxx>:
> >>
> >>> Hi,
> >>> We also had this kind of problems after upgrading to octopus. Maybe you
> >>> can play with the hearthbeat grace time (
> >>>
> https://docs.ceph.com/en/latest/rados/configuration/mon-osd-interaction/
> >>> ) to tell osds to wait a little more before declaring another osd down
> !
> >>> We also try to fix the problem by manually compact the down osd
> >>> (something like : systemctl stop ceph-osd@74; sleep 10;
> >>> ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-74 compact;
> >>> systemctl start ceph-osd@74).
> >>> This worked a few times, but some osd went down again, thus we simply
> >>> wait for the datas to be reconstructed elswhere and then reinstall the
> >>> dead osd :
> >>> ceph osd destroy 74 --yes-i-really-mean-it
> >>> ceph-volume lvm zap /dev/sde --destroy
> >>> ceph-volume lvm create --osd-id 74 --data /dev/sde
> >>>
> >>> This seems to fix the issue for us (up to now).
> >>>
> >>> F.
> >>>
> >>> Le 08/03/2022 à 09:35, Boris Behrens a écrit :
> >>>> Yes, this is something we know and we disabled it, because we ran into
> >>> the
> >>>> problem that PGs went unavailable when two or more OSDs went offline.
> >>>>
> >>>> I am searching for the reason WHY this happens.
> >>>> Currently we have set the service file to restart=always and removed
> the
> >>>> StartLimitBurst from the service file.
> >>>>
> >>>> We just don't understand why the OSDs don't answer the heathbeat. The
> >>> OSDs
> >>>> that are flapping are random in terms of Host, Disksize, having SSD
> >>>> block.db or not.
> >>>> Network connectivity issues is something that I would rule out,
> because
> >>> the
> >>>> cluster went from "nothing ever happens except IOPS" to "random OSDs
> are
> >>>> marked DOWN until they kill themself" with the update from nautilus to
> >>>> octopus.
> >>>>
> >>>> I am out of ideas and hoped this was a bug in 15.2.15, but after the
> >>> update
> >>>> things got worse (happen more often).
> >>>> We tried to:
> >>>> * disable swap
> >>>> * more swap
> >>>> * disable bluefs_buffered_io
> >>>> * disable write cache for all disks
> >>>> * disable scrubbing
> >>>> * reinstall with new OS (from centos7 to ubuntu 20.04)
> >>>> * disable cluster_network (so there is only one way to communicate)
> >>>> * increase txqueuelen on the network interfaces
> >>>> * everything together
> >>>>
> >>>>
> >>>> What we try next: add more SATA controllers, so there are not 24 disks
> >>>> attached to a single controller, but I doubt this will help.
> >>>>
> >>>> Cheers
> >>>> Boris
> >>>>
> >>>>
> >>>>
> >>>> Am Di., 8. März 2022 um 09:10 Uhr schrieb Dan van der Ster <
> >>>> dvanders@xxxxxxxxx>:
> >>>>
> >>>>> Here's the reason they exit:
> >>>>>
> >>>>> 7f1605dc9700 -1 osd.97 486896 _committed_osd_maps marked down 6 >
> >>>>> osd_max_markdown_count 5 in last 600.000000 seconds, shutting down
> >>>>>
> >>>>> If an osd flaps (marked down, then up) 6 times in 10 minutes, it
> >>>>> exits. (This is a safety measure).
> >>>>>
> >>>>> It's normally caused by a network issue -- other OSDs are telling the
> >>>>> mon that he is down, but then the OSD himself tells the mon that he's
> >>>>> up!
> >>>>>
> >>>>> Cheers, Dan
> >>>>>
> >>>>> On Mon, Mar 7, 2022 at 10:36 PM Boris Behrens <bb@xxxxxxxxx> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> we've had the problem with OSDs marked as offline since we updated
> to
> >>>>>> octopus and hope the problem would be fixed with the latest patch.
> We
> >>>>> have
> >>>>>> this kind of problem only with octopus and there only with the big
> s3
> >>>>>> cluster.
> >>>>>> * Hosts are all Ubuntu 20,04 and we've set the txqueuelen to 10k
> >>>>>> * Network interfaces are 20gbit (2x10 in a 802.3ad encap3+4 bond)
> >>>>>> * We only use the frontend network.
> >>>>>> * All disks are spinning, some have block.db devices.
> >>>>>> * All disks are bluestore
> >>>>>> * configs are mostly defaults
> >>>>>> * we've set the OSDs to restart=always without a limit, because we
> had
> >>>>> the
> >>>>>> problem with unavailable PGs when two OSDs are marked as offline and
> >>> the
> >>>>>> share PGs.
> >>>>>>
> >>>>>> But since we installed the latest patch we are experiencing more OSD
> >>>>> downs
> >>>>>> and even crashes.
> >>>>>> I tried to remove as much duplicated lines as possible.
> >>>>>>
> >>>>>> Is the numa error a problem?
> >>>>>> Why do OSD daemons not respond to hearthbeats? I mean even when the
> >>> disk
> >>>>> is
> >>>>>> totally loaded with IO, the system itself should answer heathbeats,
> or
> >>>>> am I
> >>>>>> missing something?
> >>>>>>
> >>>>>> I really hope some of you could send me on the correct way to solve
> >>> this
> >>>>>> nasty problem.
> >>>>>>
> >>>>>> This is how the latest crash looks like
> >>>>>> Mar 07 17:44:15 s3db18 ceph-osd[4530]: 2022-03-07T17:44:15.099+0000
> >>>>>> 7f5f05d2a700 -1 osd.161 489755 set_numa_affinity unable to identify
> >>>>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> ...
> >>>>>> Mar 07 17:49:07 s3db18 ceph-osd[4530]: 2022-03-07T17:49:07.678+0000
> >>>>>> 7f5f05d2a700 -1 osd.161 489774 set_numa_affinity unable to identify
> >>>>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: *** Caught signal (Aborted)
> **
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: in thread 7f5ef1501700
> >>>>>> thread_name:tp_osd_tp
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: ceph version 15.2.16
> >>>>>> (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 1: (()+0x143c0)
> >>> [0x7f5f0d4623c0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 2: (pthread_kill()+0x38)
> >>>>>> [0x7f5f0d45ef08]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 3:
> >>>>>> (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char
> >>> const*,
> >>>>>> unsigned long)+0x471) [0x55a699a01201]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 4:
> >>>>>> (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*,
> unsigned
> >>>>>> long, unsigned long)+0x8e) [0x55a699a0199e]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 5:
> >>>>>> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3f0)
> >>>>>> [0x55a699a224b0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 6:
> >>>>>> (ShardedThreadPool::WorkThreadSharded::entry()+0x14)
> [0x55a699a252c4]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 7: (()+0x8609)
> >>> [0x7f5f0d456609]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 8: (clone()+0x43)
> >>>>> [0x7f5f0cfc0163]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 2022-03-07T17:53:07.387+0000
> >>>>>> 7f5ef1501700 -1 *** Caught signal (Aborted) **
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: in thread 7f5ef1501700
> >>>>>> thread_name:tp_osd_tp
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: ceph version 15.2.16
> >>>>>> (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 1: (()+0x143c0)
> >>> [0x7f5f0d4623c0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 2: (pthread_kill()+0x38)
> >>>>>> [0x7f5f0d45ef08]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 3:
> >>>>>> (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char
> >>> const*,
> >>>>>> unsigned long)+0x471) [0x55a699a01201]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 4:
> >>>>>> (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*,
> unsigned
> >>>>>> long, unsigned long)+0x8e) [0x55a699a0199e]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 5:
> >>>>>> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3f0)
> >>>>>> [0x55a699a224b0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 6:
> >>>>>> (ShardedThreadPool::WorkThreadSharded::entry()+0x14)
> [0x55a699a252c4]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 7: (()+0x8609)
> >>> [0x7f5f0d456609]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 8: (clone()+0x43)
> >>>>> [0x7f5f0cfc0163]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: NOTE: a copy of the
> >>> executable,
> >>>>> or
> >>>>>> `objdump -rdS <executable>` is needed to interpret this.
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: -5246>
> >>>>> 2022-03-07T17:49:07.678+0000
> >>>>>> 7f5f05d2a700 -1 osd.161 489774 set_numa_affinity unable to identify
> >>>>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 0>
> >>>>> 2022-03-07T17:53:07.387+0000
> >>>>>> 7f5ef1501700 -1 *** Caught signal (Aborted) **
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: in thread 7f5ef1501700
> >>>>>> thread_name:tp_osd_tp
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: ceph version 15.2.16
> >>>>>> (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 1: (()+0x143c0)
> >>> [0x7f5f0d4623c0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 2: (pthread_kill()+0x38)
> >>>>>> [0x7f5f0d45ef08]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 3:
> >>>>>> (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char
> >>> const*,
> >>>>>> unsigned long)+0x471) [0x55a699a01201]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 4:
> >>>>>> (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*,
> unsigned
> >>>>>> long, unsigned long)+0x8e) [0x55a699a0199e]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 5:
> >>>>>> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3f0)
> >>>>>> [0x55a699a224b0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 6:
> >>>>>> (ShardedThreadPool::WorkThreadSharded::entry()+0x14)
> [0x55a699a252c4]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 7: (()+0x8609)
> >>> [0x7f5f0d456609]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 8: (clone()+0x43)
> >>>>> [0x7f5f0cfc0163]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: NOTE: a copy of the
> >>> executable,
> >>>>> or
> >>>>>> `objdump -rdS <executable>` is needed to interpret this.
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: -5246>
> >>>>> 2022-03-07T17:49:07.678+0000
> >>>>>> 7f5f05d2a700 -1 osd.161 489774 set_numa_affinity unable to identify
> >>>>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 0>
> >>>>> 2022-03-07T17:53:07.387+0000
> >>>>>> 7f5ef1501700 -1 *** Caught signal (Aborted) **
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: in thread 7f5ef1501700
> >>>>>> thread_name:tp_osd_tp
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: ceph version 15.2.16
> >>>>>> (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 1: (()+0x143c0)
> >>> [0x7f5f0d4623c0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 2: (pthread_kill()+0x38)
> >>>>>> [0x7f5f0d45ef08]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 3:
> >>>>>> (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char
> >>> const*,
> >>>>>> unsigned long)+0x471) [0x55a699a01201]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 4:
> >>>>>> (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*,
> unsigned
> >>>>>> long, unsigned long)+0x8e) [0x55a699a0199e]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 5:
> >>>>>> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3f0)
> >>>>>> [0x55a699a224b0]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 6:
> >>>>>> (ShardedThreadPool::WorkThreadSharded::entry()+0x14)
> [0x55a699a252c4]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 7: (()+0x8609)
> >>> [0x7f5f0d456609]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: 8: (clone()+0x43)
> >>>>> [0x7f5f0cfc0163]
> >>>>>> Mar 07 17:53:07 s3db18 ceph-osd[4530]: NOTE: a copy of the
> >>> executable,
> >>>>> or
> >>>>>> `objdump -rdS <executable>` is needed to interpret this.
> >>>>>> Mar 07 17:53:09 s3db18 systemd[1]: ceph-osd@161.service: Main
> process
> >>>>>> exited, code=killed, status=6/ABRT
> >>>>>> Mar 07 17:53:09 s3db18 systemd[1]: ceph-osd@161.service: Failed
> with
> >>>>> result
> >>>>>> 'signal'.
> >>>>>> Mar 07 17:53:19 s3db18 systemd[1]: ceph-osd@161.service: Scheduled
> >>>>> restart
> >>>>>> job, restart counter is at 1.
> >>>>>> Mar 07 17:53:19 s3db18 systemd[1]: Stopped Ceph object storage
> daemon
> >>>>>> osd.161.
> >>>>>> Mar 07 17:53:19 s3db18 systemd[1]: Starting Ceph object storage
> daemon
> >>>>>> osd.161...
> >>>>>> Mar 07 17:53:19 s3db18 systemd[1]: Started Ceph object storage
> daemon
> >>>>>> osd.161.
> >>>>>> Mar 07 17:53:20 s3db18 ceph-osd[4009440]:
> 2022-03-07T17:53:20.498+0000
> >>>>>> 7f9617781d80 -1 Falling back to public interface
> >>>>>> Mar 07 17:53:33 s3db18 ceph-osd[4009440]:
> 2022-03-07T17:53:33.906+0000
> >>>>>> 7f9617781d80 -1 osd.161 489778 log_to_monitors {default=true}
> >>>>>> Mar 07 17:53:34 s3db18 ceph-osd[4009440]:
> 2022-03-07T17:53:34.206+0000
> >>>>>> 7f96106f2700 -1 osd.161 489778 set_numa_affinity unable to identify
> >>>>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> ...
> >>>>>> Mar 07 18:58:12 s3db18 ceph-osd[4009440]:
> 2022-03-07T18:58:12.717+0000
> >>>>>> 7f96106f2700 -1 osd.161 489880 set_numa_affinity unable to identify
> >>>>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>>
> >>>>>> And this is how an it looks like when OSDs get marked as out:
> >>>>>> Mar 03 19:29:04 s3db13 ceph-osd[5792]: 2022-03-03T19:29:04.857+0000
> >>>>>> 7f16115e0700 -1 osd.97 485814 heartbeat_check: no reply from
> >>>>>> [XX:22::65]:6886 osd.124 since back 2022-03-03T19:28:41.250692+0000
> >>> front
> >>>>>> 2022-03-03T19:28:41.250649+0000 (oldest deadline
> >>>>>> 2022-03-03T19:29:04.150352+0000)
> >>>>>> ...130 time...
> >>>>>> Mar 03 21:55:37 s3db13 ceph-osd[5792]: 2022-03-03T21:55:37.844+0000
> >>>>>> 7f16115e0700 -1 osd.97 486383 heartbeat_check: no reply from
> >>>>>> [XX:22::65]:6941 osd.124 since back 2022-03-03T21:55:12.514627+0000
> >>> front
> >>>>>> 2022-03-03T21:55:12.514649+0000 (oldest deadline
> >>>>>> 2022-03-03T21:55:36.613469+0000)
> >>>>>> Mar 04 00:00:05 s3db13 ceph-osd[5792]: 2022-03-04T00:00:05.035+0000
> >>>>>> 7f1613080700 -1 received signal: Hangup from killall -q -1 ceph-mon
> >>>>>> ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror (PID:
> >>> 1385079)
> >>>>>> UID: 0
> >>>>>> Mar 04 00:00:05 s3db13 ceph-osd[5792]: 2022-03-04T00:00:05.047+0000
> >>>>>> 7f1613080700 -1 received signal: Hangup from (PID: 1385080) UID: 0
> >>>>>> Mar 04 00:06:00 s3db13 sudo[1389262]: ceph : TTY=unknown ;
> PWD=/ ;
> >>>>>> USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sde
> >>>>>> Mar 04 00:06:00 s3db13 sudo[1389262]: pam_unix(sudo:session):
> session
> >>>>>> opened for user root by (uid=0)
> >>>>>> Mar 04 00:06:00 s3db13 sudo[1389262]: pam_unix(sudo:session):
> session
> >>>>>> closed for user root
> >>>>>> Mar 04 00:06:01 s3db13 sudo[1389287]: ceph : TTY=unknown ;
> PWD=/ ;
> >>>>>> USER=root ; COMMAND=/usr/sbin/nvme ata smart-log-add --json /dev/sde
> >>>>>> Mar 04 00:06:01 s3db13 sudo[1389287]: pam_unix(sudo:session):
> session
> >>>>>> opened for user root by (uid=0)
> >>>>>> Mar 04 00:06:01 s3db13 sudo[1389287]: pam_unix(sudo:session):
> session
> >>>>>> closed for user root
> >>>>>> Mar 05 00:00:10 s3db13 ceph-osd[5792]: 2022-03-05T00:00:10.213+0000
> >>>>>> 7f1613080700 -1 received signal: Hangup from killall -q -1 ceph-mon
> >>>>>> ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw rbd-mirror (PID:
> >>> 2406262)
> >>>>>> UID: 0
> >>>>>> Mar 05 00:00:10 s3db13 ceph-osd[5792]: 2022-03-05T00:00:10.237+0000
> >>>>>> 7f1613080700 -1 received signal: Hangup from (PID: 2406263) UID: 0
> >>>>>> Mar 05 00:08:03 s3db13 sudo[2411721]: ceph : TTY=unknown ;
> PWD=/ ;
> >>>>>> USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sde
> >>>>>> Mar 05 00:08:03 s3db13 sudo[2411721]: pam_unix(sudo:session):
> session
> >>>>>> opened for user root by (uid=0)
> >>>>>> Mar 05 00:08:04 s3db13 sudo[2411721]: pam_unix(sudo:session):
> session
> >>>>>> closed for user root
> >>>>>> Mar 05 00:08:04 s3db13 sudo[2411725]: ceph : TTY=unknown ;
> PWD=/ ;
> >>>>>> USER=root ; COMMAND=/usr/sbin/nvme ata smart-log-add --json /dev/sde
> >>>>>> Mar 05 00:08:04 s3db13 sudo[2411725]: pam_unix(sudo:session):
> session
> >>>>>> opened for user root by (uid=0)
> >>>>>> Mar 05 00:08:04 s3db13 sudo[2411725]: pam_unix(sudo:session):
> session
> >>>>>> closed for user root
> >>>>>> Mar 05 19:19:49 s3db13 ceph-osd[5792]: 2022-03-05T19:19:49.189+0000
> >>>>>> 7f160fddd700 -1 osd.97 486852 set_numa_affinity unable to identify
> >>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 05 19:21:18 s3db13 ceph-osd[5792]: 2022-03-05T19:21:18.377+0000
> >>>>>> 7f160fddd700 -1 osd.97 486858 set_numa_affinity unable to identify
> >>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 05 19:21:45 s3db13 ceph-osd[5792]: 2022-03-05T19:21:45.304+0000
> >>>>>> 7f16115e0700 -1 osd.97 486863 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 since back 2022-03-05T19:21:21.762744+0000
> >>> front
> >>>>>> 2022-03-05T19:21:21.762723+0000 (oldest deadline
> >>>>>> 2022-03-05T19:21:45.261347+0000)
> >>>>>> Mar 05 19:21:46 s3db13 ceph-osd[5792]: 2022-03-05T19:21:46.260+0000
> >>>>>> 7f16115e0700 -1 osd.97 486863 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 since back 2022-03-05T19:21:21.762744+0000
> >>> front
> >>>>>> 2022-03-05T19:21:21.762723+0000 (oldest deadline
> >>>>>> 2022-03-05T19:21:45.261347+0000)
> >>>>>> Mar 05 19:21:47 s3db13 ceph-osd[5792]: 2022-03-05T19:21:47.252+0000
> >>>>>> 7f16115e0700 -1 osd.97 486863 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 since back 2022-03-05T19:21:21.762744+0000
> >>> front
> >>>>>> 2022-03-05T19:21:21.762723+0000 (oldest deadline
> >>>>>> 2022-03-05T19:21:45.261347+0000)
> >>>>>> Mar 05 19:22:59 s3db13 ceph-osd[5792]: 2022-03-05T19:22:59.636+0000
> >>>>>> 7f160fddd700 -1 osd.97 486869 set_numa_affinity unable to identify
> >>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 05 19:23:33 s3db13 ceph-osd[5792]: 2022-03-05T19:23:33.439+0000
> >>>>>> 7f16115e0700 -1 osd.97 486872 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:23:34 s3db13 ceph-osd[5792]: 2022-03-05T19:23:34.458+0000
> >>>>>> 7f16115e0700 -1 osd.97 486872 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:23:35 s3db13 ceph-osd[5792]: 2022-03-05T19:23:35.434+0000
> >>>>>> 7f16115e0700 -1 osd.97 486872 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 since back 2022-03-05T19:23:09.928097+0000
> >>> front
> >>>>>> 2022-03-05T19:23:09.928150+0000 (oldest deadline
> >>>>>> 2022-03-05T19:23:35.227545+0000)
> >>>>>> ...
> >>>>>> Mar 05 19:23:48 s3db13 ceph-osd[5792]: 2022-03-05T19:23:48.386+0000
> >>>>>> 7f16115e0700 -1 osd.97 486872 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:23:49 s3db13 ceph-osd[5792]: 2022-03-05T19:23:49.362+0000
> >>>>>> 7f16115e0700 -1 osd.97 486872 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 since back 2022-03-05T19:23:09.928097+0000
> >>> front
> >>>>>> 2022-03-05T19:23:09.928150+0000 (oldest deadline
> >>>>>> 2022-03-05T19:23:35.227545+0000)
> >>>>>> Mar 05 19:23:49 s3db13 ceph-osd[5792]: 2022-03-05T19:23:49.362+0000
> >>>>>> 7f16115e0700 -1 osd.97 486872 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:23:50 s3db13 ceph-osd[5792]: 2022-03-05T19:23:50.358+0000
> >>>>>> 7f16115e0700 -1 osd.97 486873 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:23:51 s3db13 ceph-osd[5792]: 2022-03-05T19:23:51.330+0000
> >>>>>> 7f16115e0700 -1 osd.97 486874 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d
> >>> 4:b0b12ee9:::gc.22:head
> >>>>>> [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=9
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:23:52 s3db13 ceph-osd[5792]: 2022-03-05T19:23:52.326+0000
> >>>>>> 7f16115e0700 -1 osd.97 486874 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d
> >>> 4:b0b12ee9:::gc.22:head
> >>>>>> [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=9
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:23:53 s3db13 ceph-osd[5792]: 2022-03-05T19:23:53.338+0000
> >>>>>> 7f16115e0700 -1 osd.97 486874 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d
> >>> 4:b0b12ee9:::gc.22:head
> >>>>>> [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=9
> >>>>>> ondisk+retry+read+known_if_redirected e486872)
> >>>>>> Mar 05 19:25:02 s3db13 ceph-osd[5792]: 2022-03-05T19:25:02.342+0000
> >>>>>> 7f160fddd700 -1 osd.97 486878 set_numa_affinity unable to identify
> >>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 05 19:25:33 s3db13 ceph-osd[5792]: 2022-03-05T19:25:33.569+0000
> >>>>>> 7f16115e0700 -1 osd.97 486880 get_health_metrics reporting 2 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224857.0:4271104 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486879)
> >>>>>> ...
> >>>>>> Mar 05 19:25:44 s3db13 ceph-osd[5792]: 2022-03-05T19:25:44.476+0000
> >>>>>> 7f16115e0700 -1 osd.97 486880 get_health_metrics reporting 3 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224857.0:4271104 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486879)
> >>>>>> Mar 05 19:25:45 s3db13 ceph-osd[5792]: 2022-03-05T19:25:45.456+0000
> >>>>>> 7f16115e0700 -1 osd.97 486880 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 ever on either front or back, first ping
> sent
> >>>>>> 2022-03-05T19:25:25.281582+0000 (oldest deadline
> >>>>>> 2022-03-05T19:25:45.281582+0000)
> >>>>>> Mar 05 19:25:45 s3db13 ceph-osd[5792]: 2022-03-05T19:25:45.456+0000
> >>>>>> 7f16115e0700 -1 osd.97 486880 get_health_metrics reporting 3 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224857.0:4271104 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486879)
> >>>>>> ...
> >>>>>> Mar 05 19:26:08 s3db13 ceph-osd[5792]: 2022-03-05T19:26:08.363+0000
> >>>>>> 7f16115e0700 -1 osd.97 486880 get_health_metrics reporting 3 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224857.0:4271104 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486879)
> >>>>>> Mar 05 19:26:09 s3db13 ceph-osd[5792]: 2022-03-05T19:26:09.371+0000
> >>>>>> 7f16115e0700 -1 osd.97 486880 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 ever on either front or back, first ping
> sent
> >>>>>> 2022-03-05T19:25:25.281582+0000 (oldest deadline
> >>>>>> 2022-03-05T19:25:45.281582+0000)
> >>>>>> Mar 05 19:26:09 s3db13 ceph-osd[5792]: 2022-03-05T19:26:09.375+0000
> >>>>>> 7f16115e0700 -1 osd.97 486880 get_health_metrics reporting 3 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224857.0:4271104 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486879)
> >>>>>> Mar 05 19:26:10 s3db13 ceph-osd[5792]: 2022-03-05T19:26:10.383+0000
> >>>>>> 7f16115e0700 -1 osd.97 486881 get_health_metrics reporting 3 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224857.0:4271104 4.d 4.97748d0d
> >>> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486879)
> >>>>>> Mar 05 19:26:11 s3db13 ceph-osd[5792]: 2022-03-05T19:26:11.407+0000
> >>>>>> 7f16115e0700 -1 osd.97 486882 get_health_metrics reporting 1 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d
> >>> 4:b0b12ee9:::gc.22:head
> >>>>>> [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=11
> >>>>>> ondisk+retry+read+known_if_redirected e486879)
> >>>>>> Mar 05 19:26:12 s3db13 ceph-osd[5792]: 2022-03-05T19:26:12.399+0000
> >>>>>> 7f16115e0700 -1 osd.97 486882 get_health_metrics reporting 1 slow
> ops,
> >>>>>> oldest is osd_op(client.2304224848.0:3139913 4.d
> >>> 4:b0b12ee9:::gc.22:head
> >>>>>> [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=11
> >>>>>> ondisk+retry+read+known_if_redirected e486879)
> >>>>>> Mar 05 19:27:24 s3db13 ceph-osd[5792]: 2022-03-05T19:27:24.975+0000
> >>>>>> 7f160fddd700 -1 osd.97 486887 set_numa_affinity unable to identify
> >>> public
> >>>>>> interface '' numa node: (2) No such file or directory
> >>>>>> Mar 05 19:27:58 s3db13 ceph-osd[5792]: 2022-03-05T19:27:58.114+0000
> >>>>>> 7f16115e0700 -1 osd.97 486890 get_health_metrics reporting 4 slow
> ops,
> >>>>>> oldest is osd_op(client.2304235452.0:811825 4.d 4.97748d0d
> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486889)
> >>>>>> ...
> >>>>>> Mar 05 19:28:08 s3db13 ceph-osd[5792]: 2022-03-05T19:28:08.137+0000
> >>>>>> 7f16115e0700 -1 osd.97 486890 get_health_metrics reporting 4 slow
> ops,
> >>>>>> oldest is osd_op(client.2304235452.0:811825 4.d 4.97748d0d
> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486889)
> >>>>>> Mar 05 19:28:09 s3db13 ceph-osd[5792]: 2022-03-05T19:28:09.125+0000
> >>>>>> 7f16115e0700 -1 osd.97 486890 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 ever on either front or back, first ping
> sent
> >>>>>> 2022-03-05T19:27:48.548094+0000 (oldest deadline
> >>>>>> 2022-03-05T19:28:08.548094+0000)
> >>>>>> Mar 05 19:28:09 s3db13 ceph-osd[5792]: 2022-03-05T19:28:09.125+0000
> >>>>>> 7f16115e0700 -1 osd.97 486890 get_health_metrics reporting 4 slow
> ops,
> >>>>>> oldest is osd_op(client.2304235452.0:811825 4.d 4.97748d0d
> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486889)
> >>>>>> ...
> >>>>>> Mar 05 19:28:29 s3db13 ceph-osd[5792]: 2022-03-05T19:28:29.060+0000
> >>>>>> 7f16115e0700 -1 osd.97 486890 get_health_metrics reporting 4 slow
> ops,
> >>>>>> oldest is osd_op(client.2304235452.0:811825 4.d 4.97748d0d
> (undecoded)
> >>>>>> ondisk+retry+write+known_if_redirected e486889)
> >>>>>> Mar 05 19:28:30 s3db13 ceph-osd[5792]: 2022-03-05T19:28:30.040+0000
> >>>>>> 7f16115e0700 -1 osd.97 486890 heartbeat_check: no reply from
> >>>>>> [XX:22::60]:6834 osd.171 ever on either front or back, first ping
> sent
> >>>>>> 2022-03-05T19:27:48.548094+0000 (oldest deadline
> >>>>>> 2022-03-05T19:28:08.548094+0000)
> >>>>>> Mar 05 19:29:43 s3db13 ceph-osd[5792]: 2022-03-05T19:29:43.696+0000
> >>>>>> 7f1605dc9700 -1 osd.97 486896 _committed_osd_maps marked down 6 >
> >>>>>> osd_max_markdown_count 5 in last 600.000000 seconds, shutting down
> >>>>>> Mar 05 19:29:43 s3db13 ceph-osd[5792]: 2022-03-05T19:29:43.700+0000
> >>>>>> 7f1613080700 -1 received signal: Interrupt from Kernel ( Could be
> >>>>>> generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
> >>>>>> Mar 05 19:29:43 s3db13 ceph-osd[5792]: 2022-03-05T19:29:43.700+0000
> >>>>>> 7f1613080700 -1 osd.97 486896 *** Got signal Interrupt ***
> >>>>>> Mar 05 19:29:43 s3db13 ceph-osd[5792]: 2022-03-05T19:29:43.700+0000
> >>>>>> 7f1613080700 -1 osd.97 486896 *** Immediate shutdown
> >>>>>> (osd_fast_shutdown=true) ***
> >>>>>> Mar 05 19:29:44 s3db13 systemd[1]: ceph-osd@97.service: Succeeded.
> >>>>>> Mar 05 19:29:54 s3db13 systemd[1]: ceph-osd@97.service: Scheduled
> >>>>> restart
> >>>>>> job, restart counter is at 1.
> >>>>>> Mar 05 19:29:54 s3db13 systemd[1]: Stopped Ceph object storage
> daemon
> >>>>>> osd.97.
> >>>>>> Mar 05 19:29:54 s3db13 systemd[1]: Starting Ceph object storage
> daemon
> >>>>>> osd.97...
> >>>>>> Mar 05 19:29:54 s3db13 systemd[1]: Started Ceph object storage
> daemon
> >>>>>> osd.97.
> >>>>>> Mar 05 19:29:55 s3db13 ceph-osd[3236773]:
> 2022-03-05T19:29:55.116+0000
> >>>>>> 7f5852f74d80 -1 Falling back to public interface
> >>>>>> Mar 05 19:30:34 s3db13 ceph-osd[3236773]:
> 2022-03-05T19:30:34.746+0000
> >>>>>> 7f5852f74d80 -1 osd.97 486896 log_to_monitors {default=true}
> >>>>>> --
> >>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal
> abweichend
> >>> im
> >>>>>> groüen Saal.
> >>>>>> _______________________________________________
> >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>>
> >>>
> >>> _______________________________________________
> >>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >>>
> >>
> >>
> >> --
> >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> >> groüen Saal.
> >>
> >>
> >>
> >
> > --
> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
> > groüen Saal.
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
--
Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im
groüen Saal.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
