Hi,
can you elaborate a bit what happened and why "a few reboots" were
required? 64% inactive PGs and 700 unkown PGs doesn't look too good.
Has this improved a bit since your post? If ceph orch commands are not
responding it could point to a broken mgr, do you see anything in the
logs of the active MGR? Have you tried to failover the MGR and then
watched the logs of the new active one?
Zitat von Neil Brown <nebrown@xxxxxxxxxxxxxxxxx>:
Hi all,
I have a Ceph 17.2.5 cluster deployed via cephadm. After a few
reboots it has now entered a fairly broken state as shown below.
I am having trouble even beginning to diagnose this as a lot of the
commands just hang. For example “cephadm ps”, “ceph orch ls” just
hang forever. Other commands like “ceph pg 7.4e query” return JSON
errors.
As it stands, the CephFS filesystem is inaccessible, as too is my
RBD mount onto Windows Server 2019. Even though the cluster says
HEALTH_WARN, it seems to be in a pretty terminal state right now ☹
I wonder if any of you wonderful people could help point me in the
right direction?
root@c-dc01-ceph01:~# ceph pg 7.4e query
Couldn't parse JSON : Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/usr/bin/ceph", line 1326, in <module>
retval = main()
File "/usr/bin/ceph", line 1246, in main
sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli')
File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 993,
in parse_json_funcsigs
raise e
File "/usr/lib/python3/dist-packages/ceph_argparse.py", line 990,
in parse_json_funcsigs
overall = json.loads(s)
File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
root@c-dc01-ceph01:~# ceph status
cluster:
id: 2a6ec9f2-56c4-11ed-a428-bdec5d6d07e0
health: HEALTH_WARN
3 failed cephadm daemon(s)
1 filesystem is degraded
1 MDSs report slow metadata IOs
Reduced data availability: 6686 pgs inactive, 5982 pgs peering
services:
mon: 3 daemons, quorum
c-dc02-ceph01,c-dc03-ceph01,c-dc01-ceph01 (age 2h)
mgr: c-dc02-ceph01.touart(active, since 39h), standbys:
c-dc01-ceph01.owmpxa
mds: 1/1 daemons up, 2 standby
osd: 144 osds: 144 up (since 39h), 144 in (since 4w);
2607 remapped pgs
rbd-mirror: 2 daemons active (2 hosts)
data:
volumes: 0/1 healthy, 1 recovering
pools: 15 pools, 9293 pgs
objects: 597.03k objects, 2.1 TiB
usage: 3.8 TiB used, 248 TiB / 252 TiB avail
pgs: 7.576% pgs unknown
64.371% pgs not active
691820/1791087 objects misplaced (38.626%)
5982 peering
2607 active+clean+remapped
704 unknown
io:
client: 850 B/s rd, 0 op/s rd, 0 op/s wr
progress:
Global Recovery Event (15h)
[=======.....................] (remaining: 4d)
root@c-dc01-ceph01:~# ceph version
ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757)
quincy (stable)
Thanks,
Neil.
This email comprises confidential information of Mercedes-Benz Grand
Prix Limited ("MGP") unless it contains an explicit statement to the
contrary made by an authorised representative of MGP.
Contracts may only be concluded on behalf of MGP by its authorised
signatories and not solely by email communication. No employee,
agent, contractor, consultant and/or other representative of MGP is
authorised to conclude any legally binding agreement on behalf of
MGP by email alone without the express prior written confirmation of
two authorised signatories of MGP.
Mercedes-Benz Grand Prix Limited. Registered in England No. 787446.
Registered Office at Mercedes-Benz Grand Prix Limited, Operations
Centre, Brackley, Northants NN13 7BD.
Note: The MGP Legal Department also acts on behalf of Mercedes-Benz
Motorsport Limited ("MBM") and the above notice applies mutatis
mutandis in respect of all email communications of MBM. MBM:
Mercedes-Benz Motorsport Limited. Registered in England No.
13057973. Registered office at Mercedes-Benz Motorsport Limited,
Lauda Drive, Brackley, Northants NN13 7BD.
Please consider the environment before printing this email.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
