I'm afraid the parameter mgr mgr/cephadm/default_cephadm_command_timeout is buggy.
Once not on default anymore, MGR is preparing the parameter a bit (e.g. substracting 5 secs)
And there making it float, but cephadm is not having it (not even if I try the default 900 myself):
[WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices
osd.all-available-devices: cephadm exited with an error code: 2, stderr:usage: cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b
[-h] [--image IMAGE] [--docker] [--data-dir DATA_DIR]
[--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR]
[--sysctl-dir SYSCTL_DIR] [--unit-dir UNIT_DIR] [--verbose]
[--timeout TIMEOUT] [--retry RETRY] [--env ENV] [--no-container-init]
[--no-cgroups-split]
{version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,zap-osds,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,registry-login,gather-facts,host-maintenance,agent,disk-rescan}
...
cephadm.8b92cafd937eb89681ee011f9e70f85937fd09c4bd61ed4a59981d275a1f255b: error: argument --timeout: invalid int value: '895.0'
This also let to a status panic spiral reporting plenty of host and services missing failing (I assume orch failing due to cephadm complaining about the parameter).
I got it under control by removing the parameter again from the config (ceph config rm mgr mgr/cephadm/default_cephadm_command_timeout).
And the restarting all MGRs manually (systemctl restart..., again since orch was kinda useless at this stage).
Anyhow, is there any other way I can adapt this parameter?
Or maybe look into speeding up LV creation (if this is the bootleneck)?
Thanks a lot,
Mathias
-----Original Message-----
From: Kuhring, Mathias <mathias.kuhring@xxxxxxxxxxxxxx>
Sent: Friday, March 22, 2024 5:38 PM
To: Eugen Block <eblock@xxxxxx>; ceph-users@xxxxxxx
Subject: Re: [ext] Re: cephadm auto disk preparation and OSD installation incomplete
Hey Eugen,
Thank you for the quick reply.
The 5 missing disks on the one host were completely installed after I fully cleaned them up as I described.
So, seems a smaller number of disks can make it.
Regarding the other host with 40 disks:
Failing the MGR didn't have any effect.
There are nor errors in `/var/log/ceph/cephadm.log`.
But a bunch of repeating image listings like:
cephadm --image quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 --timeout 895 ls
But `ceph log last 200 debug cephadm` gave me a bunch of interesting errors (Excerpt below. Is there any preferred method to provide bigger logs?).
So, there are some timeouts, which might play into the assumption that ceph-volume is a bit overwhelmed by the number of disks.
Shy assumption, but maybe LV creation is taking way too long (is cephadm waiting for all of them in bulk?) and times out with the default 900 secs.
However, LVs are created and cephadm will not consider them next round ("has a filesystem").
I'm testing this theory right now by bumping up the limit to 2 hours (and the restart with "fresh" disks again):
ceph config set mgr mgr/cephadm/default_cephadm_command_timeout 7200
However, there are also mentions of the host being not reachable: "Unable to reach remote host ceph-3-11"
But this seems to be limited to cephadm / ceph orch, so basically MGR but not the rest of the cluster (i.e. MONs, OSDs, etc. are communicating happily, as far as I can tell).
During my fresh run, I do notice more hosts being apperently down:
0|0[root@ceph-3-10 ~]# ceph orch host ls | grep Offline
ceph-3-7 172.16.62.38 rgw,osd,_admin Offline
ceph-3-10 172.16.62.41 rgw,osd,_admin,prometheus Offline
ceph-3-11 172.16.62.43 rgw,osd,_admin Offline
osd-mirror-2 172.16.62.23 rgw,osd,_admin Offline
osd-mirror-3 172.16.62.24 rgw,osd,_admin Offline
But I wonder if this just a side effect of the MGR (cephadm/orch) being too busy/overwhelmed with e.g. deploying the new OSDs.
I will update you once the next round is done or failed.
Best Wishes,
Mathias
ceph log last 200 debug cephadm
...
2024-03-20T09:19:24.917834+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 82122 : cephadm [INF] Detected new or changed devices on ceph-3-11
2024-03-20T09:34:28.877718+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83339 : cephadm [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
service_id: all-available-devices
service_name: osd.all-available-devices
placement:
host_pattern: '*'
spec:
data_devices:
all: true
filter_logic: AND
objectstore: bluestore
''')): Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout) ...
raise TimeoutError()
concurrent.futures._base.TimeoutError
During handling of the above exception, another exception occurred:
...
orchestrator._interface.OrchestratorError: Command timed out on host cephadm deploy (osd daemon) (default 900 second timeout)
2024-03-20T09:34:28.881472+0000 mgr.osd-mirror-4.dkzbkw (mgr.339518816) 83340 : cephadm [ERR] Task exception was never retrieved
future: <Task finished coro=<OSDService.create_from_spec.<locals>.all_hosts() done, defined at /usr/share/ceph/mgr/cephadm/services/osd.py:72> exception=RuntimeError('cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11. ',)> Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 75, in all_hosts
return await gather(*futures)
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 64, in create_from_spec_one
replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 96, in create_single_host
code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:Unable to reach remote host ceph-3-11.
...
''')): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf
Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae165222496e09 -e NODE_NAME=ceph-3-11 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=all-available-devices -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/run/ceph:z -v /var/log/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b:/var/log/ceph:z -v /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpfoibulv3:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpjq5uxhj1:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:1fb108217b110c01c480e32d0cfea0e19955733537af7bb8cbae16522249
6e09 lvm batch --no-auto /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz --yes --no-systemd ...
/usr/bin/docker: stderr raise RuntimeError("Device {} has a filesystem.".format(self.dev_path))
/usr/bin/docker: stderr RuntimeError: Device /dev/sdm has a filesystem.
...
RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/7efa00f9-182f-40f4-9136-d51895db1f0b/config/ceph.conf
...
-----Original Message-----
From: Eugen Block <eblock@xxxxxx>
Sent: Thursday, March 21, 2024 3:28 PM
To: ceph-users@xxxxxxx
Subject: [ext] Re: cephadm auto disk preparation and OSD installation incomplete
Hi,
before getting into that the first thing I would do is to fail the mgr. There have been too many issues where failing over the mgr resolved many of them.
If that doesn't help, the cephadm.log should show something useful (/var/log/ceph/cephadm.log on the OSD hosts, I'm still not too familiar with the whole 'ceph log last 200 debug cephadm' thing).
I remember reports in earlier versions of ceph-volume (probably
pre-cephadm) where not all OSDs were created if the host had many disks to deploy. But I can't find those threads right now.
And it's strange that on the second cluster no OSD is created at all, but again, maybe fail the mgr first before looking deeper into it.
Regards,
Eugen
Zitat von "Kuhring, Mathias" <mathias.kuhring@xxxxxxxxxxxxxx>:
> Dear ceph community,
>
> We have trouble with new disks not being properly prepared resp.
> OSDs not being fully installed by cephadm.
> We just added one new node each with ~40 HDDs each to two of our ceph
> clusters.
> In one cluster all but 5 disks got installed automatically.
> In the other none got installed.
>
> We are on ceph version 17.2.7
> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both
> clusters.
> (I haven't added new disks since the last upgrade if I recall correctly).
>
> This is our OSD service definition:
> ```
> 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export
> service_type: osd
> service_id: all-available-devices
> service_name: osd.all-available-devices
> placement:
> host_pattern: '*'
> spec:
> data_devices:
> all: true
> filter_logic: AND
> objectstore: bluestore
> ---
> service_type: osd
> service_id: unmanaged
> service_name: osd.unmanaged
> unmanaged: true
> spec:
> filter_logic: AND
> objectstore: bluestore
> ```
>
> Usually, new disks are installed properly (as expected due to
> all-available-devices).
> This time, I can see that LVs were created (via `lsblk`, `lvs`,
> `cephadm ceph-volume lvm list`).
> And OSDs are entered to the crushmap.
> However, they are not assigned to a host yet, nor do they have a type
> or weight, e.g.:
> ```
> 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd"
> 518 0 osd.518 down 0 1.00000
> 519 0 osd.519 down 0 1.00000
> 520 0 osd.520 down 0 1.00000
> 521 0 osd.521 down 0 1.00000
> 522 0 osd.522 down 0 1.00000
> ```
>
> And there is also no OSD daemon created (no docker container).
> So, OSD creation is somehow stuck halfway.
>
> I thought of fully cleaning up the OSD/disks.
> Hopping cephadm might pick them up properly next time.
> Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap
> --destroy /dev/sdab` results in these errors:
> ```
> /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing
> initialization failed: Device or resource busy
> /usr/bin/docker: stderr --> failed to wipefs device, will try again to
> workaround probable race condition ```
>
> So, I cleaned up more manually with purging them from crush and
> "resetting" disk and LV with dd and dmsetup, resp.:
> ```
> ceph osd purge 480 --force
> dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove
> ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f
> 513--4513--86c1--c7ad63e6c142
> ```
>
> ceph-volume still reported the old volumes, but then zapping actually
> got rid of them (only cleaned out the left-over entries, I guess).
>
> Now, cephadm was able to get one OSD up, when I did this cleanup for
> only one disk.
> When I did it in bulk for the rest, they all got stuck again the same way.
>
> Looking into ceph-volume logs (here for osd.522 as representative):
> ```
> 0|0[root@ceph-2-11
> /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316
> -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316
> -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316
> ```
>
> ceph-volume only reports keyring creation:
> ```
> [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating
> keyring file for osd.522
> [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running
> command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring
> --create-keyring --name osd.522 --add-key
> AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ==
> ```
>
> In the OSD logs I found a couple of these, but don't know if they are
> related:
> ```
> 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb:
> [db/column_family.cc:546] Failed to register data paths of column
> family (id: 11, name: P) ```
>
> Has anyone seen this behaviour before?
> Or could tell me where I should look next to troubleshoot this (which logs)?
> Any help is appreciated.
>
> Best Wishes,
> Mathias
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an
> email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
