On Tue, Jan 16, 2024 at 12:11 AM Chris Palmer <chris.palmer@xxxxxxxxx> wrote: > Updates on both problems: > > Problem 1 > -------------- > > The bookworm/reef cephadm package needs updating to accommodate the last > change in /usr/share/doc/adduser/NEWS.Debian.gz: > > System user home defaults to /nonexistent if --home is not specified. > Packages that call adduser to create system accounts should explicitly > specify a location for /home (see Lintian check > maintainer-script-lacks-home-in-adduser). > > i.e. when creating the cephadm user as a system user it needs to > explicitly specify the expected home directory of /home/cephadm. > Hi Chris, thank you for the bug report and the suggestion. could you please file a tracker ticket, so we can track and backport the related fixes? i just created https://github.com/ceph/ceph/pull/55218 in hope to alleviate the problem. > > A workaround is to manually create the user+directory before installing > ceph. > > > Problem 2 > -------------- > > This is a complex set of interactions that prevent many mgr modules > (including dashboard) from running. It is NOT debian-specific and will > eventually bite other distributions as well. At the moment Ceph PR54710 > looks the most promising fix (full or partial). Detail is spread across > the following: > > https://github.com/pyca/cryptography/issues/9016 > https://github.com/ceph/ceph/pull/54710 > https://tracker.ceph.com/issues/63529 > > https://forum.proxmox.com/threads/ceph-warning-post-upgrade-to-v8.129371/page-5 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055212 > https://github.com/pyca/bcrypt/issues/694 IIUC, a backport of https://github.com/ceph/ceph/pull/54710 to reef would address this issue, am i right? > > > > On 12/01/2024 14:29, Chris Palmer wrote: > > More info on problem 2: > > > > When starting the dashboard, the mgr seems to try to initialise > > cephadm, which in turn uses python crypto libraries that lead to the > > python error: > > > > $ ceph crash info > > 2024-01-12T11:10:03.938478Z_2263d2c8-8120-417e-84bc-bb01f5d81e52 > > { > > "backtrace": [ > > " File \"/usr/share/ceph/mgr/cephadm/__init__.py\", line 1, > > in <module>\n from .module import CephadmOrchestrator", > > " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 15, in > > <module>\n from cephadm.service_discovery import ServiceDiscovery", > > " File \"/usr/share/ceph/mgr/cephadm/service_discovery.py\", > > line 20, in <module>\n from cephadm.ssl_cert_utils import SSLCerts", > > " File \"/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py\", > > line 8, in <module>\n from cryptography import x509", > > " File > > \"/lib/python3/dist-packages/cryptography/x509/__init__.py\", line 6, > > in <module>\n from cryptography.x509 import certificate_transparency", > > " File > > > \"/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py\", > > > line 10, in <module>\n from cryptography.hazmat.bindings._rust > > import x509 as rust_x509", > > "ImportError: PyO3 modules may only be initialized once per > > interpreter process" > > ], > > "ceph_version": "18.2.1", > > "crash_id": > > "2024-01-12T11:10:03.938478Z_2263d2c8-8120-417e-84bc-bb01f5d81e52", > > "entity_name": "mgr.xxxxx01", > > "mgr_module": "cephadm", > > "mgr_module_caller": "PyModule::load_subclass_of", > > "mgr_python_exception": "ImportError", > > "os_id": "12", > > "os_name": "Debian GNU/Linux 12 (bookworm)", > > "os_version": "12 (bookworm)", > > "os_version_id": "12", > > "process_name": "ceph-mgr", > > "stack_sig": > > "7815ad73ced094695056319d1241bf7847da19b4b0dfee7a216407b59a7e3d84", > > "timestamp": "2024-01-12T11:10:03.938478Z", > > "utsname_hostname": "xxxxx01.xxx.xxx", > > "utsname_machine": "x86_64", > > "utsname_release": "6.1.0-17-amd64", > > "utsname_sysname": "Linux", > > "utsname_version": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 > > (2023-12-30)" > > } > > > > > > On 12/01/2024 12:39, Chris Palmer wrote: > >> I was delighted to see the native Debian 12 (bookworm) packages turn > >> up in Reef 18.2.1. > >> > >> We currently run a number of ceph clusters on Debian11 (bullseye) / > >> Quincy 17.2.7. These are not cephadm-managed. > >> > >> I have attempted to upgrade a test cluster, and it is not going well. > >> Quincy only supports bullseye, and Reef only supports bookworm, we > >> are reinstalling from bare metal. However I don't think either of > >> these two problems are related to that. > >> > >> Problem 1 > >> -------------- > >> > >> A simple "apt install ceph" goes most of the way, then errors with > >> > >> Setting up cephadm (18.2.1-1~bpo12+1) ... > >> usermod: unlocking the user's password would result in a passwordless > >> account. > >> You should set a password with usermod -p to unlock this user's > >> password. > >> mkdir: cannot create directory ‘/home/cephadm/.ssh’: No such file or > >> directory > >> dpkg: error processing package cephadm (--configure): > >> installed cephadm package post-installation script subprocess > >> returned error exit status 1 > >> dpkg: dependency problems prevent configuration of ceph-mgr-cephadm: > >> ceph-mgr-cephadm depends on cephadm; however: > >> Package cephadm is not configured yet. > >> > >> dpkg: error processing package ceph-mgr-cephadm (--configure): > >> dependency problems - leaving unconfigured > >> > >> > >> The two cephadm-related packages are then left in an error state, > >> which apt tries to continue each time it is run. > >> > >> The cephadm user has a login directory of /nonexistent, however the > >> cephadm --configure script is trying to use /home/cephadm (as it was > >> on Quincy/bullseye). > >> > >> So, we aren't using cephadm, and decided to keep going as the other > >> packages were actually installed, and deal with the package state later. > >> > >> Problem 2 > >> --------------- > >> > >> I upgraded 2/3 monitor nodes without any other problems, and (for the > >> moment) removed the other Quincy monitor prior to rebuild. > >> > >> I then shutdown the remaining Quincy manager, and attempted to start > >> the Reef manager. Although the manager is running, "ceph mgr > >> services" shows it is only providing the restful and not the > >> dashboard service. The log file has lots of the following error: > >> > >> ImportError: PyO3 modules may only be initialized once per > >> interpreter process > >> > >> and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 > >> modules may only be initialized once per interpreter process > >> > >> > >> Questions > >> --------------- > >> > >> 1. Have the Reef/bookworm packages ever been tested in a non-cephadm > >> environment? > >> 2. I want to revert this cluster back to a fully functional state. I > >> cannot bring back up the remaining Quincy monitor though ("require > >> release 18 > 17"). Would I have to go through the procedure of > >> starting over, and trying to rescue the monmap from the OSDs? (OSDs > >> and an active MDS are still up and running Quincy). I'm aware that > >> process exists but have never had to delve into it. > >> > >> > >> Thanks, Chris > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > -- Regards Kefu Chai _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |