Upgrade failed, now ceph orch broken

["Ragan, Tj (Dr.)" <tj.ragan@xxxxxxxxxxxxxxx>]

Hi All,

I have a cluster that that I tried upgrading from 15.2.4 to 15.2.5 using the command ‘ceph orch upgrade start —ceph-version 15.2.5’.  After upgrading two of the three mgrs, the third mgr failed and the upgrade stopped.  I was able to get the third mgr upgraded by changing the systemd file and manually removing the podman image.  I have upgraded ceph-common outside the containers on all the mgr/mon nodes.  The cluster repots HEALTH_OK.  However, any ‘ceph orch’ command I issue now always returns:

Error ENOENT: Module not found

If I run ‘ceph orch ps —verbose’, I get:


parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name=None, cluster=None, cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['orch', 'ps']
cmd000: pg stat

...

better match: 2.5 > 1.5: orch ps [<hostname>] [<service_name>] [<daemon_type>] [<daemon_id>] [plain|json|json-pretty|yaml] [--refresh]
bestcmds_sorted:
[{'flags': 8,
  'help': 'List daemons known to orchestrator',
  'module': 'mgr',
  'perm': 'r',
  'sig': [argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=orch),
          argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=ps),
          argdesc(<class 'ceph_argparse.CephString'>, req=False, name=hostname, n=1, numseen=0),
          argdesc(<class 'ceph_argparse.CephString'>, req=False, name=service_name, n=1, numseen=0),
          argdesc(<class 'ceph_argparse.CephString'>, req=False, name=daemon_type, n=1, numseen=0),
          argdesc(<class 'ceph_argparse.CephString'>, req=False, name=daemon_id, n=1, numseen=0),
          argdesc(<class 'ceph_argparse.CephChoices'>, req=False, name=format, n=1, numseen=0, strings=plain|json|json-pretty|yaml),
          argdesc(<class 'ceph_argparse.CephBool'>, req=False, name=refresh, n=1, numseen=0)]}]
Submitting command:  {'prefix': 'orch ps', 'target': ('mon-mgr', '')}
submit ['{"prefix": "orch ps", "target": ["mon-mgr", ""]}'] to mon-mgr
Error ENOENT: Module not found


Any help would be appreciated.

Thanks,
-TJ Ragan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx