Re: Upgrade from v15.2.16 to v16.2.7 not starting

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Do you see anything suspicious in /var/log/ceph/cephadm.log? Also check the mgr logs for any hints.


Zitat von Lo Re  Giuseppe <giuseppe.lore@xxxxxxx>:

Hi,

We have happily tested the upgrade from v15.2.16 to v16.2.7 with cephadm on a test cluster made of 3 nodes and everything went smoothly. Today we started the very same operation on the production one (20 OSD servers, 720 HDDs) and the upgrade process doesn’t do anything at all…

To be more specific, we have issued the command

ceph orch upgrade start --image quay.io/ceph/ceph:v16.2.7

and soon after “ceph -s” reports

    Upgrade to quay.io/ceph/ceph:v16.2.7 (0s)
      [............................]

But only for few seconds, after that

[root@naret-monitor01 ~]# ceph -s
  cluster:
    id:     63334166-d991-11eb-99de-40a6b72108d0
    health: HEALTH_OK

  services:
mon: 3 daemons, quorum naret-monitor01,naret-monitor02,naret-monitor03 (age 7d) mgr: naret-monitor01.tvddjv(active, since 60s), standbys: naret-monitor02.btynnb
    mds: cephfs:1 {0=cephfs.naret-monitor01.uvevbf=up:active} 2 up:standby
    osd: 760 osds: 760 up (since 6d), 760 in (since 2w)
rgw: 3 daemons active (cscs-realm.naret-zone.naret-rgw01.qvhhbi, cscs-realm.naret-zone.naret-rgw02.pduagk, cscs-realm.naret-zone.naret-rgw03.aqdkkb)

  task status:

  data:
    pools:   30 pools, 16497 pgs
    objects: 833.14M objects, 3.1 PiB
    usage:   5.0 PiB used, 5.9 PiB / 11 PiB avail
    pgs:     16460 active+clean
             37    active+clean+scrubbing+deep

  io:
    client:   4.7 MiB/s rd, 4.0 MiB/s wr, 122 op/s rd, 47 op/s wr

  progress:
    Removing image fulen-hdd/c991f6fdf41964 from trash (53s)
      [............................] (remaining: 81m)



The command “ceph orch upgrade status” says:

{
    "target_image": "quay.io/ceph/ceph:v16.2.7",
    "in_progress": true,
    "services_complete": [],
    "message": ""
}

It doesn’t even pull the container image.
I have tested that the podman pull command works, I was able to pull quay.io/ceph/ceph:v16.2.7.

“ceph -w” and “ceph -W cephadm” don’t report any activity related to the upgrade.


Does anyone have seen anything similar?
Do you have advises on how to understand what’s holding the upgrade process to actually start?

Thanks in advance,

Giuseppe
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux