Re: octopus upgrade stuck: Assertion `map->require_osd_release >= ceph_release_t::mimic' failed.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



This is a little beyond my understanding of Ceph, but let me take a crack at it...  I've found that Ceph tends to be fairly logical, mostly.

require_osd_release looks like a cluster wide configuration value which controls the minimum required version for an OSD daemon to join the cluster.
check_osdmap_features looks like an upgrade validation which checks that the above minimum version is.
unknown -> luminous suggests that a) require_osd_release is not set in your cluster, and b) because it is unset it is being assumed to be luminous.

This looks like a sane way to check, before upgrading, if the upgrade is likely to complete correctly.

>From your previous email "`map->require_osd_release >= ceph_release_t::mimic' failed." suggests that the upgrade is looking for the above configuration value needs to be mimic or higher.

I would suggest that you set this configuration value to mimic or nautilus, as appropriate for your cluster, and retry the upgrade.

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
DHilsbos@xxxxxxxxxxxxxx 
www.PerformAir.com


-----Original Message-----
From: Ml Ml [mailto:mliebherr99@xxxxxxxxxxxxxx] 
Sent: Wednesday, March 25, 2020 2:16 PM
To: ceph-users
Subject:  Re: octopus upgrade stuck: Assertion `map->require_osd_release >= ceph_release_t::mimic' failed.

in the logs it says:

2020-03-25T22:10:00.823+0100 7f0bd5320e00  0 <cls>
/build/ceph-15.2.0/src/cls/hello/cls_hello.cc:312: loading cls_hello
2020-03-25T22:10:00.823+0100 7f0bd5320e00  0 osd.32 57223 crush map
has features 288232576282525696, adjusting msgr requires for clients
2020-03-25T22:10:00.823+0100 7f0bd5320e00  0 osd.32 57223 crush map
has features 288232576282525696 was 8705, adjusting msgr requires for
mons
2020-03-25T22:10:00.823+0100 7f0bd5320e00  0 osd.32 57223 crush map
has features 1008808516661821440, adjusting msgr requires for osds
2020-03-25T22:10:00.823+0100 7f0bd5320e00  1 osd.32 57223
check_osdmap_features require_osd_release unknown -> luminous
2020-03-25T22:10:04.695+0100 7f0bd5320e00  0 osd.32 57223 load_pgs
2020-03-25T22:10:10.907+0100 7f0bcc01d700  4 rocksdb:
[db/compaction_job.cc:1332] [default] [JOB 3] Generated table #59886:
2107241 keys, 72886355 bytes
2020-03-25T22:10:10.907+0100 7f0bcc01d700  4 rocksdb: EVENT_LOG_v1
{"time_micros": 1585170610911598, "cf_name": "default", "job": 3,
"event": "table_file_creation", "file_number": 59886, "file_size":
72886355, "table_properties": {"data_size": 67112666, "index_size":
504659, "filter_size": 5268165, "raw_key_size": 38673953,
"raw_average_key_size": 18, "raw_value_size": 35746098,
"raw_average_value_size": 16, "num_data_blocks": 16488, "num_entries":
2107241, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
2020-03-25T22:10:13.047+0100 7f0bd5320e00  0 osd.32 57223 load_pgs
opened 230 pgs
2020-03-25T22:10:13.047+0100 7f0bd5320e00 -1 osd.32 57223
log_to_monitors {default=true}
2020-03-25T22:10:13.107+0100 7f0bd5320e00  0 osd.32 57223 done with
init, starting boot process
2020-03-25T22:10:13.107+0100 7f0bd5320e00  1 osd.32 57223 start_boot


does the line:
  check_osdmap_features require_osd_release unknown -> luminous
mean it thinks the local osd itself is luminous?

On Wed, Mar 25, 2020 at 8:12 PM Ml Ml <mliebherr99@xxxxxxxxxxxxxx> wrote:
>
> Hello List,
>
> i followed:
>  https://ceph.io/releases/v15-2-0-octopus-released/
>
> I came from a healthy nautilus and i am stuck at:
>   5.) Upgrade all OSDs by installing the new packages and restarting
> the ceph-osd daemons on all OSD host
>
> When i try to start an osd like this, i get:
>   /usr/bin/ceph-osd -f --cluster ceph --id 32 --setuser ceph --setgroup ceph
> ...
> 2020-03-25T20:11:03.292+0100 7f2762874e00 -1 osd.32 57223
> log_to_monitors {default=true}
> ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
> PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
> `map->require_osd_release >= ceph_release_t::mimic' failed.
> ceph-osd: /build/ceph-15.2.0/src/osd/PeeringState.cc:109: void
> PGPool::update(ceph::common::CephContext*, OSDMapRef): Assertion
> `map->require_osd_release >= ceph_release_t::mimic' failed.
> *** Caught signal (Aborted) **
>  in thread 7f274854f700 thread_name:tp_osd_tp
> Aborted
>
>
>
> My current status:
>
> root@ceph03:~# ceph osd tree
> ID  CLASS  WEIGHT    TYPE NAME        STATUS  REWEIGHT  PRI-AFF
> -1         60.70999  root default
> -2         20.25140      host ceph01
>  0    hdd   1.71089          osd.0        up   1.00000  1.00000
>  8    hdd   2.67029          osd.8        up   1.00000  1.00000
> 11    hdd   1.59999          osd.11       up   1.00000  1.00000
> 12    hdd   1.59999          osd.12       up   1.00000  1.00000
> 14    hdd   2.79999          osd.14       up   1.00000  1.00000
> 18    hdd   1.59999          osd.18       up   1.00000  1.00000
> 22    hdd   2.79999          osd.22       up   1.00000  1.00000
> 23    hdd   2.79999          osd.23       up   1.00000  1.00000
> 26    hdd   2.67029          osd.26       up   1.00000  1.00000
> -3         23.05193      host ceph02
>  2    hdd   2.67029          osd.2        up   1.00000  1.00000
>  3    hdd   2.00000          osd.3        up   1.00000  1.00000
>  7    hdd   2.67029          osd.7        up   1.00000  1.00000
>  9    hdd   2.67029          osd.9        up   1.00000  1.00000
> 13    hdd   2.00000          osd.13       up   1.00000  1.00000
> 16    hdd   1.59999          osd.16       up   1.00000  1.00000
> 19    hdd   2.38409          osd.19       up   1.00000  1.00000
> 24    hdd   2.67020          osd.24       up   1.00000  1.00000
> 25    hdd   1.71649          osd.25       up   1.00000  1.00000
> 28    hdd   2.67029          osd.28       up   1.00000  1.00000
> -4         17.40666      host ceph03
>  5    hdd   1.71660          osd.5      down   1.00000  1.00000
>  6    hdd   1.71660          osd.6      down   1.00000  1.00000
> 10    hdd   2.67029          osd.10     down   1.00000  1.00000
> 15    hdd   2.00000          osd.15     down   1.00000  1.00000
> 17    hdd   1.20000          osd.17     down   1.00000  1.00000
> 20    hdd   1.71649          osd.20     down   1.00000  1.00000
> 21    hdd   2.00000          osd.21     down   1.00000  1.00000
> 27    hdd   1.71649          osd.27     down   1.00000  1.00000
> 32    hdd   2.67020          osd.32     down   1.00000  1.00000
>
> root@ceph03:~#  ceph osd dump | grep require_osd_release
> require_osd_release nautilus
>
> root@ceph03:~# ceph osd versions
> {
>     "ceph version 14.2.8 (88c3b82e8bc76d3444c2d84a30c4a380d6169d46)
> nautilus (stable)": 19
> }
>
> oot@ceph03:~# ceph mon dump | grep min_mon_release
> dumped monmap epoch 12
> min_mon_release 15 (octopus)
>
>
> ceph versions
> {
>     "mon": {
>         "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
>     },
>     "mgr": {
>         "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
>     },
>     "osd": {
>         "ceph version 14.2.8
> (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19
>     },
>     "mds": {
>         "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 3
>     },
>     "overall": {
>         "ceph version 14.2.8
> (88c3b82e8bc76d3444c2d84a30c4a380d6169d46) nautilus (stable)": 19,
>         "ceph version 15.2.0
> (dc6a0b5c3cbf6a5e1d6d4f20b5ad466d76b96247) octopus (rc)": 9
>     }
> }
>
>
> Why does it complain about map->require_osd_release >= ceph_release_t::mimic ?
>
> Cheers,
> Michael
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux