FWIW, this is still required with Jewel 10.2.5. It sounded like it was finally fixed from the release notes, but i had the same issue. Fortunately Micha's steps are easy and fix it right up.
In my case i didn't think i had any mixed RGWs - was planning to stop them all first - but i had forgotten about my monitoring system which runs 'radosgw-admin' -- that part upgraded first, before i'd stopped any of my Infernalis RGW's.
-Ben
On Thu, Jul 28, 2016 at 7:50 AM, Arvydas Opulskis <Arvydas.Opulskis@xxxxxxxxxx> wrote:
Hi,
We solved it by running Micha scripts, plus we needed to run period update and commit commands (for some reason we had to do it in separate commands):
radosgw-admin period update
radosgw-admin period commit
Btw, we added endpoints to json file, but I am not sure these are needed.
And I agree with Micha - this should be noticed in upgrade instructions on Ceph site. We run into this trap on our prod env (upgrading Infernalis -> Jewel). Maybe we should test it more next time..
Br,
Arvydas
-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com ] On Behalf Of Micha Krause
Sent: Wednesday, July 6, 2016 2:46 PM
To: ceph-users@xxxxxxxxxxxxxx
Subject: Re: Can't create bucket (ERROR: endpoints not configured for upstream zone)
Hi,
I think I found a Solution for my Problem, here are my findings:
This Bug can be easily reproduced in a test environment:
1. Delete all rgw related pools.
2. Start infernalis radosgw to initialize them again.
3. Create user.
4. User creates bucket.
5. Upgrade radosgw to jewel
6. User creates bucket -> fail
I found this scary script from Yehuda: https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-defa ult-zone/src/fix-zone
which needs to be modified according to http://www.spinics.net/lists/ceph-users/msg27957.html .
After the modification, a lot of the script becomes obsolete (in my opinion), and can be rewritten to this (less scary):
#!/bin/sh
set -x
RADOSGW_ADMIN=radosgw-admin
echo "Exercise initialization code"
$RADOSGW_ADMIN user info --uid=foo # exercise init code (???)
echo "Get default zonegroup"
$RADOSGW_ADMIN zonegroup get --rgw-zonegroup=default | sed 's/"id":.*/"id": "default",/g' | sed 's/"master_zone.*/"master_zone": "default",/g' > default-zg.json
echo "Get default zone"
$RADOSGW_ADMIN zone get --zone-id=default > default-zone.json
echo "Creating realm"
$RADOSGW_ADMIN realm create --rgw-realm=myrealm
echo "Creating default zonegroup"
$RADOSGW_ADMIN zonegroup set --rgw-zonegroup=default < default-zg.json
echo "Creating default zone"
$RADOSGW_ADMIN zone set --rgw-zone=default < default-zone.json
echo "Setting default zonegroup to 'default'"
$RADOSGW_ADMIN zonegroup default --rgw-zonegroup=default
echo "Setting default zone to 'default'"
$RADOSGW_ADMIN zone default --rgw-zone=default
My plan to do this in production is now:
1. Stop all rados-gateways
2. Upgrade rados-gateways to jewel
3. Run less scary script
4. Start rados-gateways
This whole thing is a serious problem, there should at least be a clear notice in the Jewel release notes about this. I was lucky to catch this in my test-cluster, I'm sure a lot of people will run into this in production.
Micha Krause
Am 05.07.2016 um 09:30 schrieb Micha Krause:
> *bump*
>
> Am 01.07.2016 um 13:00 schrieb Micha Krause:
>> Hi,
>>
>> > In Infernalis there was this command:
>>>
>>> radosgw-admin regions list
>>>
>>> But this is missing in Jewel.
>>
>> Ok, I just found out that this was renamed to zonegroup list:
>>
>> root@rgw01:~ # radosgw-admin --id radosgw.rgw zonegroup list
>> read_default_id : -2 {
>> "default_info": "",
>> "zonegroups": [
>> "default"
>> ]
>> }
>>
>> This looks to me like there is indeed only one zonegroup or region configured.
>>
>> Micha Krause
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
