I've rebuilt my rgw several times since our last emails and every time I
get stuck more or less to the same point, so let me share with you all
what I am doing step by step, along with the specs I use.
1. I create a default RGW container and put it on my second monitor.
Default configs are set
ceph orch apply rgw staging staging --placement="1 ceph-monitor2"
2. I create a default realm
radosgw-admin realm create --rgw-realm=staging
3. I create a second realm
radosgw-admin realm create --rgw-realm=test
4. I create a master zonegroup for my second realm
radosgw-admin zonegroup create --rgw-zonegroup=test --rgw-realm=test
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef
5. I create a master zone for my second realm
radosgw-admin zone create --rgw-zonegroup=test --rgw-realm=test
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef --master --rgw-zone=test
6. I try to commit my second realm (it fails, usually with a can't init
error)
radosgw-admin period update --commit
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef
7.I try to create my container with the following spec file
ceph orch apply -i multi-realm.spec
cat ceph orch apply -i multi-realm.spec
service_type: rgw
service_id: test.test
placement:
label: rgw-aux
count: 1
spec:
rgw_realm: test
rgw_zone: test
ssl: false
rgw_frontend_port: 9999
I've set Cephadm to output to a logfile and that logfile tells me:
Cannot find zone id=35ea43e5-e0d2-4ebe-81d6-9274231bd542 (name=test)
And that is even though the zone exists and is in the realm this gateway
is set to serve. Is the commit supposed to work before or after the
gateway is setup?
On 11/16/21 2:45 AM, Eugen Block wrote:
Hi,
Couldn't init storage provider (RADOS)
I usually see this when my rgw config is wrong, can you share your rgw
spec(s)?
Zitat von J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx>:
After searching through logs and figuring out how cephadm works, I've
figured out that when cephadm tries to create the new systemd service
and launch the new RGW container, it fails with this error :
Couldn't init storage provider (RADOS)
Systemd then complains that the service doesn't exist :
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.twltpl.service:
Main process exited, code=exited, status=5/NOTINSTALLED
I can't seem to find further logs anywhere regarding why the storage
provider fails to init. Googling indicate issues with the number of
pgs in past versions, but I believe it shouldn't be an issue here,
since the amount of PG auto-adjusts. How could I find out why this
happens?
On 11/15/21 11:02 AM, J-P Methot wrote:
Yes, I'm trying to add a RGW container on a second port on the same
server. For example, I do :
ceph orch apply rgw test test
--placement="ceph-monitor1:[10.50.47.3:9999]"
and this results in :
ceph orch ls
NAME RUNNING REFRESHED AGE
PLACEMENT IMAGE NAME
IMAGE ID
rgw.test.test 0/1 2s ago 5s
ceph-monitor1:[10.50.47.3:9999] docker.io/ceph/ceph:v15 <unknown>
the image and container ID being unknown is making me scratch my
head. A look in the log files show this:
2021-11-15 10:50:12,253 INFO Deploy daemon
rgw.test.test.ceph-monitor1.rtoiwh ...
2021-11-15 10:50:12,254 DEBUG Running command: /usr/bin/docker run
--rm --ipc=host --net=host --entrypoint stat -e
CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=ceph-monitor1
docker.io/ceph/ceph:v15 -c %u %g /var/lib/ceph
2021-11-15 10:50:12,452 DEBUG stat: stdout 167 167
2021-11-15 10:50:12,525 DEBUG Running command: install -d -m0770 -o
167 -g 167 /var/run/ceph/04c5d4a4-8815-45fb-b97f-027252d1aea5
2021-11-15 10:50:12,534 DEBUG Running command: systemctl daemon-reload
2021-11-15 10:50:12,869 DEBUG Running command: systemctl stop
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh
2021-11-15 10:50:12,879 DEBUG Running command: systemctl
reset-failed
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh
2021-11-15 10:50:12,884 DEBUG systemctl: stderr Failed to reset
failed state of unit
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh.service:
Unit
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh.service
not loaded
journalctl -xe shows the service entered failed state, without any
real useful information
Nov 15 10:50:24 ceph-monitor1 systemd[1]:
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh.service:
Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
What I understand from this is that I'm doing the right thing, it's
just my cephadm that's breaking, somehow.
On 11/15/21 5:59 AM, Eugen Block wrote:
Hi,
it's not entirely clear how your setup looks like, are you trying
to setup multiple RGW containers on the same host(s) to serve
multiple realms or do you have multiple RGWs for that?
You can add a second realm with a spec file or via cli (which you
already did). If you want to create multiple RGW containers per
host you need to specify a different port for every RGW, see the
docs [1] for some examples.
This worked just fine in my Octopus lab except for a little mistake
in the "port" spec, apparently this
spec:
port: 8000
doesn't work:
host1:~ # ceph orch apply -i rgw2.yaml
Error EINVAL: ServiceSpec: __init__() got an unexpected keyword
argument 'port'
But this does:
spec:
rgw_frontend_port: 8000
Now I have two RGW containers on each host, serving two different
realms.
[1] https://docs.ceph.com/en/latest/cephadm/services/rgw/
Zitat von J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx>:
Hi,
I'm testing out adding a second RGW realm to my single ceph
cluster. This is not very well documented though, since obviously
realms were designed for multi-site deployments.
Now, what I can't seem to figure is if I need to deploy a
container with cephadm to act as a frontend for this second realm
and, if so, how? I've set a frontend port and address when I
created the second realm, but my attempts at creating a RGW
container for that realm didn't work at all, with the container
just not booting up.
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
