This lab environment runs on:
"ceph version 15.2.14-84-gb6e5642e260
(b6e5642e2603d3e6bdbc158feff6a51722214e72) octopus (stable)": 3
Zitat von J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx>:
Yes, the single realm config works without issue. Same with the rgw
container. It's with the second realm that everything stops working.
Tell me, what version of Ceph are you using? We are on 15.2.13 on
our test environment, as it seems to be the latest Octopus release
for cephadm containers, despite 15.2.14 and 15.2.15 existing.
On 11/17/21 2:39 AM, Eugen Block wrote:
Hi,
I retried it with your steps, they worked for me.
The first non-default realm runs on the default port 80 on two
RGWs, the second realm is on the same hosts on port 8081 as
configured in the spec file. The period commit ran successfully,
though, so maybe there's something wrong with your setup, it's hard
to tell.
Does the single realm config work as expected? The rgw container
starts successfully?
Zitat von J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx>:
I've rebuilt my rgw several times since our last emails and every
time I get stuck more or less to the same point, so let me share
with you all what I am doing step by step, along with the specs I
use.
1. I create a default RGW container and put it on my second
monitor. Default configs are set
ceph orch apply rgw staging staging --placement="1 ceph-monitor2"
2. I create a default realm
radosgw-admin realm create --rgw-realm=staging
3. I create a second realm
radosgw-admin realm create --rgw-realm=test
4. I create a master zonegroup for my second realm
radosgw-admin zonegroup create --rgw-zonegroup=test
--rgw-realm=test --realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef
5. I create a master zone for my second realm
radosgw-admin zone create --rgw-zonegroup=test --rgw-realm=test
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef --master
--rgw-zone=test
6. I try to commit my second realm (it fails, usually with a can't
init error)
radosgw-admin period update --commit
--realm-id=aa7d5bdd-afd7-449e-8f5b-c055877d7fef
7.I try to create my container with the following spec file
ceph orch apply -i multi-realm.spec
cat ceph orch apply -i multi-realm.spec
service_type: rgw
service_id: test.test
placement:
label: rgw-aux
count: 1
spec:
rgw_realm: test
rgw_zone: test
ssl: false
rgw_frontend_port: 9999
I've set Cephadm to output to a logfile and that logfile tells me:
Cannot find zone id=35ea43e5-e0d2-4ebe-81d6-9274231bd542 (name=test)
And that is even though the zone exists and is in the realm this
gateway is set to serve. Is the commit supposed to work before or
after the gateway is setup?
On 11/16/21 2:45 AM, Eugen Block wrote:
Hi,
Couldn't init storage provider (RADOS)
I usually see this when my rgw config is wrong, can you share
your rgw spec(s)?
Zitat von J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx>:
After searching through logs and figuring out how cephadm works,
I've figured out that when cephadm tries to create the new
systemd service and launch the new RGW container, it fails with
this error :
Couldn't init storage provider (RADOS)
Systemd then complains that the service doesn't exist :
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.twltpl.service: Main process exited, code=exited,
status=5/NOTINSTALLED
I can't seem to find further logs anywhere regarding why the
storage provider fails to init. Googling indicate issues with
the number of pgs in past versions, but I believe it shouldn't
be an issue here, since the amount of PG auto-adjusts. How could
I find out why this happens?
On 11/15/21 11:02 AM, J-P Methot wrote:
Yes, I'm trying to add a RGW container on a second port on the
same server. For example, I do :
ceph orch apply rgw test test
--placement="ceph-monitor1:[10.50.47.3:9999]"
and this results in :
ceph orch ls
NAME RUNNING REFRESHED AGE
PLACEMENT IMAGE
NAME IMAGE ID
rgw.test.test 0/1 2s ago 5s
ceph-monitor1:[10.50.47.3:9999] docker.io/ceph/ceph:v15 <unknown>
the image and container ID being unknown is making me scratch
my head. A look in the log files show this:
2021-11-15 10:50:12,253 INFO Deploy daemon
rgw.test.test.ceph-monitor1.rtoiwh ...
2021-11-15 10:50:12,254 DEBUG Running command: /usr/bin/docker
run --rm --ipc=host --net=host --entrypoint stat -e
CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e
NODE_NAME=ceph-monitor1 docker.io/ceph/ceph:v15 -c %u %g
/var/lib/ceph
2021-11-15 10:50:12,452 DEBUG stat: stdout 167 167
2021-11-15 10:50:12,525 DEBUG Running command: install -d
-m0770 -o 167 -g 167
/var/run/ceph/04c5d4a4-8815-45fb-b97f-027252d1aea5
2021-11-15 10:50:12,534 DEBUG Running command: systemctl daemon-reload
2021-11-15 10:50:12,869 DEBUG Running command: systemctl stop
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh 2021-11-15 10:50:12,879 DEBUG Running command: systemctl reset-failed
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh
2021-11-15 10:50:12,884 DEBUG systemctl: stderr Failed to reset
failed state of unit
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh.service: Unit ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh.service not
loaded
journalctl -xe shows the service entered failed state, without
any real useful information
Nov 15 10:50:24 ceph-monitor1 systemd[1]:
ceph-04c5d4a4-8815-45fb-b97f-027252d1aea5@xxxxxxxxxxxxxxxxxx-monitor1.rtoiwh.service: Failed with result
'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
What I understand from this is that I'm doing the right thing,
it's just my cephadm that's breaking, somehow.
On 11/15/21 5:59 AM, Eugen Block wrote:
Hi,
it's not entirely clear how your setup looks like, are you
trying to setup multiple RGW containers on the same host(s) to
serve multiple realms or do you have multiple RGWs for that?
You can add a second realm with a spec file or via cli (which
you already did). If you want to create multiple RGW
containers per host you need to specify a different port for
every RGW, see the docs [1] for some examples.
This worked just fine in my Octopus lab except for a little
mistake in the "port" spec, apparently this
spec:
port: 8000
doesn't work:
host1:~ # ceph orch apply -i rgw2.yaml
Error EINVAL: ServiceSpec: __init__() got an unexpected
keyword argument 'port'
But this does:
spec:
rgw_frontend_port: 8000
Now I have two RGW containers on each host, serving two
different realms.
[1] https://docs.ceph.com/en/latest/cephadm/services/rgw/
Zitat von J-P Methot <jp.methot@xxxxxxxxxxxxxxxxx>:
Hi,
I'm testing out adding a second RGW realm to my single ceph
cluster. This is not very well documented though, since
obviously realms were designed for multi-site deployments.
Now, what I can't seem to figure is if I need to deploy a
container with cephadm to act as a frontend for this second
realm and, if so, how? I've set a frontend port and address
when I created the second realm, but my attempts at creating
a RGW container for that realm didn't work at all, with the
container just not booting up.
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
