thanks for the details. this is a regression from changes to the datalog storage for multisite - this -5 error is coming from the new 'fifo' backend. as a workaround, you can set the new 'rgw_data_log_backing' config variable back to 'omap' Adam has fixes already merged to the pacific branch; be aware that the first pacific point release will change the name of 'rgw_data_log_backing' to 'rgw_default_data_log_backing' and default back to 'fifo' On Tue, Apr 6, 2021 at 2:37 AM Martin Verges <martin.verges@xxxxxxxx> wrote: > > Hello, > > we see same problems. Deleting all the pools and redeploy rgw solved it on > that test cluster, however that is no solution for production ;) > > systemd[1]: Started Ceph rados gateway. > radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 deferred set > uid:gid to 167:167 (ceph:ceph) > radosgw[7171]: failed to chown /dev/null: (30) Read-only file system > radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 ceph version > 16.2.0-31-g5922b2b9c1 (5922b2b9c17f0877f84b0b3f2557ab72a628cbfe) pacific > (stable), process radosgw, pid 7171 > radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework: > beast > radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf > key: ssl_port, val: 443 > radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf > key: port, val: 80 > radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 0 framework conf > key: ssl_certificate, val: /etc/ceph/rgwcert.pem > radosgw[7171]: 2021-04-04T14:37:51.508+0000 7fc6641efc00 1 radosgw_Main > not setting numa affinity > radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int > rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, > std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, > optional_yield, std::optional<rados::cls::fifo::objv>, > std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 > create_meta failed: r=-5 > radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 static int > rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, > std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, > optional_yield, std::optional<rados::cls::fifo::objv>, > std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 > create_meta failed: r=-5 > radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int > RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, > librados::v14_2_0::Rados*): Error when starting backend: Input/output error > radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed > to start datalog_rados service ((5) Input/output error > radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 -1 int > RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, > librados::v14_2_0::Rados*): Error when starting backend: Input/output error > radosgw[7171]: 2021-04-04T14:37:51.680+0000 7fc6641efc00 0 ERROR: failed > to init services (ret=(5) Input/output error) > radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init > storage provider (RADOS) > radosgw[7171]: 2021-04-04T14:37:51.700+0000 7fc6641efc00 -1 Couldn't init > storage provider (RADOS) > systemd[1]: ceph-radosgw@xxxxxxx-croit-host-C0DE01.service: Main process > exited, code=exited, status=5/NOTINSTALLED > systemd[1]: ceph-radosgw@xxxxxxx-croit-host-C0DE01.service: Unit entered > failed state. > systemd[1]: ceph-radosgw@xxxxxxx-croit-host-C0DE01.service: Failed with > result 'exit-code'. > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges@xxxxxxxx > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > -- > Martin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.verges@xxxxxxxx > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > > Web: https://croit.io > YouTube: https://goo.gl/PGE1Bx > > > On Mon, 5 Apr 2021 at 19:59, Robert Sander <r.sander@xxxxxxxxxxxxxxxxxxx> > wrote: > > > Hi, > > > > Am 04.04.21 um 15:22 schrieb 胡 玮文: > > > > > bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static > > int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, > > std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, > > optional_yield, std::optional<rados::cls::fifo::objv>, > > std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 > > create_meta failed: r=-5 > > > bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int > > RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, > > librados::v14_2_0::Rados*): Error when starting backend: Input/output error > > > bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: > > failed to start datalog_rados service ((5) Input/output error > > > bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: > > failed to init services (ret=(5) Input/output error) > > > > I see the same issues on an upgraded cluster. > > > > Regards > > -- > > Robert Sander > > Heinlein Consulting GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > http://www.heinlein-support.de > > > > Tel: 030 / 405051-43 > > Fax: 030 / 405051-19 > > > > Zwangsangaben lt. §35a GmbHG: > > HRB 93818 B / Amtsgericht Berlin-Charlottenburg, > > Geschäftsführer: Peer Heinlein -- Sitz: Berlin > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |