Re: Having problem to start Radosgw

B L <super.iterator@xxxxxxxxx> · Sun, 15 Feb 2015 00:56:54 +0200

Yehuda .. 

In case you will need to know more about my system

Here is my full cluster configuration: https://gist.github.com/anonymous/fb4c314320d7df75569a

And, that’s my ceph cluster status:

$ ceph -s

    cluster 17bea68b-1634-4cd1-8b2a-00a60ef4761d
     health HEALTH_WARN 203 pgs degraded; 203 pgs stuck unclean; recovery 6/151 objects degraded (3.974%)
     monmap e1: 1 mons at {ceph-node1=172.31.0.84:6789/0}, election epoch 2, quorum 0 ceph-node1
     osdmap e93: 6 osds: 6 up, 6 in
      pgmap v3676: 1920 pgs, 16 pools, 10241 kB data, 51 objects
            279 MB used, 18086 MB / 18365 MB avail
            6/151 objects degraded (3.974%)
                 203 active+degraded
                1717 active+clean

It was fully healthy before adding the radosgw pools .. yet, I still can put objects to the cluster (without using RGW)

Best!

On Feb 15, 2015, at 12:39 AM, B L <super.iterator@xxxxxxxxx> wrote:

That’s what I usually do to check if rgw is running with no problems: sudo radosgw -c ceph.conf -d

I already pumped up the log level, but I can’t see any change or verbosity level increase of the logs, I still get the same:

2015-02-14 22:27:57.513151 7f26c79d27c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 7924
2015-02-14 22:27:57.573564 7f26c79d27c0  0 framework: fastcgi
2015-02-14 22:27:57.573569 7f26c79d27c0  0 starting handler: fastcgi
2015-02-14 22:27:57.575349 7f269affd700  0 ERROR: FCGX_Accept_r returned -9
2015-02-14 22:27:57.670610 7f269bfff700  0 ERROR: can't read user header: ret=-2
2015-02-14 22:27:57.670613 7f269bfff700  0 ERROR: sync_user() failed, user=cephtest ret=-2
2015-02-14 22:27:57.671382 7f269bfff700  0 ERROR: can't read user header: ret=-2
2015-02-14 22:27:57.671384 7f269bfff700  0 ERROR: sync_user() failed, user=cephtestss ret=-2
^C2015-02-14 22:28:30.693140 7f269b7fe700  1 handle_sigterm
2015-02-14 22:28:30.693170 7f269b7fe700  1 handle_sigterm set alarm for 120
2015-02-14 22:28:30.693179 7f26c79d27c0 -1 shutting down
2015-02-14 22:28:30.717340 7f26c79d27c0  1 final shutdown

Please let me know if I can do something more ..

Now I have 2 questions:
1- what RADOS user you refer to?
2- How would I know that I use wrong cephx keys unless I see authentication error or relevant warning?

Thanks!
Beanos

On Feb 14, 2015, at 11:29 PM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:

From: "B L" <super.iterator@xxxxxxxxx>
To: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Sent: Saturday, February 14, 2015 11:03:42 AM
Subject: Re:  Having problem to start Radosgw

Hello Yehyda,

The strace command you referred to me, shows this: https://gist.github.com/anonymous/8e9f1ced485996a263bb

Additionally, I traced this log file: /var/log/radosgw/ceph-client.radosgw.gateway

it has the following:

2015-02-12 18:23:32.247679 7fecca5257c0 -1 did not load config file, using default settings.
2015-02-12 18:23:32.247745 7fecca5257c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 20477
2015-02-12 18:23:32.251192 7fecca5257c0 -1 Couldn't init storage provider (RADOS)
2015-02-12 18:23:58.494026 7faab31377c0 -1 did not load config file, using default settings.
2015-02-12 18:23:58.494092 7faab31377c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 20509
2015-02-12 18:23:58.497420 7faab31377c0 -1 Couldn't init storage provider (RADOS)
2015-02-14 17:13:03.478688 7f86f09567c0 -1 did not load config file, using default settings.
2015-02-14 17:13:03.478778 7f86f09567c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 2989
2015-02-14 17:13:03.482850 7f86f09567c0 -1 Couldn't init storage provider (RADOS)
2015-02-14 17:13:29.477530 7ff18226a7c0 -1 did not load config file, using default settings.
2015-02-14 17:13:29.477595 7ff18226a7c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 3033
2015-02-14 17:13:29.481173 7ff18226a7c0 -1 Couldn't init storage provider (RADOS)
2015-02-14 17:21:00.950847 7ffee3a3b7c0 -1 did not load config file, using default settings.
2015-02-14 17:21:00.950916 7ffee3a3b7c0  0 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3), process radosgw, pid 3086
2015-02-14 17:21:00.954085 7ffee3a3b7c0 -1 Couldn't init storage provider (RADOS)

Turns out to be that the last line of the logs is thrown out by this piece of code in rgw_main.cc:

…
…

  FCGX_Init();

  RGWStoreManager store_manager;

  if (!store_manager.init("rados", g_ceph_context)) {
    derr << "Couldn't init storage provider (RADOS)" << dendl;  
    return EIO;
  }

  RGWProcess process(g_ceph_context, 20);

  process.run();

  return 0;

N.B.  you can find it in:(http://workbench.dachary.org/ceph/ceph/raw/8d63e140777bbdd061baa6845d57e6c3cc771f76/src/rgw/rgw_main.cc) , 10th line from below.

Is that by any means related to the problem?

Not related. This actually means that it couldn't connect to the rados backend, so there's a different issue now. The strace log doesn't provide much with regard to the original issue as it didn't get to that part now. You can try bumping up the debug level (debug rgw = 20, debug ms = 1). I assume that the issue that you're seeing is that the wrong rados user and/or wrong cephx keys are being used. Try to run it again as you do usually, and see what the regular params that are being passed when starting radosgw; use these when running the strace command.

Yehuda

On Feb 14, 2015, at 7:24 PM, Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:

sudo strace -F -T -tt -o/tmp/strace.out radosgw -c ceph.conf -f

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com