Thanks for the reply. This eventually resolved itself when I upgraded the client kernel from the Ubuntu Server 12.04.2 default to the 3.6.10 kernel. Not sure if there is a good causal explanation there or if it might be a coincidence. I did see the kernel recommendations in the docs but I had assumed those just applied to the Ceph machines and not clients - perhaps that is a bad assumption. Anyway, it works now, so I guess the next steps are to try moving the client back to the public network and to re-enable authentication and see if it works or if I still have an issue there. With regard to versions: ceph@cephtest06:/etc/ceph$ ceph-mon --version ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a) ceph@cephtest06:/etc/ceph$ uname -a Linux cephtest06 3.6.10-030610-generic #201212101650 SMP Mon Dec 10 21:51:40 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ceph@cephclient01:~/cos$ rados --version ceph version 0.67.3 (408cd61584c72c0d97b774b3d8f95c6b1b06341a) ceph@cephclient01:~/cos$ uname -a Linux cephclient01 3.6.10-030610-generic #201212101650 SMP Mon Dec 10 21:51:40 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux Thanks, Joe >-----Original Message----- >From: Gregory Farnum [mailto:greg@xxxxxxxxxxx] >Sent: Monday, October 07, 2013 1:27 PM >To: Gruher, Joseph R >Cc: ceph-users@xxxxxxxxxxxxxx >Subject: Re: Client Timeout on Rados Gateway > >The ping tests you're running are connecting to different interfaces >(10.23.37.175) than those you specify in the "mon_hosts" option (10.0.0.2, >10.0.0.3, 10.0.0.4). The client needs to be able to connect to the specified >address; I'm guessing it's not routable from outside that network? > >The error you're getting once you put it inside the network is more >interesting. What version of the Ceph packages do you have installed there, >and what's installed on the monitors? (run "ceph-mon --version" >on the monitor, and "rados --version" on the client, and it'll >output.) >-Greg >Software Engineer #42 @ http://inktank.com | http://ceph.com > >On Tue, Oct 1, 2013 at 12:45 PM, Gruher, Joseph R ><joseph.r.gruher@xxxxxxxxx> wrote: >> Hello- >> >> >> >> I've set up a rados gateway but I'm having trouble accessing it from >> clients. I can access it using rados command line just fine from any >> system in my ceph deployment, including my monitors and OSDs, the >> gateway system, and even the admin system I used to run ceph-deploy. >> However, when I set up a client outside the ceph nodes I get a timeout >> error as shown at the bottom of the output pasted below. I've turned >> off authentication for the moment to simplify things. Systems are >> able to resolve names and reach each other via ping. Any thoughts on what >could be the issue here or how to debug? >> >> >> >> The failure: >> >> >> >> ceph@cephclient01:/etc/ceph$ rados df >> >> 2013-10-01 19:57:07.488970 7fd381db0780 monclient(hunting): >> authenticate timed out after 30 >> >> 2013-10-01 19:57:07.489174 7fd381db0780 librados: client.admin >> authentication error (110) Connection timed out >> >> couldn't connect to cluster! error -110 >> >> >> >> >> >> ceph@cephclient01:/etc/ceph$ sudo rados df >> >> 2013-10-01 19:57:44.461273 7fb6712d5780 monclient(hunting): >> authenticate timed out after 30 >> >> 2013-10-01 19:57:44.461440 7fb6712d5780 librados: client.admin >> authentication error (110) Connection timed out >> >> couldn't connect to cluster! error -110 >> >> ceph@cephclient01:/etc/ceph$ >> >> >> >> >> >> Some details from the client: >> >> >> >> ceph@cephclient01:/etc/ceph$ pwd >> >> /etc/ceph >> >> >> >> >> >> ceph@cephclient01:/etc/ceph$ ls >> >> ceph.client.admin.keyring ceph.conf keyring.radosgw.gateway >> >> >> >> >> >> ceph@cephclient01:/etc/ceph$ cat ceph.conf >> >> [global] >> >> fsid = a45e6e54-70ef-4470-91db-2152965deec5 >> >> mon_initial_members = cephtest02, cephtest03, cephtest04 >> >> mon_host = 10.0.0.2,10.0.0.3,10.0.0.4 >> >> osd_journal_size = 1024 >> >> filestore_xattr_use_omap = true >> >> auth_cluster_required = none #cephx >> >> auth_service_required = none #cephx >> >> auth_client_required = none #cephx >> >> >> >> [client.radosgw.gateway] >> >> host = cephtest06 >> >> keyring = /etc/ceph/keyring.radosgw.gateway >> >> rgw_socket_path = /tmp/radosgw.sock >> >> log_file = /var/log/ceph/radosgw.log >> >> >> >> >> >> ceph@cephclient01:/etc/ceph$ ping cephtest06 >> >> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data. >> >> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=1 >> ttl=64 >> time=0.216 ms >> >> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=2 >> ttl=64 >> time=0.209 ms >> >> ^C >> >> --- cephtest06.jf.intel.com ping statistics --- >> >> 2 packets transmitted, 2 received, 0% packet loss, time 999ms >> >> rtt min/avg/max/mdev = 0.209/0.212/0.216/0.015 ms >> >> >> >> >> >> ceph@cephclient01:/etc/ceph$ ping cephtest06.jf.intel.com >> >> PING cephtest06.jf.intel.com (10.23.37.175) 56(84) bytes of data. >> >> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=1 >> ttl=64 >> time=0.223 ms >> >> 64 bytes from cephtest06.jf.intel.com (10.23.37.175): icmp_req=2 >> ttl=64 >> time=0.242 ms >> >> ^C >> >> --- cephtest06.jf.intel.com ping statistics --- >> >> 2 packets transmitted, 2 received, 0% packet loss, time 999ms >> >> rtt min/avg/max/mdev = 0.223/0.232/0.242/0.017 ms >> >> >> >> >> >> I did try putting the client on the 10.0.0.x network to see if that >> would affect behavior but that just seemed to introduce a new problem: >> >> >> >> ceph@cephclient01:/etc/ceph$ rados df >> >> 2013-10-01 21:37:29.439410 7f60d2a43700 failed to decode message of >> type 59 >> v1: buffer::end_of_buffer >> >> 2013-10-01 21:37:29.439583 7f60d4a47700 monclient: hunting for new mon >> >> >> >> ceph@cephclient01:/etc/ceph$ ceph -m 10.0.0.2 -s >> >> 2013-10-01 21:37:42.341480 7f61eacd5700 monclient: hunting for new mon >> >> 2013-10-01 21:37:45.341024 7f61eacd5700 monclient: hunting for new mon >> >> 2013-10-01 21:37:45.343274 7f61eacd5700 monclient: hunting for new mon >> >> >> >> ceph@cephclient01:/etc/ceph$ ceph health >> >> 2013-10-01 21:39:52.833560 mon <- [health] >> >> 2013-10-01 21:39:52.834671 mon.0 -> 'unparseable JSON health' (-22) >> >> ceph@cephclient01:/etc/ceph$ >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |