On 02/16/2014 09:22 PM, Sage Weil wrote:
Hi Wido,
On Sun, 16 Feb 2014, Wido den Hollander wrote:
On 02/16/2014 06:49 PM, Gregory Farnum wrote:
Did you maybe upgrade that box to v0.67.6? This sounds like one of the
bugs Sage mentioned in it.
No, I checked it again. Version is: ceph version 0.67.5
(a60ac9194718083a4b6a225fc17cad6096c69bd1)
All machines in the cluster are on that version.
Are you sure none of the running ceph-osd processes aren't 0.67.6? Maybe
check 'ceph daemon osd.NNN version'...
I double-verified it again, but they are all running 0.67.5
Since for example osd.25 is down right now I can't run 'ceph daemon',
but the md5sum of /usr/bin/ceph-osd is the same as on the other machines
which are all on 0.67.5
Auto updates with Apt are not enabled, so there is no way these machines
could be running 0.67.6
So I'm still confused.
Wido
sage
Wido
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
On Sun, Feb 16, 2014 at 4:23 AM, Wido den Hollander <wido@xxxxxxxx> wrote:
Hi,
Yesterday I got a notification that a RGW setup was having issues with
objects suddenly giving errors (403 and 404) when trying to access them.
I started digging and after cranking up the logs with 'debug rados' and
'debug rgw' set to 20 I found what caused RGW to throw a error:
librados: Objecter returned from getxattrs r=-2
Using "ceph osd map .rgw.buckets <object>" I found which OSDs were primary
for that object's PG and I saw that they all came from one machine which
got
a clean shutdown and start just 24 hours before that.
After taking that machine out of production the other OSDs took over and
RGW
started serving the objects again, but I'm confused.
The underlying filesystem is XFS and all 6 filesystems were clean and
healthy. Like I said, the machine only got a clean shutdown 24 hours
before
that due to a physical migration, but that's all.
Did anybody see this before? Suddenly the xattrs for those objects were
gone.
This was with Ceph 0.67.5
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
