On Sun, 31 Jul 2011, Fyodor Ustinov wrote:
> On 07/31/2011 07:54 AM, Sage Weil wrote:
> > On Sat, 30 Jul 2011, Ted Ts'o wrote:
> > > On Sat, Jul 30, 2011 at 10:21:13AM -0700, Sage Weil wrote:
> > > > We do use xattrs extensively, though; that was the last extN bug we
> > > > uncovered. That's where my money is.
> > > Hmm, yes. That could very well be. How big are the xattrs, and are
> > > there cases where you:
> > >
> > > a) start with a small xattr (where the total size is less than 128
> > > bytes, so it can be stored in the inode table), and then increase it
> > > something where it needs to be stored in an external block?
> > >
> > > b) start with enough xattrs so it's large, and then delete all or most
> > > of them?
> > >
> > > I could easily believe we might have some bugs as we transition from
> > > in-inode to external block storage, or vice versa. I'll take a look
> > > at the code and try to create some reproduction cases, but if you
> > > could give me a handle on workload patterns of ceph around xattrs,
> > > that would be interesting.
> > I would guess a, but it could also be a+b.
> >
> > Fyodor, can you take some of the corrupt inos that fsck complained about
> > and see what files/directories they are? find /osd.0 -inum NNN. (I'm
> > guessing the largest xattrs are on the collection directories, like
> > /osd.0/current/something_head/.) Then grep that filename out of the log
> > to see exactly which operations took place. The setattr log normally
> > includes xattr size.
>
>
> /etc/init.d/ceph stop
> umount /mnt/osd.0
> mke2fs -t ext4 -I 128 /dev/sdc1
> tune2fs -o journal_data_writeback /dev/sdc1
> mount -a
> mon getmap -o /tmp/monmap
> cosd --mkfs -i 0 --monmap /tmp/monmap
> /etc/init.d/ceph start
> sleep 300
> /etc/init.d/ceph stop
> umount /osd.0
> fsck.ext4 -f /dev/sdc1
>
> Inode 99356878, i_blocks is 8208, should be 8200.
>
> mount -a
> root@osd0:~# find /osd.0 -inum 99356878
>
> /osd.0/current/0.2a4_head/10000000468.0000007e_head
>
> root@osd0:~# grep "10000000468\.0000007e" /var/log/ceph/osd.0.log
> 2011-07-31 09:57:20.859834 7f624c82a700 filestore(/osd.0) remove
> temp/10000000468.0000007e/head = -1
> 2011-07-31 09:57:20.861166 7f624c82a700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 0~1048576 = 1048576
> 2011-07-31 09:57:20.990464 7f624c029700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 1048576~1048576 = 1048576
> 2011-07-31 09:57:21.121648 7f624c029700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 2097152~1048576 = 1048576
> 2011-07-31 09:57:21.265879 7f624c029700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 3145728~1048576 = 1048576
> 2011-07-31 09:57:21.265952 7f624c029700 filestore(/osd.0) remove
> 0.2a4_head/10000000468.0000007e/head = -1
> 2011-07-31 09:57:21.265995 7f624c029700 filestore(/osd.0) collection_add
> 0.2a4_head/10000000468.0000007e/head temp/10000000468.0000007e/head = 0
> 2011-07-31 09:57:21.266025 7f624c029700 filestore(/osd.0) collection_remove
> temp/10000000468.0000007e/head = 0
> 2011-07-31 09:57:21.266134 7f624c029700 filestore(/osd.0) setattrs
> 0.2a4_head/10000000468.0000007e/head = 26
Hrm, I was hoping it wouldn't be a setattrs call. The below will tell us
what xattrs it is setting, but sadly you'll need to reproduce the whole
thing again. The above is only telling us the size of the last xattr of
the set.
Ted, I'm assuming the i_blocks mismatch is likely on the same files that
make the transition from/to in-inode xattrs?
sage
diff --git a/src/os/FileStore.cc b/src/os/FileStore.cc
index 8bdee6b..fb00cff 100644
--- a/src/os/FileStore.cc
+++ b/src/os/FileStore.cc
@@ -3570,6 +3570,7 @@ int FileStore::_setattrs(coll_t cid, const sobject_t& oid, map<string,bufferptr>
val = "";
// ??? Why do we skip setting all the other attrs if one fails?
r = lfn_setxattr(cid, oid, n, val, p->second.length());
+ dout(10) << "setattrs " << cid << "/" << oid << " '" << p->first << "' len " << p->second.length() << " = " << r << dendl;
if (r < 0) {
derr << "FileStore::_setattrs: do_setxattr returned " << r << dendl;
break;
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
