On 07/31/2011 07:54 AM, Sage Weil wrote:
On Sat, 30 Jul 2011, Ted Ts'o wrote:
On Sat, Jul 30, 2011 at 10:21:13AM -0700, Sage Weil wrote:
We do use xattrs extensively, though; that was the last extN bug we
uncovered. That's where my money is.
Hmm, yes. That could very well be. How big are the xattrs, and are
there cases where you:
a) start with a small xattr (where the total size is less than 128
bytes, so it can be stored in the inode table), and then increase it
something where it needs to be stored in an external block?
b) start with enough xattrs so it's large, and then delete all or most
of them?
I could easily believe we might have some bugs as we transition from
in-inode to external block storage, or vice versa. I'll take a look
at the code and try to create some reproduction cases, but if you
could give me a handle on workload patterns of ceph around xattrs,
that would be interesting.
I would guess a, but it could also be a+b.
Fyodor, can you take some of the corrupt inos that fsck complained about
and see what files/directories they are? find /osd.0 -inum NNN. (I'm
guessing the largest xattrs are on the collection directories, like
/osd.0/current/something_head/.) Then grep that filename out of the log
to see exactly which operations took place. The setattr log normally
includes xattr size.
/etc/init.d/ceph stop
umount /mnt/osd.0
mke2fs -t ext4 -I 128 /dev/sdc1
tune2fs -o journal_data_writeback /dev/sdc1
mount -a
mon getmap -o /tmp/monmap
cosd --mkfs -i 0 --monmap /tmp/monmap
/etc/init.d/ceph start
sleep 300
/etc/init.d/ceph stop
umount /osd.0
fsck.ext4 -f /dev/sdc1
Inode 99356878, i_blocks is 8208, should be 8200.
mount -a
root@osd0:~# find /osd.0 -inum 99356878
/osd.0/current/0.2a4_head/10000000468.0000007e_head
root@osd0:~# grep "10000000468\.0000007e" /var/log/ceph/osd.0.log
2011-07-31 09:57:20.859834 7f624c82a700 filestore(/osd.0) remove
temp/10000000468.0000007e/head = -1
2011-07-31 09:57:20.861166 7f624c82a700 filestore(/osd.0) write
temp/10000000468.0000007e/head 0~1048576 = 1048576
2011-07-31 09:57:20.990464 7f624c029700 filestore(/osd.0) write
temp/10000000468.0000007e/head 1048576~1048576 = 1048576
2011-07-31 09:57:21.121648 7f624c029700 filestore(/osd.0) write
temp/10000000468.0000007e/head 2097152~1048576 = 1048576
2011-07-31 09:57:21.265879 7f624c029700 filestore(/osd.0) write
temp/10000000468.0000007e/head 3145728~1048576 = 1048576
2011-07-31 09:57:21.265952 7f624c029700 filestore(/osd.0) remove
0.2a4_head/10000000468.0000007e/head = -1
2011-07-31 09:57:21.265995 7f624c029700 filestore(/osd.0) collection_add
0.2a4_head/10000000468.0000007e/head temp/10000000468.0000007e/head = 0
2011-07-31 09:57:21.266025 7f624c029700 filestore(/osd.0)
collection_remove temp/10000000468.0000007e/head = 0
2011-07-31 09:57:21.266134 7f624c029700 filestore(/osd.0) setattrs
0.2a4_head/10000000468.0000007e/head = 26
WBR,
Fyodor.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
