2011/10/27 Josef Bacik <josef@xxxxxxxxxx>: > On Tue, Oct 25, 2011 at 01:56:48PM +0200, Christian Brunner wrote: >> 2011/10/24 Josef Bacik <josef@xxxxxxxxxx>: >> > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> >> [adding linux-btrfs to cc] >> >> >> >> Josef, Chris, any ideas on the below issues? >> >> >> >> On Mon, 24 Oct 2011, Christian Brunner wrote: >> >> > >> >> > - When I run ceph with btrfs snaps disabled, the situation is getting >> >> > slightly better. I can run an OSD for about 3 days without problems, >> >> > but then again the load increases. This time, I can see that the >> >> > ceph-osd (blkdev_issue_flush) and btrfs-endio-wri are doing more work >> >> > than usual. >> >> >> >> FYI in this scenario you're exposed to the same journal replay issues that >> >> ext4 and XFS are. The btrfs workload that ceph is generating will also >> >> not be all that special, though, so this problem shouldn't be unique to >> >> ceph. >> >> >> > >> > Can you get sysrq+w when this happens? I'd like to see what btrfs-endio-write >> > is up to. >> >> Capturing this seems to be not easy. I have a few traces (see >> attachment), but with sysrq+w I do not get a stacktrace of >> btrfs-endio-write. What I have is a "latencytop -c" output which is >> interesting: >> >> In our Ceph-OSD server we have 4 disks with 4 btrfs filesystems. Ceph >> tries to balance the load over all OSDs, so all filesystems should get >> an nearly equal load. At the moment one filesystem seems to have a >> problem. When running with iostat I see the following >> >> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s >> avgrq-sz avgqu-sz await svctm %util >> sdd 0.00 0.00 0.00 4.33 0.00 53.33 >> 12.31 0.08 19.38 12.23 5.30 >> sdc 0.00 1.00 0.00 228.33 0.00 1957.33 >> 8.57 74.33 380.76 2.74 62.57 >> sdb 0.00 0.00 0.00 1.33 0.00 16.00 >> 12.00 0.03 25.00 19.75 2.63 >> sda 0.00 0.00 0.00 0.67 0.00 8.00 >> 12.00 0.01 19.50 12.50 0.83 >> >> The PID of the ceph-osd taht is running on sdc is 2053 and when I look >> with top I see this process and a btrfs-endio-writer (PID 5447): >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 2053 root 20 0 537m 146m 2364 S 33.2 0.6 43:31.24 ceph-osd >> 5447 root 20 0 0 0 0 S 22.6 0.0 19:32.18 btrfs-endio-wri >> >> In the latencytop output you can see that those processes have a much >> higher latency, than the other ceph-osd and btrfs-endio-writers. >> >> Regards, >> Christian > > Ok just a shot in the dark, but could you give this a whirl and see if it helps > you? Thanks Thanks for the patch! I'll install it tomorrow and I think that I can report back on Monday. It always takes a few days until the load goes up. Regards, Christian -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
