2011/10/24 Josef Bacik <josef@xxxxxxxxxx>: > On Mon, Oct 24, 2011 at 10:06:49AM -0700, Sage Weil wrote: >> [adding linux-btrfs to cc] >> >> Josef, Chris, any ideas on the below issues? >> >> On Mon, 24 Oct 2011, Christian Brunner wrote: >> > >> > - When I run ceph with btrfs snaps disabled, the situation is getting >> > slightly better. I can run an OSD for about 3 days without problems, >> > but then again the load increases. This time, I can see that the >> > ceph-osd (blkdev_issue_flush) and btrfs-endio-wri are doing more work >> > than usual. >> >> FYI in this scenario you're exposed to the same journal replay issues that >> ext4 and XFS are. The btrfs workload that ceph is generating will also >> not be all that special, though, so this problem shouldn't be unique to >> ceph. >> > > Can you get sysrq+w when this happens? I'd like to see what btrfs-endio-write > is up to. Capturing this seems to be not easy. I have a few traces (see attachment), but with sysrq+w I do not get a stacktrace of btrfs-endio-write. What I have is a "latencytop -c" output which is interesting: In our Ceph-OSD server we have 4 disks with 4 btrfs filesystems. Ceph tries to balance the load over all OSDs, so all filesystems should get an nearly equal load. At the moment one filesystem seems to have a problem. When running with iostat I see the following Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sdd 0.00 0.00 0.00 4.33 0.00 53.33 12.31 0.08 19.38 12.23 5.30 sdc 0.00 1.00 0.00 228.33 0.00 1957.33 8.57 74.33 380.76 2.74 62.57 sdb 0.00 0.00 0.00 1.33 0.00 16.00 12.00 0.03 25.00 19.75 2.63 sda 0.00 0.00 0.00 0.67 0.00 8.00 12.00 0.01 19.50 12.50 0.83 The PID of the ceph-osd taht is running on sdc is 2053 and when I look with top I see this process and a btrfs-endio-writer (PID 5447): PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2053 root 20 0 537m 146m 2364 S 33.2 0.6 43:31.24 ceph-osd 5447 root 20 0 0 0 0 S 22.6 0.0 19:32.18 btrfs-endio-wri In the latencytop output you can see that those processes have a much higher latency, than the other ceph-osd and btrfs-endio-writers. Regards, Christian
Attachment:
dmesg.txt.bz2
Description: BZip2 compressed data
Attachment:
latencytop.txt.bz2
Description: BZip2 compressed data
