Looks like the OSD didn't like an error return it got from the
underlying fs. Can you reproduce with
debug filestore = 20
debug osd = 20
debug ms = 1
on the osd and post the whole log?
-Sam
On Wed, Sep 21, 2016 at 12:10 AM, Peter Maloney
<peter.maloney@xxxxxxxxxxxxxxxxxxxx> wrote:
> Hi,
>
> I created a one disk osd with data and separate journal on the same lvm
> volume group just for test, one mon, one mds on my desktop.
>
> I managed to crash the osd just by mounting cephfs and doing cp -a of
> the linux-stable git tree into it. It crashed after copying 2.1G which
> only covers some of the .git dir and none of the rest. And then when I
> killed ceph-mds and restarted the osd and mds, ceph -s said something
> about the pgs being stuck or unclean or something, and the computer
> froze. :/ After booting again, everything is fine, and the problem was
> reproducable the same way...just copying the files again.[but after
> writing this mail, I can't seem to cause it as easily again... copying
> again works, but sha1sum doesn't, even if I drop caches]
>
> Also reading seems to do the same.
>
> And then I tried adding a 2nd osd (also from vlm, with osd and journal
> on same volume group). And that seemed to stop the crashing, but not
> sure about corruption.I guess the corruption was on the cephfs but RAM
> had good copies or something, so rebooting, etc. is what made the
> corruption appear? (I tried to reproduce, but couldn't...didn't try
> killing daemons)
>
>> root@client:/mnt/test # ls -l
>> total 447
>> drwx------ 1 root root 4 2016-09-20 20:37 1/
>> drwx------ 1 root root 4 2016-09-20 20:37 2/
>> drwx------ 1 root root 4 2016-09-20 20:37 linux-stable/
>> -rw-r--r-- 1 root root 457480 2016-09-20 21:38 sums.txt
>> root@client:/mnt/test # (cd linux-stable/; sha1sum -c --quiet
>> ../sums.txt )
> (osd crashed before that finished ... and then impressively, starting
> the osd again made the command finish gracefully... and then tried rsync
> to finish copying and 6 or so restarts later it finished with just the 1
> rsync run)
>
> And then the checksums didn't match... (corruption)
>> root@client:/mnt/test # (cd linux-stable/; sha1sum -c --quiet
>> ../sums.txt )
>> ./.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd: FAILED
>> ./.git/objects/e6/d58d90213a4a283d428988a398281663dd68e4: FAILED
>> ./.git/objects/81/281381965b21d3c23b2f877e214c4af65d6fa4: FAILED
>> ./.git/objects/4c/f549c4a9b23638ab49cc0f8b47c395b1fc8ede: FAILED
>> sha1sum: WARNING: 4 computed checksums did NOT match
>
>> root@client:/mnt/test # hexdump -C
>> linux-stable/.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd
>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> *
>> 00000dd0 00 00 00 |...|
>> 00000dd3
>> peter@peter:~/projects $ hexdump -C
>> linux-stable/.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd | head
>> 00000000 78 01 bd 5b 7d 6f db c6 19 df bf d4 a7 38 20 40
>> |x..[}o.......8 @|
>> 00000010 27 0b 8e ec 14 2b 06 24 5d 90 34 75 5c 63 4e 6c
>> |'....+.$].4u\cNl|
>> 00000020 d8 f1 82 62 19 08 9a 3a 59 ac 29 52 25 29 bb 5e
>> |...b...:Y.)R%).^|
>> 00000030 9a ef be df f3 dc 1d 79 c7 77 39 c1 84 c0 12 79
>> |.......y.w9....y|
>> 00000040 77 cf fb 3b 99 eb 38 bd 16 cf 7e 38 fc e1 f0 2f
>> |w..;..8...~8.../|
>> 00000050 07 b3 89 98 89 fc 21 5f e6 f3 95 78 2a 16 72 19
>> |......!_...x*.r.|
>> 00000060 25 51 11 a5 49 2e 96 69 26 8a 95 c4 bd bb 28 c4
>> |%Q..I..i&.....(.|
>> 00000070 57 16 dd c9 4c 2c a3 58 62 7f 21 d7 38 49 87 df
>> |W...L,.Xb.!.8I..|
>> 00000080 a4 9b 87 2c ba 59 15 62 1a ee 89 ef 0f 0f 9f ed
>> |...,.Y.b........|
>> 00000090 e3 cf f7 e2 3c 28 b2 28 bc 15 ef d2 70 25 e3 d6
>> |....<(.(....p%..|
> and then copying that and testing checksums again has even more failures
>
>> root@client:/mnt/test # cp -a linux-stable 3
>> root@client:/mnt/test # (cd 3/; sha1sum -c --quiet ../sums.txt )
>> ./net/iucv/iucv.c: FAILED
>> ./net/kcm/kcmsock.c: FAILED
>> ./net/irda/ircomm/ircomm_event.c: FAILED
>> ./net/irda/ircomm/ircomm_tty_attach.c: FAILED
>> ./net/llc/Makefile: FAILED
>> ./net/llc/Kconfig: FAILED
>> ./net/llc/af_llc.c: FAILED
>> ./net/lapb/lapb_timer.c: FAILED
>> ./net/lapb/lapb_subr.c: FAILED
>> ./net/lapb/lapb_iface.c: FAILED
>> ./net/lapb/Makefile: FAILED
>> ./net/lapb/lapb_in.c: FAILED
>> ./net/lapb/lapb_out.c: FAILED
>> ./net/l2tp/l2tp_eth.c: FAILED
>> ./net/l2tp/Kconfig: FAILED
>> ./net/l2tp/l2tp_core.h: FAILED
>> ./.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd: FAILED
>> ./.git/objects/e6/d58d90213a4a283d428988a398281663dd68e4: FAILED
>> ./.git/objects/81/281381965b21d3c23b2f877e214c4af65d6fa4: FAILED
>> ./.git/objects/4c/f549c4a9b23638ab49cc0f8b47c395b1fc8ede: FAILED
>> sha1sum: WARNING: 20 computed checksums did NOT match
>> root@client:/mnt/test # hexdump -C 3/net/iucv/iucv.c
>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> *
>> 0000d420 00 00 00 00 00 00 00 00 |........|
>> 0000d428
>> root@client:/mnt/test # hexdump -C 3/net/kcm/kcmsock.c
>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> *
>> 0000d010 00 00 00 00 00 00 00 |.......|
>> 0000d017
>> root@client:/mnt/test # hexdump -C 3/net/lapb/Makefile
>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |................|
>> *
>> 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00
>> |.............|
>> 0000008d
>
> And then I thought it couldn't possibly be only me with such an obvious
> bug, so it must be that I'm doing something differently... nobody would
> run only 1 osd, right? So I added a 2nd on another disk, with separate
> journal on same disk, just like the other...
>
>> root@client:/mnt/test # cp -a linux-stable 4
> no crashes this time...
>> root@client:/mnt/test # (cd 4/; sha1sum -c --quiet ../sums.txt )
> same failures as 3/ ... strange
>> root@client:/mnt/test # (cd linux-stable/; sha1sum -c --quiet
>> ../sums.txt )
> same failures as 3/... but this only had 4 failures before... even more
> strange... reading files corrupts them too? or did they corrupt some
> other way?
>> root@client:/mnt/test # (cd 1/; sha1sum -c --quiet ../sums.txt )
> same failures as before, just the 4 files...so this one is fine (copied
> from one with 4 failures, so as fine as can be)
>
>> root@peter:~ # ceph --version
>> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
>> root@peter:~ # uname -a
>> Linux peter 4.6.4-1-grsec-kvm-host #22 SMP PREEMPT Tue Jul 26 15:12:19
>> CEST 2016 x86_64 GNU/Linux
>
> Here's the osd log for the first crash... (which says to do objdump, but
> the output ends up being 21MB gzipped... do you really want that?)
>> -72> 2016-09-20 20:38:02.006876 708f66acb700 5 -- op tracker --
>> seq: 1910, time: 2016-09-20 20:38:02.006875, event: commit_sent, op:
>> osd_op(client.104119.1:1626 24.fe033d78 100000004c5.00000000 [write
>> 0~785] snapc 1=[] ondisk+write e320)
>> -71> 2016-09-20 20:38:02.006909 708f662b0700 5 -- op tracker --
>> seq: 1910, time: 2016-09-20 20:38:02.006908, event: op_applied, op:
>> osd_op(client.104119.1:1626 24.fe033d78 100000004c5.00000000 [write
>> 0~785] snapc 1=[] ondisk+write e320)
>> -70> 2016-09-20 20:38:02.006912 708f56c6c700 5 -- op tracker --
>> seq: 2017, time: 2016-09-20 20:38:02.006911, event: started, op:
>> osd_op(client.104119.1:1733 24.31bc808d 10000000534.00000000 [write
>> 0~2792 [1@-1]] snapc 1=[] ondisk+write e320)
>> -69> 2016-09-20 20:38:02.006959 708f662b0700 5 -- op tracker --
>> seq: 1910, time: 2016-09-20 20:38:02.006958, event: done, op:
>> osd_op(client.104119.1:1626 24.fe033d78 100000004c5.00000000 [write
>> 0~785] snapc 1=[] ondisk+write e320)
>> -68> 2016-09-20 20:38:02.006999 708f662b0700 5 -- op tracker --
>> seq: 1941, time: 2016-09-20 20:38:02.006999, event: op_applied, op:
>> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
>> 0~3676] snapc 1=[] ondisk+write e320)
>> -67> 2016-09-20 20:38:02.007016 708f56c6c700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'22, trimmed:
>> -66> 2016-09-20 20:38:02.007019 708f57d97700 5 -- op tracker --
>> seq: 1997, time: 2016-09-20 20:38:02.007018, event: started, op:
>> osd_op(client.104119.1:1713 24.c6dc2afd 1000000051f.00000000 [write
>> 0~455 [1@-1]] snapc 1=[] ondisk+write e320)
>> -65> 2016-09-20 20:38:02.007053 708f53a27700 5 -- op tracker --
>> seq: 2003, time: 2016-09-20 20:38:02.007052, event:
>> commit_queued_for_journal_write, op: osd_op(client.104119.1:1719
>> 24.29c023c9 10000000525.00000000 [write 0~1243] snapc 1=[]
>> ondisk+write e320)
>> -64> 2016-09-20 20:38:02.007118 708f57d97700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'26, trimmed:
>> -63> 2016-09-20 20:38:02.007088 708f53a27700 5 -- op tracker --
>> seq: 2010, time: 2016-09-20 20:38:02.007088, event: reached_pg, op:
>> osd_op(client.104119.1:1726 24.efd71449 1000000052d.00000000 [write
>> 0~6723 [1@-1]] snapc 1=[] ondisk+write e320)
>> -62> 2016-09-20 20:38:02.007200 708f58600700 5 -- op tracker --
>> seq: 1998, time: 2016-09-20 20:38:02.007199, event:
>> commit_queued_for_journal_write, op: osd_op(client.104119.1:1714
>> 24.a0eb378f 10000000520.00000000 [write 0~592] snapc 1=[] ondisk+write
>> e320)
>> -61> 2016-09-20 20:38:02.007243 708f4d8c9700 1 --
>> 192.168.178.200:6801/11774 <== client.104119
>> 192.168.178.200:0/3572238878 1750 ==== osd_op(client.104119.1:1750
>> 24.27ed6b04 10000000545.00000000 [write 0~29685 [1@-1]] snapc 1=[]
>> ondisk+write e320) v4 ==== 159+0+29685 (3814620007 0 2860778659)
>> 0x37dfccffa80 con 0x37df7806f00
>> -60> 2016-09-20 20:38:02.007257 708f58600700 5 -- op tracker --
>> seq: 2002, time: 2016-09-20 20:38:02.007256, event: reached_pg, op:
>> osd_op(client.104119.1:1718 24.dbb483a8 10000000524.00000000 [write
>> 0~599 [1@-1]] snapc 1=[] ondisk+write e320)
>> -59> 2016-09-20 20:38:02.007286 708f4d8c9700 5 -- op tracker --
>> seq: 2034, time: 2016-09-20 20:38:02.007285, event: queued_for_pg, op:
>> osd_op(client.104119.1:1750 24.27ed6b04 10000000545.00000000 [write
>> 0~29685 [1@-1]] snapc 1=[] ondisk+write e320)
>> -58> 2016-09-20 20:38:02.007403 708f58600700 5 -- op tracker --
>> seq: 2002, time: 2016-09-20 20:38:02.007402, event: started, op:
>> osd_op(client.104119.1:1718 24.dbb483a8 10000000524.00000000 [write
>> 0~599 [1@-1]] snapc 1=[] ondisk+write e320)
>> -57> 2016-09-20 20:38:02.007418 708f55c48700 5 -- op tracker --
>> seq: 2001, time: 2016-09-20 20:38:02.007417, event:
>> commit_queued_for_journal_write, op: osd_op(client.104119.1:1717
>> 24.e5d704b7 10000000523.00000000 [write 0~11341] snapc 1=[]
>> ondisk+write e320)
>> -56> 2016-09-20 20:38:02.007458 708f55c48700 5 -- op tracker --
>> seq: 2004, time: 2016-09-20 20:38:02.007457, event: reached_pg, op:
>> osd_op(client.104119.1:1720 24.745f6494 10000000526.00000000 [write
>> 0~7299 [1@-1]] snapc 1=[] ondisk+write e320)
>> -55> 2016-09-20 20:38:02.007472 708f58600700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'22, trimmed:
>> -54> 2016-09-20 20:38:02.007610 708f55c48700 5 -- op tracker --
>> seq: 2004, time: 2016-09-20 20:38:02.007610, event: started, op:
>> osd_op(client.104119.1:1720 24.745f6494 10000000526.00000000 [write
>> 0~7299 [1@-1]] snapc 1=[] ondisk+write e320)
>> -53> 2016-09-20 20:38:02.007610 708f55379700 5 -- op tracker --
>> seq: 1989, time: 2016-09-20 20:38:02.007610, event:
>> commit_queued_for_journal_write, op: osd_op(client.104119.1:1705
>> 24.6f9a4190 10000000517.00000000 [write 0~26985] snapc 1=[]
>> ondisk+write e320)
>> -52> 2016-09-20 20:38:02.007659 708f55379700 5 -- op tracker --
>> seq: 2006, time: 2016-09-20 20:38:02.007658, event: reached_pg, op:
>> osd_op(client.104119.1:1722 24.ff02c65a 10000000528.00000000 [write
>> 0~291 [1@-1]] snapc 1=[] ondisk+write e320)
>> -51> 2016-09-20 20:38:02.007659 708f53a27700 5 -- op tracker --
>> seq: 2010, time: 2016-09-20 20:38:02.007659, event: started, op:
>> osd_op(client.104119.1:1726 24.efd71449 1000000052d.00000000 [write
>> 0~6723 [1@-1]] snapc 1=[] ondisk+write e320)
>> -50> 2016-09-20 20:38:02.007694 708f55c48700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'29, trimmed:
>> -49> 2016-09-20 20:38:02.007750 708f53a27700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'19, trimmed:
>> -48> 2016-09-20 20:38:02.007738 708f5424b700 5 -- op tracker --
>> seq: 1980, time: 2016-09-20 20:38:02.007738, event:
>> commit_queued_for_journal_write, op: osd_op(client.104119.1:1696
>> 24.3ef04952 1000000050e.00000000 [write 0~1944] snapc 1=[]
>> ondisk+write e320)
>> -47> 2016-09-20 20:38:02.007802 708f5424b700 5 -- op tracker --
>> seq: 2018, time: 2016-09-20 20:38:02.007802, event: reached_pg, op:
>> osd_op(client.104119.1:1734 24.88654857 10000000535.00000000 [write
>> 0~5934 [1@-1]] snapc 1=[] ondisk+write e320)
>> -46> 2016-09-20 20:38:02.007843 708f55379700 5 -- op tracker --
>> seq: 2006, time: 2016-09-20 20:38:02.007842, event: started, op:
>> osd_op(client.104119.1:1722 24.ff02c65a 10000000528.00000000 [write
>> 0~291 [1@-1]] snapc 1=[] ondisk+write e320)
>> -45> 2016-09-20 20:38:02.007916 708f55379700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'27, trimmed:
>> -44> 2016-09-20 20:38:02.007928 708f56461700 5 -- op tracker --
>> seq: 2008, time: 2016-09-20 20:38:02.007927, event:
>> commit_queued_for_journal_write, op: osd_op(client.104119.1:1724
>> 24.98531e71 1000000052b.00000000 [write 0~2947] snapc 1=[]
>> ondisk+write e320)
>> -43> 2016-09-20 20:38:02.007962 708f56461700 5 -- op tracker --
>> seq: 2014, time: 2016-09-20 20:38:02.007961, event: reached_pg, op:
>> osd_op(client.104119.1:1730 24.f89ede22 10000000531.00000000 [write
>> 0~18483 [1@-1]] snapc 1=[] ondisk+write e320)
>> -42> 2016-09-20 20:38:02.007962 708f662b0700 5 -- op tracker --
>> seq: 1944, time: 2016-09-20 20:38:02.007961, event: op_applied, op:
>> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
>> 0~670] snapc 1=[] ondisk+write e320)
>> -41> 2016-09-20 20:38:02.007981 708f69520700 5 -- op tracker --
>> seq: 2005, time: 2016-09-20 20:38:02.007980, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1721
>> 24.9141130d 10000000527.00000000 [write 0~242] snapc 1=[] ondisk+write
>> e320)
>> -40> 2016-09-20 20:38:02.007942 708f67bbd700 0
>> filestore(/var/lib/ceph/osd/ceph-0) write couldn't open
>> 24.32_head/#24:4d11884b:::10000000504.00000000:head#: (24) Too many
>> open files
>> -39> 2016-09-20 20:38:02.007759 708f673ae700 0
>> filestore(/var/lib/ceph/osd/ceph-0) write couldn't open
>> 24.1d_head/#24:b9437d99:::100000004f8.00000000:head#: (24) Too many
>> open files
>> -38> 2016-09-20 20:38:02.007992 708f662b0700 5 -- op tracker --
>> seq: 1947, time: 2016-09-20 20:38:02.007992, event: op_applied, op:
>> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
>> 0~9997] snapc 1=[] ondisk+write e320)
>> -37> 2016-09-20 20:38:02.007994 708f69520700 5 -- op tracker --
>> seq: 1979, time: 2016-09-20 20:38:02.007994, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1695
>> 24.68832578 1000000050d.00000000 [write 0~1708] snapc 1=[]
>> ondisk+write e320)
>> -36> 2016-09-20 20:38:02.007998 708f66acb700 5 -- op tracker --
>> seq: 1944, time: 2016-09-20 20:38:02.007997, event: op_commit, op:
>> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
>> 0~670] snapc 1=[] ondisk+write e320)
>> -35> 2016-09-20 20:38:02.008005 708f69520700 5 -- op tracker --
>> seq: 2003, time: 2016-09-20 20:38:02.008005, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1719
>> 24.29c023c9 10000000525.00000000 [write 0~1243] snapc 1=[]
>> ondisk+write e320)
>> -34> 2016-09-20 20:38:02.008016 708f67bbd700 0
>> filestore(/var/lib/ceph/osd/ceph-0) error (24) Too many open files
>> not handled on operation 0x37df78190c0 (11072.0.0, or op 0, counting
>> from 0)
>> -33> 2016-09-20 20:38:02.008016 708f69520700 5 -- op tracker --
>> seq: 1998, time: 2016-09-20 20:38:02.008016, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1714
>> 24.a0eb378f 10000000520.00000000 [write 0~592] snapc 1=[] ondisk+write
>> e320)
>> -32> 2016-09-20 20:38:02.008026 708f67bbd700 0
>> filestore(/var/lib/ceph/osd/ceph-0) unexpected error code
>> -31> 2016-09-20 20:38:02.008015 708f66acb700 1 --
>> 192.168.178.200:6801/11774 --> 192.168.178.200:0/3572238878 --
>> osd_op_reply(1660 100000004e8.00000000 [write 0~670] v320'34 uv34
>> ondisk = 0) v7 -- ?+0 0x37df6bb0700 con 0x37df7806f00
>> -30> 2016-09-20 20:38:02.008039 708f69520700 5 -- op tracker --
>> seq: 2001, time: 2016-09-20 20:38:02.008038, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1717
>> 24.e5d704b7 10000000523.00000000 [write 0~11341] snapc 1=[]
>> ondisk+write e320)
>> -29> 2016-09-20 20:38:02.008043 708f66acb700 5 -- op tracker --
>> seq: 1944, time: 2016-09-20 20:38:02.008043, event: commit_sent, op:
>> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
>> 0~670] snapc 1=[] ondisk+write e320)
>> -28> 2016-09-20 20:38:02.008040 708f5424b700 5 -- op tracker --
>> seq: 2018, time: 2016-09-20 20:38:02.008040, event: started, op:
>> osd_op(client.104119.1:1734 24.88654857 10000000535.00000000 [write
>> 0~5934 [1@-1]] snapc 1=[] ondisk+write e320)
>> -27> 2016-09-20 20:38:02.008050 708f69520700 5 -- op tracker --
>> seq: 1989, time: 2016-09-20 20:38:02.008049, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1705
>> 24.6f9a4190 10000000517.00000000 [write 0~26985] snapc 1=[]
>> ondisk+write e320)
>> -26> 2016-09-20 20:38:02.008060 708f69520700 5 -- op tracker --
>> seq: 1980, time: 2016-09-20 20:38:02.008060, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1696
>> 24.3ef04952 1000000050e.00000000 [write 0~1944] snapc 1=[]
>> ondisk+write e320)
>> -25> 2016-09-20 20:38:02.008070 708f69520700 5 -- op tracker --
>> seq: 2008, time: 2016-09-20 20:38:02.008070, event:
>> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1724
>> 24.98531e71 1000000052b.00000000 [write 0~2947] snapc 1=[]
>> ondisk+write e320)
>> -24> 2016-09-20 20:38:02.008070 708f66acb700 5 -- op tracker --
>> seq: 1944, time: 2016-09-20 20:38:02.008065, event: done, op:
>> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
>> 0~670] snapc 1=[] ondisk+write e320)
>> -23> 2016-09-20 20:38:02.008072 708f673ae700 0
>> filestore(/var/lib/ceph/osd/ceph-0) error (24) Too many open files
>> not handled on operation 0x37df7538b40 (11071.0.0, or op 0, counting
>> from 0)
>> -22> 2016-09-20 20:38:02.008087 708f673ae700 0
>> filestore(/var/lib/ceph/osd/ceph-0) unexpected error code
>> -21> 2016-09-20 20:38:02.008104 708f66acb700 5 -- op tracker --
>> seq: 1941, time: 2016-09-20 20:38:02.008104, event: op_commit, op:
>> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
>> 0~3676] snapc 1=[] ondisk+write e320)
>> -20> 2016-09-20 20:38:02.008116 708f66acb700 1 --
>> 192.168.178.200:6801/11774 --> 192.168.178.200:0/3572238878 --
>> osd_op_reply(1657 100000004e5.00000000 [write 0~3676] v320'25 uv25
>> ondisk = 0) v7 -- ?+0 0x37df7b54680 con 0x37df7806f00
>> -19> 2016-09-20 20:38:02.008128 708f66acb700 5 -- op tracker --
>> seq: 1941, time: 2016-09-20 20:38:02.008128, event: commit_sent, op:
>> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
>> 0~3676] snapc 1=[] ondisk+write e320)
>> -18> 2016-09-20 20:38:02.008141 708f5424b700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'30, trimmed:
>> -17> 2016-09-20 20:38:02.008138 708f56461700 5 -- op tracker --
>> seq: 2014, time: 2016-09-20 20:38:02.008138, event: started, op:
>> osd_op(client.104119.1:1730 24.f89ede22 10000000531.00000000 [write
>> 0~18483 [1@-1]] snapc 1=[] ondisk+write e320)
>> -16> 2016-09-20 20:38:02.008157 708f66acb700 5 -- op tracker --
>> seq: 1941, time: 2016-09-20 20:38:02.008157, event: done, op:
>> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
>> 0~3676] snapc 1=[] ondisk+write e320)
>> -15> 2016-09-20 20:38:02.008179 708f66acb700 5 -- op tracker --
>> seq: 1947, time: 2016-09-20 20:38:02.008179, event: op_commit, op:
>> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
>> 0~9997] snapc 1=[] ondisk+write e320)
>> -14> 2016-09-20 20:38:02.008188 708f66acb700 1 --
>> 192.168.178.200:6801/11774 --> 192.168.178.200:0/3572238878 --
>> osd_op_reply(1663 100000004eb.00000000 [write 0~9997] v320'31 uv31
>> ondisk = 0) v7 -- ?+0 0x37dfb45a9c0 con 0x37df7806f00
>> -13> 2016-09-20 20:38:02.008201 708f66acb700 5 -- op tracker --
>> seq: 1947, time: 2016-09-20 20:38:02.008201, event: commit_sent, op:
>> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
>> 0~9997] snapc 1=[] ondisk+write e320)
>> -12> 2016-09-20 20:38:02.008216 708f56461700 5 write_log with:
>> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
>> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
>> 320'34, trimmed:
>> -11> 2016-09-20 20:38:02.008218 708f66acb700 5 -- op tracker --
>> seq: 1947, time: 2016-09-20 20:38:02.008218, event: done, op:
>> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
>> 0~9997] snapc 1=[] ondisk+write e320)
>> -10> 2016-09-20 20:38:02.008354 708f4d8c9700 1 --
>> 192.168.178.200:6801/11774 <== client.104119
>> 192.168.178.200:0/3572238878 1751 ==== osd_op(client.104119.1:1751
>> 24.1ebe4f02 10000000546.00000000 [write 0~4810 [1@-1]] snapc 1=[]
>> ondisk+write e320) v4 ==== 159+0+4810 (1569622590 0 2543026334)
>> 0x37dfccfe080 con 0x37df7806f00
>> -9> 2016-09-20 20:38:02.008387 708f4d8c9700 5 -- op tracker --
>> seq: 2035, time: 2016-09-20 20:38:02.008387, event: queued_for_pg, op:
>> osd_op(client.104119.1:1751 24.1ebe4f02 10000000546.00000000 [write
>> 0~4810 [1@-1]] snapc 1=[] ondisk+write e320)
>> -8> 2016-09-20 20:38:02.008433 708f4d8c9700 1 --
>> 192.168.178.200:6801/11774 <== client.104119
>> 192.168.178.200:0/3572238878 1752 ==== osd_op(client.104119.1:1752
>> 24.f3467809 10000000547.00000000 [write 0~448 [1@-1]] snapc 1=[]
>> ondisk+write e320) v4 ==== 159+0+448 (4008681173 0 2052829007)
>> 0x37dfccfda00 con 0x37df7806f00
>> -7> 2016-09-20 20:38:02.008455 708f4d8c9700 5 -- op tracker --
>> seq: 2036, time: 2016-09-20 20:38:02.008455, event: queued_for_pg, op:
>> osd_op(client.104119.1:1752 24.f3467809 10000000547.00000000 [write
>> 0~448 [1@-1]] snapc 1=[] ondisk+write e320)
>> -6> 2016-09-20 20:38:02.008530 708f4d8c9700 1 --
>> 192.168.178.200:6801/11774 <== client.104119
>> 192.168.178.200:0/3572238878 1753 ==== osd_op(client.104119.1:1753
>> 24.ad441839 10000000549.00000000 [write 0~2671 [1@-1]] snapc 1=[]
>> ondisk+write e320) v4 ==== 159+0+2671 (2714568315 0 3869023890)
>> 0x37dfccfea40 con 0x37df7806f00
>> -5> 2016-09-20 20:38:02.008552 708f4d8c9700 5 -- op tracker --
>> seq: 2037, time: 2016-09-20 20:38:02.008552, event: queued_for_pg, op:
>> osd_op(client.104119.1:1753 24.ad441839 10000000549.00000000 [write
>> 0~2671 [1@-1]] snapc 1=[] ondisk+write e320)
>> -4> 2016-09-20 20:38:02.008028 708f67bbd700 0
>> filestore(/var/lib/ceph/osd/ceph-0) transaction dump:
>> {
>> "ops": [
>> {
>> "op_num": 0,
>> "op_name": "write",
>> "collection": "24.32_head",
>> "oid": "#24:4d11884b:::10000000504.00000000:head#",
>> "length": 1003,
>> "offset": 0,
>> "bufferlist length": 1003
>> },
>> {
>> "op_num": 1,
>> "op_name": "setattrs",
>> "collection": "24.32_head",
>> "oid": "#24:4d11884b:::10000000504.00000000:head#",
>> "attr_lens": {
>> "_": 255,
>> "snapset": 31
>> }
>> },
>> {
>> "op_num": 2,
>> "op_name": "omap_setkeys",
>> "collection": "24.32_head",
>> "oid": "#24:4c000000::::head#",
>> "attr_lens": {
>> "0000000320.00000000000000000027": 164,
>> "_info": 847
>> }
>> }
>> ]
>> }
>>
>> -3> 2016-09-20 20:38:02.008089 708f673ae700 0
>> filestore(/var/lib/ceph/osd/ceph-0) transaction dump:
>> {
>> "ops": [
>> {
>> "op_num": 0,
>> "op_name": "write",
>> "collection": "24.1d_head",
>> "oid": "#24:b9437d99:::100000004f8.00000000:head#",
>> "length": 1736,
>> "offset": 0,
>> "bufferlist length": 1736
>> },
>> {
>> "op_num": 1,
>> "op_name": "setattrs",
>> "collection": "24.1d_head",
>> "oid": "#24:b9437d99:::100000004f8.00000000:head#",
>> "attr_lens": {
>> "_": 255,
>> "snapset": 31
>> }
>> },
>> {
>> "op_num": 2,
>> "op_name": "omap_setkeys",
>> "collection": "24.1d_head",
>> "oid": "#24:b8000000::::head#",
>> "attr_lens": {
>> "0000000320.00000000000000000036": 164,
>> "_info": 847
>> }
>> }
>> ]
>> }
>>
>> -2> 2016-09-20 20:38:02.009786 708f67bbd700 -1 dump_open_fds
>> unable to open /proc/self/fd
>> -1> 2016-09-20 20:38:02.009789 708f673ae700 -1 dump_open_fds
>> unable to open /proc/self/fd
>> 0> 2016-09-20 20:38:02.015953 708f673ae700 -1
>> os/filestore/FileStore.cc: In function 'void
>> FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
>> ThreadPool::TPHandle*)' thread 708f673ae700 time 2016-09-20
>> 20:38:02.009795
>> os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")
>>
>> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x80) [0x37dea9aa180]
>> 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
>> long, int, ThreadPool::TPHandle*)+0x1c9f) [0x37dea63c5af]
>> 3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
>> std::allocator<ObjectStore::Transaction> >&, unsigned long,
>> ThreadPool::TPHandle*)+0x3b) [0x37dea64045b]
>> 4: (FileStore::_do_op(FileStore::OpSequencer*,
>> ThreadPool::TPHandle&)+0x32c) [0x37dea6407bc]
>> 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb0) [0x37dea99a4c0]
>> 6: (ThreadPool::WorkThread::entry()+0x10) [0x37dea99b670]
>> 7: (()+0x7454) [0x708f75a42454]
>> 8: (clone()+0x5f) [0x708f73ab37df]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> --- logging levels ---
>> 0/ 5 none
>> 0/ 1 lockdep
>> 0/ 1 context
>> 1/ 1 crush
>> 1/ 5 mds
>> 1/ 5 mds_balancer
>> 1/ 5 mds_locker
>> 1/ 5 mds_log
>> 1/ 5 mds_log_expire
>> 1/ 5 mds_migrator
>> 0/ 1 buffer
>> 0/ 1 timer
>> 0/ 1 filer
>> 0/ 1 striper
>> 0/ 1 objecter
>> 0/ 5 rados
>> 0/ 5 rbd
>> 0/ 5 rbd_mirror
>> 0/ 5 rbd_replay
>> 0/ 5 journaler
>> 0/ 5 objectcacher
>> 0/ 5 client
>> 0/ 5 osd
>> 0/ 5 optracker
>> 0/ 5 objclass
>> 1/ 3 filestore
>> 1/ 3 journal
>> 0/ 5 ms
>> 1/ 5 mon
>> 0/10 monc
>> 1/ 5 paxos
>> 0/ 5 tp
>> 1/ 5 auth
>> 1/ 5 crypto
>> 1/ 1 finisher
>> 1/ 5 heartbeatmap
>> 1/ 5 perfcounter
>> 1/ 5 rgw
>> 1/10 civetweb
>> 1/ 5 javaclient
>> 1/ 5 asok
>> 1/ 1 throttle
>> 0/ 0 refs
>> 1/ 5 xio
>> 1/ 5 compressor
>> 1/ 5 newstore
>> 1/ 5 bluestore
>> 1/ 5 bluefs
>> 1/ 3 bdev
>> 1/ 5 kstore
>> 4/ 5 rocksdb
>> 4/ 5 leveldb
>> 1/ 5 kinetic
>> 1/ 5 fuse
>> -2/-2 (syslog threshold)
>> -1/-1 (stderr threshold)
>> max_recent 10000
>> max_new 1000
>> log_file /var/log/ceph/ceph-osd.0.log
>> --- end dump of recent events ---
>> 2016-09-20 20:38:02.015963 708f67bbd700 -1 os/filestore/FileStore.cc:
>> In function 'void
>> FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
>> ThreadPool::TPHandle*)' thread 708f67bbd700 time 2016-09-20
>> 20:38:02.009792
>> os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")
>>
>> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x80) [0x37dea9aa180]
>> 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
>> long, int, ThreadPool::TPHandle*)+0x1c9f) [0x37dea63c5af]
>> 3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
>> std::allocator<ObjectStore::Transaction> >&, unsigned long,
>> ThreadPool::TPHandle*)+0x3b) [0x37dea64045b]
>> 4: (FileStore::_do_op(FileStore::OpSequencer*,
>> ThreadPool::TPHandle&)+0x32c) [0x37dea6407bc]
>> 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb0) [0x37dea99a4c0]
>> 6: (ThreadPool::WorkThread::entry()+0x10) [0x37dea99b670]
>> 7: (()+0x7454) [0x708f75a42454]
>> 8: (clone()+0x5f) [0x708f73ab37df]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> --- begin dump of recent events ---
>> -40> 2016-09-20 20:38:02.015963 708f67bbd700 -1
>> os/filestore/FileStore.cc: In function 'void
>> FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
>> ThreadPool::TPHandle*)' thread 708f67bbd700 time 2016-09-20
>> 20:38:02.009792
>> os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")
>>
>> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
>> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x80) [0x37dea9aa180]
>> 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
>> long, int, ThreadPool::TPHandle*)+0x1c9f) [0x37dea63c5af]
>> 3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
>> std::allocator<ObjectStore::Transaction> >&, unsigned long,
>> ThreadPool::TPHandle*)+0x3b) [0x37dea64045b]
>> 4: (FileStore::_do_op(FileStore::OpSequencer*,
>> ThreadPool::TPHandle&)+0x32c) [0x37dea6407bc]
>> 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb0) [0x37dea99a4c0]
>> 6: (ThreadPool::WorkThread::entry()+0x10) [0x37dea99b670]
>> 7: (()+0x7454) [0x708f75a42454]
>> 8: (clone()+0x5f) [0x708f73ab37df]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>>
>> -39> 2016-09-20 20:38:02.049593 708f68c64700 5 -- op tracker --
>> seq: 1963, time: 2016-09-20 20:38:02.049593, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1679
>> 24.15b72f8d 100000004fc.00000000 [write 0~1692] snapc 1=[]
>> ondisk+write e320)
>> -38> 2016-09-20 20:38:02.049612 708f68c64700 5 -- op tracker --
>> seq: 1981, time: 2016-09-20 20:38:02.049612, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1697
>> 24.417db39e 1000000050f.00000000 [write 0~109] snapc 1=[] ondisk+write
>> e320)
>> -37> 2016-09-20 20:38:02.049619 708f68c64700 5 -- op tracker --
>> seq: 1957, time: 2016-09-20 20:38:02.049619, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1673
>> 24.8a7cb939 100000004f6.00000000 [write 0~8137] snapc 1=[]
>> ondisk+write e320)
>> -36> 2016-09-20 20:38:02.049626 708f68c64700 5 -- op tracker --
>> seq: 1956, time: 2016-09-20 20:38:02.049626, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1672
>> 24.57147baa 100000004f5.00000000 [write 0~8887] snapc 1=[]
>> ondisk+write e320)
>> -35> 2016-09-20 20:38:02.049634 708f68c64700 5 -- op tracker --
>> seq: 1965, time: 2016-09-20 20:38:02.049634, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1681
>> 24.383218c3 100000004fe.00000000 [write 0~16645] snapc 1=[]
>> ondisk+write e320)
>> -34> 2016-09-20 20:38:02.049641 708f68c64700 5 -- op tracker --
>> seq: 1967, time: 2016-09-20 20:38:02.049641, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1683
>> 24.55907462 10000000500.00000000 [write 0~2454] snapc 1=[]
>> ondisk+write e320)
>> -33> 2016-09-20 20:38:02.049663 708f68c64700 5 -- op tracker --
>> seq: 1934, time: 2016-09-20 20:38:02.049648, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1650
>> 24.5197a39f 100000004de.00000000 [write 0~15245] snapc 1=[]
>> ondisk+write e320)
>> -32> 2016-09-20 20:38:02.049674 708f68c64700 5 -- op tracker --
>> seq: 1939, time: 2016-09-20 20:38:02.049674, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1655
>> 24.d21a71d0 100000004e3.00000000 [write 0~19607] snapc 1=[]
>> ondisk+write e320)
>> -31> 2016-09-20 20:38:02.049680 708f68c64700 5 -- op tracker --
>> seq: 1970, time: 2016-09-20 20:38:02.049680, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1686
>> 24.5aa9f2dd 10000000503.00000000 [write 0~845] snapc 1=[] ondisk+write
>> e320)
>> -30> 2016-09-20 20:38:02.049686 708f68c64700 5 -- op tracker --
>> seq: 1982, time: 2016-09-20 20:38:02.049686, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1698
>> 24.a25ce0f7 10000000510.00000000 [write 0~2789] snapc 1=[]
>> ondisk+write e320)
>> -29> 2016-09-20 20:38:02.049707 708f68c64700 5 -- op tracker --
>> seq: 1986, time: 2016-09-20 20:38:02.049707, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1702
>> 24.4d9e16f2 10000000514.00000000 [write 0~5569] snapc 1=[]
>> ondisk+write e320)
>> -28> 2016-09-20 20:38:02.049713 708f68c64700 5 -- op tracker --
>> seq: 1968, time: 2016-09-20 20:38:02.049713, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1684
>> 24.c8c4a730 10000000501.00000000 [write 0~325] snapc 1=[] ondisk+write
>> e320)
>> -27> 2016-09-20 20:38:02.049719 708f68c64700 5 -- op tracker --
>> seq: 1964, time: 2016-09-20 20:38:02.049719, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1680
>> 24.988934ef 100000004fd.00000000 [write 0~25737] snapc 1=[]
>> ondisk+write e320)
>> -26> 2016-09-20 20:38:02.049725 708f68c64700 5 -- op tracker --
>> seq: 1966, time: 2016-09-20 20:38:02.049725, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1682
>> 24.5dcdfed6 100000004ff.00000000 [write 0~9250] snapc 1=[]
>> ondisk+write e320)
>> -25> 2016-09-20 20:38:02.049761 708f68c64700 5 -- op tracker --
>> seq: 1969, time: 2016-09-20 20:38:02.049760, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1685
>> 24.e9038fe6 10000000502.00000000 [write 0~178] snapc 1=[] ondisk+write
>> e320)
>> -24> 2016-09-20 20:38:02.049767 708f68c64700 5 -- op tracker --
>> seq: 1976, time: 2016-09-20 20:38:02.049767, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1692
>> 24.8d10ba3b 10000000509.00000000 [write 0~668] snapc 1=[] ondisk+write
>> e320)
>> -23> 2016-09-20 20:38:02.049773 708f68c64700 5 -- op tracker --
>> seq: 1950, time: 2016-09-20 20:38:02.049773, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1666
>> 24.3df0d155 100000004ef.00000000 [write 0~3603] snapc 1=[]
>> ondisk+write e320)
>> -22> 2016-09-20 20:38:02.049779 708f68c64700 5 -- op tracker --
>> seq: 1978, time: 2016-09-20 20:38:02.049779, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1694
>> 24.c18fac18 1000000050c.00000000 [write 0~6468] snapc 1=[]
>> ondisk+write e320)
>> -21> 2016-09-20 20:38:02.049790 708f68c64700 5 -- op tracker --
>> seq: 1988, time: 2016-09-20 20:38:02.049790, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1704
>> 24.f1a7cf6d 10000000516.00000000 [write 0~115] snapc 1=[] ondisk+write
>> e320)
>> -20> 2016-09-20 20:38:02.049796 708f68c64700 5 -- op tracker --
>> seq: 1946, time: 2016-09-20 20:38:02.049796, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1662
>> 24.2d47992e 100000004ea.00000000 [write 0~12562] snapc 1=[]
>> ondisk+write e320)
>> -19> 2016-09-20 20:38:02.049802 708f68c64700 5 -- op tracker --
>> seq: 1990, time: 2016-09-20 20:38:02.049802, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1706
>> 24.96561485 10000000518.00000000 [write 0~11354] snapc 1=[]
>> ondisk+write e320)
>> -18> 2016-09-20 20:38:02.049810 708f68c64700 5 -- op tracker --
>> seq: 1974, time: 2016-09-20 20:38:02.049810, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1690
>> 24.65b5bb82 10000000507.00000000 [write 0~6552] snapc 1=[]
>> ondisk+write e320)
>> -17> 2016-09-20 20:38:02.049838 708f68c64700 5 -- op tracker --
>> seq: 1983, time: 2016-09-20 20:38:02.049838, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1699
>> 24.34dbf1e0 10000000511.00000000 [write 0~369] snapc 1=[] ondisk+write
>> e320)
>> -16> 2016-09-20 20:38:02.049845 708f68c64700 5 -- op tracker --
>> seq: 1977, time: 2016-09-20 20:38:02.049845, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1693
>> 24.f62eaa57 1000000050b.00000000 [write 0~6533] snapc 1=[]
>> ondisk+write e320)
>> -15> 2016-09-20 20:38:02.049852 708f68c64700 5 -- op tracker --
>> seq: 1973, time: 2016-09-20 20:38:02.049852, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1689
>> 24.a42a2e12 10000000506.00000000 [write 0~9490] snapc 1=[]
>> ondisk+write e320)
>> -14> 2016-09-20 20:38:02.049862 708f68c64700 5 -- op tracker --
>> seq: 1984, time: 2016-09-20 20:38:02.049862, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1700
>> 24.d91a295d 10000000512.00000000 [write 0~15083] snapc 1=[]
>> ondisk+write e320)
>> -13> 2016-09-20 20:38:02.049874 708f68c64700 5 -- op tracker --
>> seq: 1961, time: 2016-09-20 20:38:02.049874, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1677
>> 24.f80dc995 100000004fa.00000000 [write 0~650] snapc 1=[] ondisk+write
>> e320)
>> -12> 2016-09-20 20:38:02.049880 708f68c64700 5 -- op tracker --
>> seq: 1992, time: 2016-09-20 20:38:02.049880, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1708
>> 24.2e11c03c 1000000051a.00000000 [write 0~46616] snapc 1=[]
>> ondisk+write e320)
>> -11> 2016-09-20 20:38:02.049890 708f68c64700 5 -- op tracker --
>> seq: 1958, time: 2016-09-20 20:38:02.049889, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1674
>> 24.b6af1e7d 100000004f7.00000000 [write 0~39735] snapc 1=[]
>> ondisk+write e320)
>> -10> 2016-09-20 20:38:02.049897 708f68c64700 5 -- op tracker --
>> seq: 1994, time: 2016-09-20 20:38:02.049897, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1710
>> 24.d68bbdf7 1000000051c.00000000 [write 0~227] snapc 1=[] ondisk+write
>> e320)
>> -9> 2016-09-20 20:38:02.049903 708f68c64700 5 -- op tracker --
>> seq: 1985, time: 2016-09-20 20:38:02.049903, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1701
>> 24.cd6859d6 10000000513.00000000 [write 0~15879] snapc 1=[]
>> ondisk+write e320)
>> -8> 2016-09-20 20:38:02.049913 708f68c64700 5 -- op tracker --
>> seq: 1987, time: 2016-09-20 20:38:02.049913, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1703
>> 24.8231fcf6 10000000515.00000000 [write 0~73] snapc 1=[] ondisk+write
>> e320)
>> -7> 2016-09-20 20:38:02.049920 708f68c64700 5 -- op tracker --
>> seq: 1993, time: 2016-09-20 20:38:02.049919, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1709
>> 24.69a8aa5 1000000051b.00000000 [write 0~5728] snapc 1=[] ondisk+write
>> e320)
>> -6> 2016-09-20 20:38:02.049926 708f68c64700 5 -- op tracker --
>> seq: 1991, time: 2016-09-20 20:38:02.049925, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1707
>> 24.50655161 10000000519.00000000 [write 0~8081] snapc 1=[]
>> ondisk+write e320)
>> -5> 2016-09-20 20:38:02.049936 708f68c64700 5 -- op tracker --
>> seq: 1996, time: 2016-09-20 20:38:02.049936, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1712
>> 24.15fdb7dd 1000000051e.00000000 [write 0~349] snapc 1=[] ondisk+write
>> e320)
>> -4> 2016-09-20 20:38:02.049942 708f68c64700 5 -- op tracker --
>> seq: 1972, time: 2016-09-20 20:38:02.049941, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1688
>> 24.64558fe4 10000000505.00000000 [write 0~12267] snapc 1=[]
>> ondisk+write e320)
>> -3> 2016-09-20 20:38:02.049948 708f68c64700 5 -- op tracker --
>> seq: 1975, time: 2016-09-20 20:38:02.049948, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1691
>> 24.c0cae786 10000000508.00000000 [write 0~1397] snapc 1=[]
>> ondisk+write e320)
>> -2> 2016-09-20 20:38:02.050013 708f68c64700 5 -- op tracker --
>> seq: 1995, time: 2016-09-20 20:38:02.050013, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1711
>> 24.4ad66977 1000000051d.00000000 [write 0~1172] snapc 1=[]
>> ondisk+write e320)
>> -1> 2016-09-20 20:38:02.050019 708f68c64700 5 -- op tracker --
>> seq: 2000, time: 2016-09-20 20:38:02.050019, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1716
>> 24.3693e8e2 10000000522.00000000 [write 0~24056] snapc 1=[]
>> ondisk+write e320)
>> 0> 2016-09-20 20:38:02.050025 708f68c64700 5 -- op tracker --
>> seq: 1999, time: 2016-09-20 20:38:02.050024, event:
>> journaled_completion_queued, op: osd_op(client.104119.1:1715
>> 24.2141106f 10000000521.00000000 [write 0~1503] snapc 1=[]
>> ondisk+write e320)
>> --- logging levels ---
>> 0/ 5 none
>> 0/ 1 lockdep
>> 0/ 1 context
>> 1/ 1 crush
>> 1/ 5 mds
>> 1/ 5 mds_balancer
>> 1/ 5 mds_locker
>> 1/ 5 mds_log
>> 1/ 5 mds_log_expire
>> 1/ 5 mds_migrator
>> 0/ 1 buffer
>> 0/ 1 timer
>> 0/ 1 filer
>> 0/ 1 striper
>> 0/ 1 objecter
>> 0/ 5 rados
>> 0/ 5 rbd
>> 0/ 5 rbd_mirror
>> 0/ 5 rbd_replay
>> 0/ 5 journaler
>> 0/ 5 objectcacher
>> 0/ 5 client
>> 0/ 5 osd
>> 0/ 5 optracker
>> 0/ 5 objclass
>> 1/ 3 filestore
>> 1/ 3 journal
>> 0/ 5 ms
>> 1/ 5 mon
>> 0/10 monc
>> 1/ 5 paxos
>> 0/ 5 tp
>> 1/ 5 auth
>> 1/ 5 crypto
>> 1/ 1 finisher
>> 1/ 5 heartbeatmap
>> 1/ 5 perfcounter
>> 1/ 5 rgw
>> 1/10 civetweb
>> 1/ 5 javaclient
>> 1/ 5 asok
>> 1/ 1 throttle
>> 0/ 0 refs
>> 1/ 5 xio
>> 1/ 5 compressor
>> 1/ 5 newstore
>> 1/ 5 bluestore
>> 1/ 5 bluefs
>> 1/ 3 bdev
>> 1/ 5 kstore
>> 4/ 5 rocksdb
>> 4/ 5 leveldb
>> 1/ 5 kinetic
>> 1/ 5 fuse
>> -2/-2 (syslog threshold)
>> -1/-1 (stderr threshold)
>> max_recent 10000
>> max_new 1000
>> log_file /var/log/ceph/ceph-osd.0.log
>> --- end dump of recent events ---
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
