Hi,
I created a one disk osd with data and separate journal on the same lvm
volume group just for test, one mon, one mds on my desktop.
I managed to crash the osd just by mounting cephfs and doing cp -a of
the linux-stable git tree into it. It crashed after copying 2.1G which
only covers some of the .git dir and none of the rest. And then when I
killed ceph-mds and restarted the osd and mds, ceph -s said something
about the pgs being stuck or unclean or something, and the computer
froze. :/ After booting again, everything is fine, and the problem was
reproducable the same way...just copying the files again.[but after
writing this mail, I can't seem to cause it as easily again... copying
again works, but sha1sum doesn't, even if I drop caches]
Also reading seems to do the same.
And then I tried adding a 2nd osd (also from vlm, with osd and journal
on same volume group). And that seemed to stop the crashing, but not
sure about corruption.I guess the corruption was on the cephfs but RAM
had good copies or something, so rebooting, etc. is what made the
corruption appear? (I tried to reproduce, but couldn't...didn't try
killing daemons)
> root@client:/mnt/test # ls -l
> total 447
> drwx------ 1 root root 4 2016-09-20 20:37 1/
> drwx------ 1 root root 4 2016-09-20 20:37 2/
> drwx------ 1 root root 4 2016-09-20 20:37 linux-stable/
> -rw-r--r-- 1 root root 457480 2016-09-20 21:38 sums.txt
> root@client:/mnt/test # (cd linux-stable/; sha1sum -c --quiet
> ../sums.txt )
(osd crashed before that finished ... and then impressively, starting
the osd again made the command finish gracefully... and then tried rsync
to finish copying and 6 or so restarts later it finished with just the 1
rsync run)
And then the checksums didn't match... (corruption)
> root@client:/mnt/test # (cd linux-stable/; sha1sum -c --quiet
> ../sums.txt )
> ./.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd: FAILED
> ./.git/objects/e6/d58d90213a4a283d428988a398281663dd68e4: FAILED
> ./.git/objects/81/281381965b21d3c23b2f877e214c4af65d6fa4: FAILED
> ./.git/objects/4c/f549c4a9b23638ab49cc0f8b47c395b1fc8ede: FAILED
> sha1sum: WARNING: 4 computed checksums did NOT match
> root@client:/mnt/test # hexdump -C
> linux-stable/.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> *
> 00000dd0 00 00 00 |...|
> 00000dd3
> peter@peter:~/projects $ hexdump -C
> linux-stable/.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd | head
> 00000000 78 01 bd 5b 7d 6f db c6 19 df bf d4 a7 38 20 40
> |x..[}o.......8 @|
> 00000010 27 0b 8e ec 14 2b 06 24 5d 90 34 75 5c 63 4e 6c
> |'....+.$].4u\cNl|
> 00000020 d8 f1 82 62 19 08 9a 3a 59 ac 29 52 25 29 bb 5e
> |...b...:Y.)R%).^|
> 00000030 9a ef be df f3 dc 1d 79 c7 77 39 c1 84 c0 12 79
> |.......y.w9....y|
> 00000040 77 cf fb 3b 99 eb 38 bd 16 cf 7e 38 fc e1 f0 2f
> |w..;..8...~8.../|
> 00000050 07 b3 89 98 89 fc 21 5f e6 f3 95 78 2a 16 72 19
> |......!_...x*.r.|
> 00000060 25 51 11 a5 49 2e 96 69 26 8a 95 c4 bd bb 28 c4
> |%Q..I..i&.....(.|
> 00000070 57 16 dd c9 4c 2c a3 58 62 7f 21 d7 38 49 87 df
> |W...L,.Xb.!.8I..|
> 00000080 a4 9b 87 2c ba 59 15 62 1a ee 89 ef 0f 0f 9f ed
> |...,.Y.b........|
> 00000090 e3 cf f7 e2 3c 28 b2 28 bc 15 ef d2 70 25 e3 d6
> |....<(.(....p%..|
and then copying that and testing checksums again has even more failures
> root@client:/mnt/test # cp -a linux-stable 3
> root@client:/mnt/test # (cd 3/; sha1sum -c --quiet ../sums.txt )
> ./net/iucv/iucv.c: FAILED
> ./net/kcm/kcmsock.c: FAILED
> ./net/irda/ircomm/ircomm_event.c: FAILED
> ./net/irda/ircomm/ircomm_tty_attach.c: FAILED
> ./net/llc/Makefile: FAILED
> ./net/llc/Kconfig: FAILED
> ./net/llc/af_llc.c: FAILED
> ./net/lapb/lapb_timer.c: FAILED
> ./net/lapb/lapb_subr.c: FAILED
> ./net/lapb/lapb_iface.c: FAILED
> ./net/lapb/Makefile: FAILED
> ./net/lapb/lapb_in.c: FAILED
> ./net/lapb/lapb_out.c: FAILED
> ./net/l2tp/l2tp_eth.c: FAILED
> ./net/l2tp/Kconfig: FAILED
> ./net/l2tp/l2tp_core.h: FAILED
> ./.git/objects/e6/635671beff26a417c02d50adeefa2a6897a9dd: FAILED
> ./.git/objects/e6/d58d90213a4a283d428988a398281663dd68e4: FAILED
> ./.git/objects/81/281381965b21d3c23b2f877e214c4af65d6fa4: FAILED
> ./.git/objects/4c/f549c4a9b23638ab49cc0f8b47c395b1fc8ede: FAILED
> sha1sum: WARNING: 20 computed checksums did NOT match
> root@client:/mnt/test # hexdump -C 3/net/iucv/iucv.c
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> *
> 0000d420 00 00 00 00 00 00 00 00 |........|
> 0000d428
> root@client:/mnt/test # hexdump -C 3/net/kcm/kcmsock.c
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> *
> 0000d010 00 00 00 00 00 00 00 |.......|
> 0000d017
> root@client:/mnt/test # hexdump -C 3/net/lapb/Makefile
> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> |................|
> *
> 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00
> |.............|
> 0000008d
And then I thought it couldn't possibly be only me with such an obvious
bug, so it must be that I'm doing something differently... nobody would
run only 1 osd, right? So I added a 2nd on another disk, with separate
journal on same disk, just like the other...
> root@client:/mnt/test # cp -a linux-stable 4
no crashes this time...
> root@client:/mnt/test # (cd 4/; sha1sum -c --quiet ../sums.txt )
same failures as 3/ ... strange
> root@client:/mnt/test # (cd linux-stable/; sha1sum -c --quiet
> ../sums.txt )
same failures as 3/... but this only had 4 failures before... even more
strange... reading files corrupts them too? or did they corrupt some
other way?
> root@client:/mnt/test # (cd 1/; sha1sum -c --quiet ../sums.txt )
same failures as before, just the 4 files...so this one is fine (copied
from one with 4 failures, so as fine as can be)
> root@peter:~ # ceph --version
> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
> root@peter:~ # uname -a
> Linux peter 4.6.4-1-grsec-kvm-host #22 SMP PREEMPT Tue Jul 26 15:12:19
> CEST 2016 x86_64 GNU/Linux
Here's the osd log for the first crash... (which says to do objdump, but
the output ends up being 21MB gzipped... do you really want that?)
> -72> 2016-09-20 20:38:02.006876 708f66acb700 5 -- op tracker --
> seq: 1910, time: 2016-09-20 20:38:02.006875, event: commit_sent, op:
> osd_op(client.104119.1:1626 24.fe033d78 100000004c5.00000000 [write
> 0~785] snapc 1=[] ondisk+write e320)
> -71> 2016-09-20 20:38:02.006909 708f662b0700 5 -- op tracker --
> seq: 1910, time: 2016-09-20 20:38:02.006908, event: op_applied, op:
> osd_op(client.104119.1:1626 24.fe033d78 100000004c5.00000000 [write
> 0~785] snapc 1=[] ondisk+write e320)
> -70> 2016-09-20 20:38:02.006912 708f56c6c700 5 -- op tracker --
> seq: 2017, time: 2016-09-20 20:38:02.006911, event: started, op:
> osd_op(client.104119.1:1733 24.31bc808d 10000000534.00000000 [write
> 0~2792 [1@-1]] snapc 1=[] ondisk+write e320)
> -69> 2016-09-20 20:38:02.006959 708f662b0700 5 -- op tracker --
> seq: 1910, time: 2016-09-20 20:38:02.006958, event: done, op:
> osd_op(client.104119.1:1626 24.fe033d78 100000004c5.00000000 [write
> 0~785] snapc 1=[] ondisk+write e320)
> -68> 2016-09-20 20:38:02.006999 708f662b0700 5 -- op tracker --
> seq: 1941, time: 2016-09-20 20:38:02.006999, event: op_applied, op:
> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
> 0~3676] snapc 1=[] ondisk+write e320)
> -67> 2016-09-20 20:38:02.007016 708f56c6c700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'22, trimmed:
> -66> 2016-09-20 20:38:02.007019 708f57d97700 5 -- op tracker --
> seq: 1997, time: 2016-09-20 20:38:02.007018, event: started, op:
> osd_op(client.104119.1:1713 24.c6dc2afd 1000000051f.00000000 [write
> 0~455 [1@-1]] snapc 1=[] ondisk+write e320)
> -65> 2016-09-20 20:38:02.007053 708f53a27700 5 -- op tracker --
> seq: 2003, time: 2016-09-20 20:38:02.007052, event:
> commit_queued_for_journal_write, op: osd_op(client.104119.1:1719
> 24.29c023c9 10000000525.00000000 [write 0~1243] snapc 1=[]
> ondisk+write e320)
> -64> 2016-09-20 20:38:02.007118 708f57d97700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'26, trimmed:
> -63> 2016-09-20 20:38:02.007088 708f53a27700 5 -- op tracker --
> seq: 2010, time: 2016-09-20 20:38:02.007088, event: reached_pg, op:
> osd_op(client.104119.1:1726 24.efd71449 1000000052d.00000000 [write
> 0~6723 [1@-1]] snapc 1=[] ondisk+write e320)
> -62> 2016-09-20 20:38:02.007200 708f58600700 5 -- op tracker --
> seq: 1998, time: 2016-09-20 20:38:02.007199, event:
> commit_queued_for_journal_write, op: osd_op(client.104119.1:1714
> 24.a0eb378f 10000000520.00000000 [write 0~592] snapc 1=[] ondisk+write
> e320)
> -61> 2016-09-20 20:38:02.007243 708f4d8c9700 1 --
> 192.168.178.200:6801/11774 <== client.104119
> 192.168.178.200:0/3572238878 1750 ==== osd_op(client.104119.1:1750
> 24.27ed6b04 10000000545.00000000 [write 0~29685 [1@-1]] snapc 1=[]
> ondisk+write e320) v4 ==== 159+0+29685 (3814620007 0 2860778659)
> 0x37dfccffa80 con 0x37df7806f00
> -60> 2016-09-20 20:38:02.007257 708f58600700 5 -- op tracker --
> seq: 2002, time: 2016-09-20 20:38:02.007256, event: reached_pg, op:
> osd_op(client.104119.1:1718 24.dbb483a8 10000000524.00000000 [write
> 0~599 [1@-1]] snapc 1=[] ondisk+write e320)
> -59> 2016-09-20 20:38:02.007286 708f4d8c9700 5 -- op tracker --
> seq: 2034, time: 2016-09-20 20:38:02.007285, event: queued_for_pg, op:
> osd_op(client.104119.1:1750 24.27ed6b04 10000000545.00000000 [write
> 0~29685 [1@-1]] snapc 1=[] ondisk+write e320)
> -58> 2016-09-20 20:38:02.007403 708f58600700 5 -- op tracker --
> seq: 2002, time: 2016-09-20 20:38:02.007402, event: started, op:
> osd_op(client.104119.1:1718 24.dbb483a8 10000000524.00000000 [write
> 0~599 [1@-1]] snapc 1=[] ondisk+write e320)
> -57> 2016-09-20 20:38:02.007418 708f55c48700 5 -- op tracker --
> seq: 2001, time: 2016-09-20 20:38:02.007417, event:
> commit_queued_for_journal_write, op: osd_op(client.104119.1:1717
> 24.e5d704b7 10000000523.00000000 [write 0~11341] snapc 1=[]
> ondisk+write e320)
> -56> 2016-09-20 20:38:02.007458 708f55c48700 5 -- op tracker --
> seq: 2004, time: 2016-09-20 20:38:02.007457, event: reached_pg, op:
> osd_op(client.104119.1:1720 24.745f6494 10000000526.00000000 [write
> 0~7299 [1@-1]] snapc 1=[] ondisk+write e320)
> -55> 2016-09-20 20:38:02.007472 708f58600700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'22, trimmed:
> -54> 2016-09-20 20:38:02.007610 708f55c48700 5 -- op tracker --
> seq: 2004, time: 2016-09-20 20:38:02.007610, event: started, op:
> osd_op(client.104119.1:1720 24.745f6494 10000000526.00000000 [write
> 0~7299 [1@-1]] snapc 1=[] ondisk+write e320)
> -53> 2016-09-20 20:38:02.007610 708f55379700 5 -- op tracker --
> seq: 1989, time: 2016-09-20 20:38:02.007610, event:
> commit_queued_for_journal_write, op: osd_op(client.104119.1:1705
> 24.6f9a4190 10000000517.00000000 [write 0~26985] snapc 1=[]
> ondisk+write e320)
> -52> 2016-09-20 20:38:02.007659 708f55379700 5 -- op tracker --
> seq: 2006, time: 2016-09-20 20:38:02.007658, event: reached_pg, op:
> osd_op(client.104119.1:1722 24.ff02c65a 10000000528.00000000 [write
> 0~291 [1@-1]] snapc 1=[] ondisk+write e320)
> -51> 2016-09-20 20:38:02.007659 708f53a27700 5 -- op tracker --
> seq: 2010, time: 2016-09-20 20:38:02.007659, event: started, op:
> osd_op(client.104119.1:1726 24.efd71449 1000000052d.00000000 [write
> 0~6723 [1@-1]] snapc 1=[] ondisk+write e320)
> -50> 2016-09-20 20:38:02.007694 708f55c48700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'29, trimmed:
> -49> 2016-09-20 20:38:02.007750 708f53a27700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'19, trimmed:
> -48> 2016-09-20 20:38:02.007738 708f5424b700 5 -- op tracker --
> seq: 1980, time: 2016-09-20 20:38:02.007738, event:
> commit_queued_for_journal_write, op: osd_op(client.104119.1:1696
> 24.3ef04952 1000000050e.00000000 [write 0~1944] snapc 1=[]
> ondisk+write e320)
> -47> 2016-09-20 20:38:02.007802 708f5424b700 5 -- op tracker --
> seq: 2018, time: 2016-09-20 20:38:02.007802, event: reached_pg, op:
> osd_op(client.104119.1:1734 24.88654857 10000000535.00000000 [write
> 0~5934 [1@-1]] snapc 1=[] ondisk+write e320)
> -46> 2016-09-20 20:38:02.007843 708f55379700 5 -- op tracker --
> seq: 2006, time: 2016-09-20 20:38:02.007842, event: started, op:
> osd_op(client.104119.1:1722 24.ff02c65a 10000000528.00000000 [write
> 0~291 [1@-1]] snapc 1=[] ondisk+write e320)
> -45> 2016-09-20 20:38:02.007916 708f55379700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'27, trimmed:
> -44> 2016-09-20 20:38:02.007928 708f56461700 5 -- op tracker --
> seq: 2008, time: 2016-09-20 20:38:02.007927, event:
> commit_queued_for_journal_write, op: osd_op(client.104119.1:1724
> 24.98531e71 1000000052b.00000000 [write 0~2947] snapc 1=[]
> ondisk+write e320)
> -43> 2016-09-20 20:38:02.007962 708f56461700 5 -- op tracker --
> seq: 2014, time: 2016-09-20 20:38:02.007961, event: reached_pg, op:
> osd_op(client.104119.1:1730 24.f89ede22 10000000531.00000000 [write
> 0~18483 [1@-1]] snapc 1=[] ondisk+write e320)
> -42> 2016-09-20 20:38:02.007962 708f662b0700 5 -- op tracker --
> seq: 1944, time: 2016-09-20 20:38:02.007961, event: op_applied, op:
> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
> 0~670] snapc 1=[] ondisk+write e320)
> -41> 2016-09-20 20:38:02.007981 708f69520700 5 -- op tracker --
> seq: 2005, time: 2016-09-20 20:38:02.007980, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1721
> 24.9141130d 10000000527.00000000 [write 0~242] snapc 1=[] ondisk+write
> e320)
> -40> 2016-09-20 20:38:02.007942 708f67bbd700 0
> filestore(/var/lib/ceph/osd/ceph-0) write couldn't open
> 24.32_head/#24:4d11884b:::10000000504.00000000:head#: (24) Too many
> open files
> -39> 2016-09-20 20:38:02.007759 708f673ae700 0
> filestore(/var/lib/ceph/osd/ceph-0) write couldn't open
> 24.1d_head/#24:b9437d99:::100000004f8.00000000:head#: (24) Too many
> open files
> -38> 2016-09-20 20:38:02.007992 708f662b0700 5 -- op tracker --
> seq: 1947, time: 2016-09-20 20:38:02.007992, event: op_applied, op:
> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
> 0~9997] snapc 1=[] ondisk+write e320)
> -37> 2016-09-20 20:38:02.007994 708f69520700 5 -- op tracker --
> seq: 1979, time: 2016-09-20 20:38:02.007994, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1695
> 24.68832578 1000000050d.00000000 [write 0~1708] snapc 1=[]
> ondisk+write e320)
> -36> 2016-09-20 20:38:02.007998 708f66acb700 5 -- op tracker --
> seq: 1944, time: 2016-09-20 20:38:02.007997, event: op_commit, op:
> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
> 0~670] snapc 1=[] ondisk+write e320)
> -35> 2016-09-20 20:38:02.008005 708f69520700 5 -- op tracker --
> seq: 2003, time: 2016-09-20 20:38:02.008005, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1719
> 24.29c023c9 10000000525.00000000 [write 0~1243] snapc 1=[]
> ondisk+write e320)
> -34> 2016-09-20 20:38:02.008016 708f67bbd700 0
> filestore(/var/lib/ceph/osd/ceph-0) error (24) Too many open files
> not handled on operation 0x37df78190c0 (11072.0.0, or op 0, counting
> from 0)
> -33> 2016-09-20 20:38:02.008016 708f69520700 5 -- op tracker --
> seq: 1998, time: 2016-09-20 20:38:02.008016, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1714
> 24.a0eb378f 10000000520.00000000 [write 0~592] snapc 1=[] ondisk+write
> e320)
> -32> 2016-09-20 20:38:02.008026 708f67bbd700 0
> filestore(/var/lib/ceph/osd/ceph-0) unexpected error code
> -31> 2016-09-20 20:38:02.008015 708f66acb700 1 --
> 192.168.178.200:6801/11774 --> 192.168.178.200:0/3572238878 --
> osd_op_reply(1660 100000004e8.00000000 [write 0~670] v320'34 uv34
> ondisk = 0) v7 -- ?+0 0x37df6bb0700 con 0x37df7806f00
> -30> 2016-09-20 20:38:02.008039 708f69520700 5 -- op tracker --
> seq: 2001, time: 2016-09-20 20:38:02.008038, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1717
> 24.e5d704b7 10000000523.00000000 [write 0~11341] snapc 1=[]
> ondisk+write e320)
> -29> 2016-09-20 20:38:02.008043 708f66acb700 5 -- op tracker --
> seq: 1944, time: 2016-09-20 20:38:02.008043, event: commit_sent, op:
> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
> 0~670] snapc 1=[] ondisk+write e320)
> -28> 2016-09-20 20:38:02.008040 708f5424b700 5 -- op tracker --
> seq: 2018, time: 2016-09-20 20:38:02.008040, event: started, op:
> osd_op(client.104119.1:1734 24.88654857 10000000535.00000000 [write
> 0~5934 [1@-1]] snapc 1=[] ondisk+write e320)
> -27> 2016-09-20 20:38:02.008050 708f69520700 5 -- op tracker --
> seq: 1989, time: 2016-09-20 20:38:02.008049, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1705
> 24.6f9a4190 10000000517.00000000 [write 0~26985] snapc 1=[]
> ondisk+write e320)
> -26> 2016-09-20 20:38:02.008060 708f69520700 5 -- op tracker --
> seq: 1980, time: 2016-09-20 20:38:02.008060, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1696
> 24.3ef04952 1000000050e.00000000 [write 0~1944] snapc 1=[]
> ondisk+write e320)
> -25> 2016-09-20 20:38:02.008070 708f69520700 5 -- op tracker --
> seq: 2008, time: 2016-09-20 20:38:02.008070, event:
> write_thread_in_journal_buffer, op: osd_op(client.104119.1:1724
> 24.98531e71 1000000052b.00000000 [write 0~2947] snapc 1=[]
> ondisk+write e320)
> -24> 2016-09-20 20:38:02.008070 708f66acb700 5 -- op tracker --
> seq: 1944, time: 2016-09-20 20:38:02.008065, event: done, op:
> osd_op(client.104119.1:1660 24.9ac39fb1 100000004e8.00000000 [write
> 0~670] snapc 1=[] ondisk+write e320)
> -23> 2016-09-20 20:38:02.008072 708f673ae700 0
> filestore(/var/lib/ceph/osd/ceph-0) error (24) Too many open files
> not handled on operation 0x37df7538b40 (11071.0.0, or op 0, counting
> from 0)
> -22> 2016-09-20 20:38:02.008087 708f673ae700 0
> filestore(/var/lib/ceph/osd/ceph-0) unexpected error code
> -21> 2016-09-20 20:38:02.008104 708f66acb700 5 -- op tracker --
> seq: 1941, time: 2016-09-20 20:38:02.008104, event: op_commit, op:
> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
> 0~3676] snapc 1=[] ondisk+write e320)
> -20> 2016-09-20 20:38:02.008116 708f66acb700 1 --
> 192.168.178.200:6801/11774 --> 192.168.178.200:0/3572238878 --
> osd_op_reply(1657 100000004e5.00000000 [write 0~3676] v320'25 uv25
> ondisk = 0) v7 -- ?+0 0x37df7b54680 con 0x37df7806f00
> -19> 2016-09-20 20:38:02.008128 708f66acb700 5 -- op tracker --
> seq: 1941, time: 2016-09-20 20:38:02.008128, event: commit_sent, op:
> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
> 0~3676] snapc 1=[] ondisk+write e320)
> -18> 2016-09-20 20:38:02.008141 708f5424b700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'30, trimmed:
> -17> 2016-09-20 20:38:02.008138 708f56461700 5 -- op tracker --
> seq: 2014, time: 2016-09-20 20:38:02.008138, event: started, op:
> osd_op(client.104119.1:1730 24.f89ede22 10000000531.00000000 [write
> 0~18483 [1@-1]] snapc 1=[] ondisk+write e320)
> -16> 2016-09-20 20:38:02.008157 708f66acb700 5 -- op tracker --
> seq: 1941, time: 2016-09-20 20:38:02.008157, event: done, op:
> osd_op(client.104119.1:1657 24.bfa2088 100000004e5.00000000 [write
> 0~3676] snapc 1=[] ondisk+write e320)
> -15> 2016-09-20 20:38:02.008179 708f66acb700 5 -- op tracker --
> seq: 1947, time: 2016-09-20 20:38:02.008179, event: op_commit, op:
> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
> 0~9997] snapc 1=[] ondisk+write e320)
> -14> 2016-09-20 20:38:02.008188 708f66acb700 1 --
> 192.168.178.200:6801/11774 --> 192.168.178.200:0/3572238878 --
> osd_op_reply(1663 100000004eb.00000000 [write 0~9997] v320'31 uv31
> ondisk = 0) v7 -- ?+0 0x37dfb45a9c0 con 0x37df7806f00
> -13> 2016-09-20 20:38:02.008201 708f66acb700 5 -- op tracker --
> seq: 1947, time: 2016-09-20 20:38:02.008201, event: commit_sent, op:
> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
> 0~9997] snapc 1=[] ondisk+write e320)
> -12> 2016-09-20 20:38:02.008216 708f56461700 5 write_log with:
> dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615,
> dirty_divergent_priors: false, divergent_priors: 0, writeout_from:
> 320'34, trimmed:
> -11> 2016-09-20 20:38:02.008218 708f66acb700 5 -- op tracker --
> seq: 1947, time: 2016-09-20 20:38:02.008218, event: done, op:
> osd_op(client.104119.1:1663 24.4cbe23e6 100000004eb.00000000 [write
> 0~9997] snapc 1=[] ondisk+write e320)
> -10> 2016-09-20 20:38:02.008354 708f4d8c9700 1 --
> 192.168.178.200:6801/11774 <== client.104119
> 192.168.178.200:0/3572238878 1751 ==== osd_op(client.104119.1:1751
> 24.1ebe4f02 10000000546.00000000 [write 0~4810 [1@-1]] snapc 1=[]
> ondisk+write e320) v4 ==== 159+0+4810 (1569622590 0 2543026334)
> 0x37dfccfe080 con 0x37df7806f00
> -9> 2016-09-20 20:38:02.008387 708f4d8c9700 5 -- op tracker --
> seq: 2035, time: 2016-09-20 20:38:02.008387, event: queued_for_pg, op:
> osd_op(client.104119.1:1751 24.1ebe4f02 10000000546.00000000 [write
> 0~4810 [1@-1]] snapc 1=[] ondisk+write e320)
> -8> 2016-09-20 20:38:02.008433 708f4d8c9700 1 --
> 192.168.178.200:6801/11774 <== client.104119
> 192.168.178.200:0/3572238878 1752 ==== osd_op(client.104119.1:1752
> 24.f3467809 10000000547.00000000 [write 0~448 [1@-1]] snapc 1=[]
> ondisk+write e320) v4 ==== 159+0+448 (4008681173 0 2052829007)
> 0x37dfccfda00 con 0x37df7806f00
> -7> 2016-09-20 20:38:02.008455 708f4d8c9700 5 -- op tracker --
> seq: 2036, time: 2016-09-20 20:38:02.008455, event: queued_for_pg, op:
> osd_op(client.104119.1:1752 24.f3467809 10000000547.00000000 [write
> 0~448 [1@-1]] snapc 1=[] ondisk+write e320)
> -6> 2016-09-20 20:38:02.008530 708f4d8c9700 1 --
> 192.168.178.200:6801/11774 <== client.104119
> 192.168.178.200:0/3572238878 1753 ==== osd_op(client.104119.1:1753
> 24.ad441839 10000000549.00000000 [write 0~2671 [1@-1]] snapc 1=[]
> ondisk+write e320) v4 ==== 159+0+2671 (2714568315 0 3869023890)
> 0x37dfccfea40 con 0x37df7806f00
> -5> 2016-09-20 20:38:02.008552 708f4d8c9700 5 -- op tracker --
> seq: 2037, time: 2016-09-20 20:38:02.008552, event: queued_for_pg, op:
> osd_op(client.104119.1:1753 24.ad441839 10000000549.00000000 [write
> 0~2671 [1@-1]] snapc 1=[] ondisk+write e320)
> -4> 2016-09-20 20:38:02.008028 708f67bbd700 0
> filestore(/var/lib/ceph/osd/ceph-0) transaction dump:
> {
> "ops": [
> {
> "op_num": 0,
> "op_name": "write",
> "collection": "24.32_head",
> "oid": "#24:4d11884b:::10000000504.00000000:head#",
> "length": 1003,
> "offset": 0,
> "bufferlist length": 1003
> },
> {
> "op_num": 1,
> "op_name": "setattrs",
> "collection": "24.32_head",
> "oid": "#24:4d11884b:::10000000504.00000000:head#",
> "attr_lens": {
> "_": 255,
> "snapset": 31
> }
> },
> {
> "op_num": 2,
> "op_name": "omap_setkeys",
> "collection": "24.32_head",
> "oid": "#24:4c000000::::head#",
> "attr_lens": {
> "0000000320.00000000000000000027": 164,
> "_info": 847
> }
> }
> ]
> }
>
> -3> 2016-09-20 20:38:02.008089 708f673ae700 0
> filestore(/var/lib/ceph/osd/ceph-0) transaction dump:
> {
> "ops": [
> {
> "op_num": 0,
> "op_name": "write",
> "collection": "24.1d_head",
> "oid": "#24:b9437d99:::100000004f8.00000000:head#",
> "length": 1736,
> "offset": 0,
> "bufferlist length": 1736
> },
> {
> "op_num": 1,
> "op_name": "setattrs",
> "collection": "24.1d_head",
> "oid": "#24:b9437d99:::100000004f8.00000000:head#",
> "attr_lens": {
> "_": 255,
> "snapset": 31
> }
> },
> {
> "op_num": 2,
> "op_name": "omap_setkeys",
> "collection": "24.1d_head",
> "oid": "#24:b8000000::::head#",
> "attr_lens": {
> "0000000320.00000000000000000036": 164,
> "_info": 847
> }
> }
> ]
> }
>
> -2> 2016-09-20 20:38:02.009786 708f67bbd700 -1 dump_open_fds
> unable to open /proc/self/fd
> -1> 2016-09-20 20:38:02.009789 708f673ae700 -1 dump_open_fds
> unable to open /proc/self/fd
> 0> 2016-09-20 20:38:02.015953 708f673ae700 -1
> os/filestore/FileStore.cc: In function 'void
> FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
> ThreadPool::TPHandle*)' thread 708f673ae700 time 2016-09-20
> 20:38:02.009795
> os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")
>
> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x80) [0x37dea9aa180]
> 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
> long, int, ThreadPool::TPHandle*)+0x1c9f) [0x37dea63c5af]
> 3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
> std::allocator<ObjectStore::Transaction> >&, unsigned long,
> ThreadPool::TPHandle*)+0x3b) [0x37dea64045b]
> 4: (FileStore::_do_op(FileStore::OpSequencer*,
> ThreadPool::TPHandle&)+0x32c) [0x37dea6407bc]
> 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb0) [0x37dea99a4c0]
> 6: (ThreadPool::WorkThread::entry()+0x10) [0x37dea99b670]
> 7: (()+0x7454) [0x708f75a42454]
> 8: (clone()+0x5f) [0x708f73ab37df]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_mirror
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> 1/ 5 compressor
> 1/ 5 newstore
> 1/ 5 bluestore
> 1/ 5 bluefs
> 1/ 3 bdev
> 1/ 5 kstore
> 4/ 5 rocksdb
> 4/ 5 leveldb
> 1/ 5 kinetic
> 1/ 5 fuse
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.0.log
> --- end dump of recent events ---
> 2016-09-20 20:38:02.015963 708f67bbd700 -1 os/filestore/FileStore.cc:
> In function 'void
> FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
> ThreadPool::TPHandle*)' thread 708f67bbd700 time 2016-09-20
> 20:38:02.009792
> os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")
>
> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x80) [0x37dea9aa180]
> 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
> long, int, ThreadPool::TPHandle*)+0x1c9f) [0x37dea63c5af]
> 3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
> std::allocator<ObjectStore::Transaction> >&, unsigned long,
> ThreadPool::TPHandle*)+0x3b) [0x37dea64045b]
> 4: (FileStore::_do_op(FileStore::OpSequencer*,
> ThreadPool::TPHandle&)+0x32c) [0x37dea6407bc]
> 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb0) [0x37dea99a4c0]
> 6: (ThreadPool::WorkThread::entry()+0x10) [0x37dea99b670]
> 7: (()+0x7454) [0x708f75a42454]
> 8: (clone()+0x5f) [0x708f73ab37df]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- begin dump of recent events ---
> -40> 2016-09-20 20:38:02.015963 708f67bbd700 -1
> os/filestore/FileStore.cc: In function 'void
> FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
> ThreadPool::TPHandle*)' thread 708f67bbd700 time 2016-09-20
> 20:38:02.009792
> os/filestore/FileStore.cc: 2912: FAILED assert(0 == "unexpected error")
>
> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x80) [0x37dea9aa180]
> 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned
> long, int, ThreadPool::TPHandle*)+0x1c9f) [0x37dea63c5af]
> 3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction,
> std::allocator<ObjectStore::Transaction> >&, unsigned long,
> ThreadPool::TPHandle*)+0x3b) [0x37dea64045b]
> 4: (FileStore::_do_op(FileStore::OpSequencer*,
> ThreadPool::TPHandle&)+0x32c) [0x37dea6407bc]
> 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb0) [0x37dea99a4c0]
> 6: (ThreadPool::WorkThread::entry()+0x10) [0x37dea99b670]
> 7: (()+0x7454) [0x708f75a42454]
> 8: (clone()+0x5f) [0x708f73ab37df]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> -39> 2016-09-20 20:38:02.049593 708f68c64700 5 -- op tracker --
> seq: 1963, time: 2016-09-20 20:38:02.049593, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1679
> 24.15b72f8d 100000004fc.00000000 [write 0~1692] snapc 1=[]
> ondisk+write e320)
> -38> 2016-09-20 20:38:02.049612 708f68c64700 5 -- op tracker --
> seq: 1981, time: 2016-09-20 20:38:02.049612, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1697
> 24.417db39e 1000000050f.00000000 [write 0~109] snapc 1=[] ondisk+write
> e320)
> -37> 2016-09-20 20:38:02.049619 708f68c64700 5 -- op tracker --
> seq: 1957, time: 2016-09-20 20:38:02.049619, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1673
> 24.8a7cb939 100000004f6.00000000 [write 0~8137] snapc 1=[]
> ondisk+write e320)
> -36> 2016-09-20 20:38:02.049626 708f68c64700 5 -- op tracker --
> seq: 1956, time: 2016-09-20 20:38:02.049626, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1672
> 24.57147baa 100000004f5.00000000 [write 0~8887] snapc 1=[]
> ondisk+write e320)
> -35> 2016-09-20 20:38:02.049634 708f68c64700 5 -- op tracker --
> seq: 1965, time: 2016-09-20 20:38:02.049634, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1681
> 24.383218c3 100000004fe.00000000 [write 0~16645] snapc 1=[]
> ondisk+write e320)
> -34> 2016-09-20 20:38:02.049641 708f68c64700 5 -- op tracker --
> seq: 1967, time: 2016-09-20 20:38:02.049641, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1683
> 24.55907462 10000000500.00000000 [write 0~2454] snapc 1=[]
> ondisk+write e320)
> -33> 2016-09-20 20:38:02.049663 708f68c64700 5 -- op tracker --
> seq: 1934, time: 2016-09-20 20:38:02.049648, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1650
> 24.5197a39f 100000004de.00000000 [write 0~15245] snapc 1=[]
> ondisk+write e320)
> -32> 2016-09-20 20:38:02.049674 708f68c64700 5 -- op tracker --
> seq: 1939, time: 2016-09-20 20:38:02.049674, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1655
> 24.d21a71d0 100000004e3.00000000 [write 0~19607] snapc 1=[]
> ondisk+write e320)
> -31> 2016-09-20 20:38:02.049680 708f68c64700 5 -- op tracker --
> seq: 1970, time: 2016-09-20 20:38:02.049680, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1686
> 24.5aa9f2dd 10000000503.00000000 [write 0~845] snapc 1=[] ondisk+write
> e320)
> -30> 2016-09-20 20:38:02.049686 708f68c64700 5 -- op tracker --
> seq: 1982, time: 2016-09-20 20:38:02.049686, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1698
> 24.a25ce0f7 10000000510.00000000 [write 0~2789] snapc 1=[]
> ondisk+write e320)
> -29> 2016-09-20 20:38:02.049707 708f68c64700 5 -- op tracker --
> seq: 1986, time: 2016-09-20 20:38:02.049707, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1702
> 24.4d9e16f2 10000000514.00000000 [write 0~5569] snapc 1=[]
> ondisk+write e320)
> -28> 2016-09-20 20:38:02.049713 708f68c64700 5 -- op tracker --
> seq: 1968, time: 2016-09-20 20:38:02.049713, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1684
> 24.c8c4a730 10000000501.00000000 [write 0~325] snapc 1=[] ondisk+write
> e320)
> -27> 2016-09-20 20:38:02.049719 708f68c64700 5 -- op tracker --
> seq: 1964, time: 2016-09-20 20:38:02.049719, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1680
> 24.988934ef 100000004fd.00000000 [write 0~25737] snapc 1=[]
> ondisk+write e320)
> -26> 2016-09-20 20:38:02.049725 708f68c64700 5 -- op tracker --
> seq: 1966, time: 2016-09-20 20:38:02.049725, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1682
> 24.5dcdfed6 100000004ff.00000000 [write 0~9250] snapc 1=[]
> ondisk+write e320)
> -25> 2016-09-20 20:38:02.049761 708f68c64700 5 -- op tracker --
> seq: 1969, time: 2016-09-20 20:38:02.049760, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1685
> 24.e9038fe6 10000000502.00000000 [write 0~178] snapc 1=[] ondisk+write
> e320)
> -24> 2016-09-20 20:38:02.049767 708f68c64700 5 -- op tracker --
> seq: 1976, time: 2016-09-20 20:38:02.049767, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1692
> 24.8d10ba3b 10000000509.00000000 [write 0~668] snapc 1=[] ondisk+write
> e320)
> -23> 2016-09-20 20:38:02.049773 708f68c64700 5 -- op tracker --
> seq: 1950, time: 2016-09-20 20:38:02.049773, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1666
> 24.3df0d155 100000004ef.00000000 [write 0~3603] snapc 1=[]
> ondisk+write e320)
> -22> 2016-09-20 20:38:02.049779 708f68c64700 5 -- op tracker --
> seq: 1978, time: 2016-09-20 20:38:02.049779, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1694
> 24.c18fac18 1000000050c.00000000 [write 0~6468] snapc 1=[]
> ondisk+write e320)
> -21> 2016-09-20 20:38:02.049790 708f68c64700 5 -- op tracker --
> seq: 1988, time: 2016-09-20 20:38:02.049790, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1704
> 24.f1a7cf6d 10000000516.00000000 [write 0~115] snapc 1=[] ondisk+write
> e320)
> -20> 2016-09-20 20:38:02.049796 708f68c64700 5 -- op tracker --
> seq: 1946, time: 2016-09-20 20:38:02.049796, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1662
> 24.2d47992e 100000004ea.00000000 [write 0~12562] snapc 1=[]
> ondisk+write e320)
> -19> 2016-09-20 20:38:02.049802 708f68c64700 5 -- op tracker --
> seq: 1990, time: 2016-09-20 20:38:02.049802, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1706
> 24.96561485 10000000518.00000000 [write 0~11354] snapc 1=[]
> ondisk+write e320)
> -18> 2016-09-20 20:38:02.049810 708f68c64700 5 -- op tracker --
> seq: 1974, time: 2016-09-20 20:38:02.049810, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1690
> 24.65b5bb82 10000000507.00000000 [write 0~6552] snapc 1=[]
> ondisk+write e320)
> -17> 2016-09-20 20:38:02.049838 708f68c64700 5 -- op tracker --
> seq: 1983, time: 2016-09-20 20:38:02.049838, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1699
> 24.34dbf1e0 10000000511.00000000 [write 0~369] snapc 1=[] ondisk+write
> e320)
> -16> 2016-09-20 20:38:02.049845 708f68c64700 5 -- op tracker --
> seq: 1977, time: 2016-09-20 20:38:02.049845, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1693
> 24.f62eaa57 1000000050b.00000000 [write 0~6533] snapc 1=[]
> ondisk+write e320)
> -15> 2016-09-20 20:38:02.049852 708f68c64700 5 -- op tracker --
> seq: 1973, time: 2016-09-20 20:38:02.049852, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1689
> 24.a42a2e12 10000000506.00000000 [write 0~9490] snapc 1=[]
> ondisk+write e320)
> -14> 2016-09-20 20:38:02.049862 708f68c64700 5 -- op tracker --
> seq: 1984, time: 2016-09-20 20:38:02.049862, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1700
> 24.d91a295d 10000000512.00000000 [write 0~15083] snapc 1=[]
> ondisk+write e320)
> -13> 2016-09-20 20:38:02.049874 708f68c64700 5 -- op tracker --
> seq: 1961, time: 2016-09-20 20:38:02.049874, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1677
> 24.f80dc995 100000004fa.00000000 [write 0~650] snapc 1=[] ondisk+write
> e320)
> -12> 2016-09-20 20:38:02.049880 708f68c64700 5 -- op tracker --
> seq: 1992, time: 2016-09-20 20:38:02.049880, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1708
> 24.2e11c03c 1000000051a.00000000 [write 0~46616] snapc 1=[]
> ondisk+write e320)
> -11> 2016-09-20 20:38:02.049890 708f68c64700 5 -- op tracker --
> seq: 1958, time: 2016-09-20 20:38:02.049889, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1674
> 24.b6af1e7d 100000004f7.00000000 [write 0~39735] snapc 1=[]
> ondisk+write e320)
> -10> 2016-09-20 20:38:02.049897 708f68c64700 5 -- op tracker --
> seq: 1994, time: 2016-09-20 20:38:02.049897, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1710
> 24.d68bbdf7 1000000051c.00000000 [write 0~227] snapc 1=[] ondisk+write
> e320)
> -9> 2016-09-20 20:38:02.049903 708f68c64700 5 -- op tracker --
> seq: 1985, time: 2016-09-20 20:38:02.049903, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1701
> 24.cd6859d6 10000000513.00000000 [write 0~15879] snapc 1=[]
> ondisk+write e320)
> -8> 2016-09-20 20:38:02.049913 708f68c64700 5 -- op tracker --
> seq: 1987, time: 2016-09-20 20:38:02.049913, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1703
> 24.8231fcf6 10000000515.00000000 [write 0~73] snapc 1=[] ondisk+write
> e320)
> -7> 2016-09-20 20:38:02.049920 708f68c64700 5 -- op tracker --
> seq: 1993, time: 2016-09-20 20:38:02.049919, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1709
> 24.69a8aa5 1000000051b.00000000 [write 0~5728] snapc 1=[] ondisk+write
> e320)
> -6> 2016-09-20 20:38:02.049926 708f68c64700 5 -- op tracker --
> seq: 1991, time: 2016-09-20 20:38:02.049925, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1707
> 24.50655161 10000000519.00000000 [write 0~8081] snapc 1=[]
> ondisk+write e320)
> -5> 2016-09-20 20:38:02.049936 708f68c64700 5 -- op tracker --
> seq: 1996, time: 2016-09-20 20:38:02.049936, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1712
> 24.15fdb7dd 1000000051e.00000000 [write 0~349] snapc 1=[] ondisk+write
> e320)
> -4> 2016-09-20 20:38:02.049942 708f68c64700 5 -- op tracker --
> seq: 1972, time: 2016-09-20 20:38:02.049941, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1688
> 24.64558fe4 10000000505.00000000 [write 0~12267] snapc 1=[]
> ondisk+write e320)
> -3> 2016-09-20 20:38:02.049948 708f68c64700 5 -- op tracker --
> seq: 1975, time: 2016-09-20 20:38:02.049948, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1691
> 24.c0cae786 10000000508.00000000 [write 0~1397] snapc 1=[]
> ondisk+write e320)
> -2> 2016-09-20 20:38:02.050013 708f68c64700 5 -- op tracker --
> seq: 1995, time: 2016-09-20 20:38:02.050013, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1711
> 24.4ad66977 1000000051d.00000000 [write 0~1172] snapc 1=[]
> ondisk+write e320)
> -1> 2016-09-20 20:38:02.050019 708f68c64700 5 -- op tracker --
> seq: 2000, time: 2016-09-20 20:38:02.050019, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1716
> 24.3693e8e2 10000000522.00000000 [write 0~24056] snapc 1=[]
> ondisk+write e320)
> 0> 2016-09-20 20:38:02.050025 708f68c64700 5 -- op tracker --
> seq: 1999, time: 2016-09-20 20:38:02.050024, event:
> journaled_completion_queued, op: osd_op(client.104119.1:1715
> 24.2141106f 10000000521.00000000 [write 0~1503] snapc 1=[]
> ondisk+write e320)
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 rbd_mirror
> 0/ 5 rbd_replay
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 1/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/10 civetweb
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> 0/ 0 refs
> 1/ 5 xio
> 1/ 5 compressor
> 1/ 5 newstore
> 1/ 5 bluestore
> 1/ 5 bluefs
> 1/ 3 bdev
> 1/ 5 kstore
> 4/ 5 rocksdb
> 4/ 5 leveldb
> 1/ 5 kinetic
> 1/ 5 fuse
> -2/-2 (syslog threshold)
> -1/-1 (stderr threshold)
> max_recent 10000
> max_new 1000
> log_file /var/log/ceph/ceph-osd.0.log
> --- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
