Thank you for this information. I was finally able to detach all backing devices by (carefully) over writing the dirty blocks in writethrough mode. Once I did that the remaining backing devices became clean and I was able to detach them. I did not see any "split failed" messages while running a 3.15 kernel. --Larkin On 7/16/2014 10:09 PM, Jianjian Huo wrote: > So you still see "btree split failed" errors in your new 3.15 kernel? > > All the data items stored in cache are in B+ tree structure. You can > use the bcache debugfs file to find out where your dirty data items > are, just > > cat /sys/kernel/debug/bcache/<your_cache_set_uuid> | grep dirty > > On the left side of each item, it shows the data size and starting > location of backing device, the right side shows the stored location > in cache device. > > -Jianjian > > On Wed, Jul 16, 2014 at 8:04 AM, Larkin Lowrey > <llowrey@xxxxxxxxxxxxxxxxx> wrote: >> Thank you for your reply. >> >> I am now running kernel 3.15.4-200.fc20.x86_64 and the problem still >> exists. If the bad record could be found could it be repaired manually? >> >> The stuck backing devices total 17TB and I don't have that much spare >> capacity to relocate the data so I am motivated to try to repair >> in-place rather than wipe and rebuild. If you have any tips for where to >> look in the source code for the on-disk record format for the cache >> store and what to look for to identify the bad record(s) I would greatly >> appreciate it. >> >> Memory on this box is only lightly used. Of the 16GB, there is never >> less than 10GB free (free + buffers + cache). >> >> Here's a stack dump of the bcache_writebac thread: >> >> task: ffff880408ce4f00 ti: ffff8803f4eb4000 task.ti: ffff8803f4eb4000 >> RIP: 0010:[<ffffffffa0005c11>] [<ffffffffa0005c11>] >> refill_keybuf_fn+0x61/0x1d0 [bcache] >> RSP: 0018:ffff8803f4eb7b78 EFLAGS: 00000246 >> RAX: 0000000000000001 RBX: ffff8803fdc13000 RCX: ffff8803f4eb7e40 >> RDX: 0000000023ad5e98 RSI: ffff8802820a41b8 RDI: ffff8803ff830bc8 >> RBP: ffff8803f4eb7b78 R08: ffff880402520000 R09: 0000000000000001 >> R10: ffff88040fbdc000 R11: 000007ffffffffff R12: ffff8803f4eb7d70 >> R13: 0000000000000000 R14: ffff8803f4eb7d00 R15: ffff8803fdc13000 >> FS: 00007fecc4d59700(0000) GS:ffff880427c00000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b >> CR2: 00007fa1adad3000 CR3: 00000003f17dc000 CR4: 00000000000007f0 >> Stack: >> ffff8803f4eb7c30 ffffffffa0008292 00000001d06df860 ffffffffa0005bb0 >> ffff8803fdc130c8 ffff880131682990 ffff88040cc0e8c8 0000000000000004 >> 0000000000000001 ffff8802820a41d0 ffff8802820cc118 ffff880402520000 >> Call Trace: >> [<ffffffffa0008292>] bch_btree_map_keys_recurse+0x72/0x1d0 [bcache] >> [<ffffffffa0005bb0>] ? btree_insert_key+0xe0/0xe0 [bcache] >> [<ffffffffa0007ed5>] ? bch_btree_node_get+0xd5/0x290 [bcache] >> [<ffffffffa0008327>] bch_btree_map_keys_recurse+0x107/0x1d0 [bcache] >> [<ffffffffa0005bb0>] ? btree_insert_key+0xe0/0xe0 [bcache] >> [<ffffffffa000b3f7>] bch_btree_map_keys+0x127/0x150 [bcache] >> [<ffffffffa0005bb0>] ? btree_insert_key+0xe0/0xe0 [bcache] >> [<ffffffffa00209b0>] ? bch_crc64+0x50/0x50 [bcache] >> [<ffffffffa000b4d4>] bch_refill_keybuf+0xb4/0x220 [bcache] >> [<ffffffff810d0350>] ? abort_exclusive_wait+0xb0/0xb0 >> [<ffffffffa00209b0>] ? bch_crc64+0x50/0x50 [bcache] >> [<ffffffffa0021204>] bch_writeback_thread+0x154/0x840 [bcache] >> [<ffffffffa00210b0>] ? write_dirty_finish+0x260/0x260 [bcache] >> [<ffffffff810ac538>] kthread+0xd8/0xf0 >> [<ffffffff810ac460>] ? insert_kthread_work+0x40/0x40 >> [<ffffffff816ff23c>] ret_from_fork+0x7c/0xb0 >> [<ffffffff810ac460>] ? insert_kthread_work+0x40/0x40 >> >> When this snapshot was taken both bcache_writebac processes had similar >> stack traces but one was writing data and the other was not. The one >> that was writing was only writing at 3-4 4K blocks per second. I tried >> setting writeback_rate to a very large value and writeback_percent to >> zero but neither changed the rate. >> >> The backing device that is now writing does not write consistently. It >> will be idle for many hours then spontaneously start writing at ~4 4K >> blocks per second for several hours then stop again. >> >> --Larkin >> >> On 7/16/2014 1:27 AM, Samuel Huo wrote: >>> It seems your original 3.14 kernel hit the "btree split failed" bug >>> when bcache wrote back some dirty data into backing device and >>> inserted keys to invalidate those dirty data in btree index. But I >>> think that shouldn't prevent future writeback. >>> The "btree split failed" problem gets fixed in 3.15, as Slava >>> suggested in another post. >>> >>> You said writeback takes near 100% CPU, can you try to find out where >>> the writeback thread gets stuck at? and how is your memory usage? >>> >>> -Jianjian >>> >>> On Mon, Jul 14, 2014 at 5:47 AM, Larkin Lowrey >>> <llowrey@xxxxxxxxxxxxxxxxx> wrote: >>>> I am still unable to detach either backing device. There is still dirty >>>> data and it's not being written out. The bcache_writebac processes are >>>> at max CPU and the backing devices are idle. It appears to me that >>>> something is corrupt in the cache device which prevents the full dirty >>>> data from being written. >>>> >>>> Is there anything useful I can do to debug? >>>> >>>> Several GB of dirty data had accumulated since the data reported below. >>>> I set writeback_percent to zero in order to flush the dirty data and >>>> most did but not all. In the case of md1 it had grown to 6GB but only >>>> dropped to 2GB (it had been as low as 1.1GB a few days ago). The md3 >>>> device had 1.5GB but only dropped to 696KB (the same as the minimum from >>>> a few days ago). I have switch to writethrough mode in hopes that I >>>> don't get into further trouble. >>>> >>>> Attempting to detach (via /sys/block/md?/bcache/detach) has had no effect. >>>> >>>> Any help would be much appreciated! >>>> >>>> --Larkin >>>> >>>> On 7/8/2014 12:18 PM, Larkin Lowrey wrote: >>>>> %CPU %MEM TIME+ COMMAND >>>>> 98.8 0.0 645:05.50 bcache_writebac >>>>> 97.2 0.0 600:50.18 bcache_writebac >>>>> >>>>> The machine in question is a backup server. Backups usually take >>>>> 40-60min but this morning took 8.5-9 hours. I did make several changes >>>>> yesterday. >>>>> >>>>> There is a single (md raid 1) cache device and I had 2 md raid6 arrays >>>>> attached. Yesterday I attached a third raid6. >>>>> >>>>> Since noticing the high CPU usage (and poor IO performance) I attempted >>>>> to detach each of the three backing devices with the idea that I would >>>>> rebuild the cache set. One of the backing devices did detach but two >>>>> have not. One of the two remaining devices has 1GB of dirty data and the >>>>> other has 696KB. >>>>> >>>>> I had been running kernel 3.14.9-200.fc20.x86_64 but once this high load >>>>> situation happened I switched to 3.15.3-200.fc20.x86_64. >>>>> >>>>> When I reboot the two bcache_writebac processes start out immediately at >>>>> high load. >>>>> >>>>> /sys/block/md[13]/bcache/writeback_percent start out at 10%. >>>>> >>>>> There is no IO other than to the root fs which is on a separate raid1. >>>>> >>>>> ==> /sys/block/md1/bcache/writeback_rate_debug <== >>>>> rate: 512/sec >>>>> dirty: 1.1G >>>>> target: 15.5G >>>>> proportional: -13.7M >>>>> derivative: 0 >>>>> change: -13.7M/sec >>>>> next io: -62ms >>>>> >>>>> ==> /sys/block/md3/bcache/writeback_rate_debug <== >>>>> rate: 512/sec >>>>> dirty: 696k >>>>> target: 4.4G >>>>> proportional: -4.2M >>>>> derivative: 0 >>>>> change: -4.2M/sec >>>>> next io: 0ms >>>>> >>>>> After switching writeback_percent to 0%, there is still no IO and... >>>>> >>>>> ==> /sys/block/md1/bcache/writeback_rate_debug <== >>>>> rate: 512/sec >>>>> dirty: 1.1G >>>>> target: 15.5G >>>>> proportional: -13.7M >>>>> derivative: 0 >>>>> change: -13.7M/sec >>>>> next io: -50ms >>>>> >>>>> ==> /sys/block/md3/bcache/writeback_rate_debug <== >>>>> rate: 512/sec >>>>> dirty: 696k >>>>> target: 4.4G >>>>> proportional: -4.2M >>>>> derivative: 0 >>>>> change: -4.2M/sec >>>>> next io: 0ms >>>>> >>>>> ... and switching back to 10%, still no IO and... >>>>> >>>>> ==> /sys/block/md1/bcache/writeback_rate_debug <== >>>>> rate: 512/sec >>>>> dirty: 1.1G >>>>> target: 15.5G >>>>> proportional: -13.7M >>>>> derivative: 0 >>>>> change: -13.7M/sec >>>>> next io: -67ms >>>>> >>>>> ==> /sys/block/md3/bcache/writeback_rate_debug <== >>>>> rate: 512/sec >>>>> dirty: 696k >>>>> target: 4.4G >>>>> proportional: -4.2M >>>>> derivative: 0 >>>>> change: -4.2M/sec >>>>> next io: 0ms >>>>> >>>>> The only log messages for bcache are: >>>>> >>>>> [ 23.728302] bcache: register_bdev() registered backing device md2 >>>>> [ 23.750124] bcache: register_bdev() registered backing device md1 >>>>> [ 23.776249] bcache: register_bdev() registered backing device md3 >>>>> [ 25.414604] bcache: bch_journal_replay() journal replay done, 66 keys >>>>> in 10 entries, seq 26249932 >>>>> [ 25.534038] bcache: bch_cached_dev_attach() Caching md3 as bcache2 on >>>>> set 66c1a39b-5898-4aae-abe4-6ab609c18155 >>>>> [ 25.682785] bcache: bch_cached_dev_attach() Caching md1 as bcache1 on >>>>> set 66c1a39b-5898-4aae-abe4-6ab609c18155 >>>>> [ 25.695961] bcache: register_cache() registered cache device dm-2 >>>>> >>>>> Status (shortly after reboot): >>>>> >>>>> # bcache-status -s >>>>> --- bcache --- >>>>> UUID 66c1a39b-5898-4aae-abe4-6ab609c18155 >>>>> Block Size 4.00 KiB >>>>> Bucket Size 2.0 MiB >>>>> Congested? False >>>>> Read Congestion 2.0ms >>>>> Write Congestion 20.0ms >>>>> Total Cache Size 200 GiB >>>>> Total Cache Used 148 GiB (74%) >>>>> Total Cache Unused 52 GiB (26%) >>>>> Evictable Cache 200 GiB (100%) >>>>> Replacement Policy [lru] fifo random >>>>> Cache Mode writethrough [writeback] writearound none >>>>> Total Hits 6174 (94%) >>>>> Total Misses 380 >>>>> Total Bypass Hits 0 >>>>> Total Bypass Misses 0 >>>>> Total Bypassed 0 B >>>>> --- Backing Device --- >>>>> Device File /dev/md1 (9:1) >>>>> bcache Device File /dev/bcache1 (252:1) >>>>> Size 13 TiB >>>>> Cache Mode writethrough [writeback] writearound none >>>>> Readahead 0 >>>>> Sequential Cutoff 16.0 MiB >>>>> Merge sequential? False >>>>> State dirty >>>>> Writeback? True >>>>> Dirty Data 1 GiB >>>>> Total Hits 6108 (99%) >>>>> Total Misses 12 >>>>> Total Bypass Hits 0 >>>>> Total Bypass Misses 0 >>>>> Total Bypassed 0 B >>>>> --- Backing Device --- >>>>> Device File /dev/md3 (9:3) >>>>> bcache Device File /dev/bcache2 (252:2) >>>>> Size 4 TiB >>>>> Cache Mode writethrough [writeback] writearound none >>>>> Readahead 0 >>>>> Sequential Cutoff 16.0 MiB >>>>> Merge sequential? False >>>>> State dirty >>>>> Writeback? True >>>>> Dirty Data 696.00 KiB >>>>> Total Hits 66 (15%) >>>>> Total Misses 368 >>>>> Total Bypass Hits 0 >>>>> Total Bypass Misses 0 >>>>> Total Bypassed 0 B >>>>> --- Cache Device --- >>>>> Device File /dev/dm-2 (253:2) >>>>> Size 200 GiB >>>>> Block Size 4.00 KiB >>>>> Bucket Size 2.0 MiB >>>>> Replacement Policy [lru] fifo random >>>>> Discard? False >>>>> I/O Errors 0 >>>>> Metadata Written 2.0 MiB >>>>> Data Written 1.4 MiB >>>>> Buckets 102400 >>>>> Cache Used 148 GiB (74%) >>>>> Cache Unused 52 GiB (26%) >>>>> >>>>> I did find the following in the logs from a day prior to when I did the >>>>> work. The cacti graphs show high load at that time but a raid-check >>>>> started shortly before this which usually causes this kind of load. So, >>>>> the problem may have been occurring since the 6th without being noticing. >>>>> >>>>> Jul 6 00:26:32 mcp kernel: WARNING: CPU: 1 PID: 15568 at >>>>> drivers/md/bcache/btree.c:1979 btree_split+0x5bb/0x630 [bcache]() >>>>> Jul 6 00:26:32 mcp kernel: bcache: btree split failed >>>>> Jul 6 00:26:32 mcp kernel: Modules linked in: >>>>> Jul 6 00:26:32 mcp kernel: [30957.447979] bcache: btree split failed >>>>> Jul 6 00:26:32 mcp kernel: [30957.451654] Modules linked in: >>>>> binfmt_misc cfg80211 rfkill it87 hwmon_vid nf_conntrack_ipv4 ip6t_REJECT >>>>> nf_defrag_ipv4 nf_conntrack_ipv6 nf_defrag_ip >>>>> v6 xt_conntrack ip6table_filter nf_conntrack ip6_tables raid456 >>>>> async_raid6_recov async_memcpy async_pq async_xor async_tx kvm microcode >>>>> edac_core serio_raw sp5100_tco edac_mce_amd >>>>> k10temp snd_hda_codec_realtek i2c_piix4 snd_hda_codec_generic >>>>> snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_seq >>>>> snd_seq_device snd_pcm snd_timer snd r8169 soundcore >>>>> mii shpchp wmi acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd sunrpc btrfs >>>>> xor raid6_pq raid1 radeon i2c_algo_bit firewire_ohci drm_kms_helper >>>>> firewire_core ttm crc_itu_t mvsas drm l >>>>> ibsas scsi_transport_sas i2c_core cpufreq_stats bcache >>>>> Jul 6 00:26:32 mcp kernel: [30957.528183] CPU: 1 PID: 15568 Comm: >>>>> kworker/1:2 Not tainted 3.14.8-200.fc20.x86_64 #1 >>>>> Jul 6 00:26:32 mcp kernel: [30957.537550] Hardware name: Gigabyte >>>>> Technology Co., Ltd. GA-890GPA-UD3H/GA-890GPA-UD3H, BIOS F9 09/09/2011 >>>>> Jul 6 00:26:32 mcp kernel: binfmt_misc cfg80211 rfkill it87 hwmon_vid >>>>> nf_conntrack_ipv4 ip6t_REJECT nf_defrag_ipv4 nf_conntrack_ipv6 >>>>> nf_defrag_ipv6 xt_conntrack ip6table_filter nf >>>>> _conntrack ip6_tables raid456 async_raid6_recov async_memcpy async_pq >>>>> async_xor async_tx kvm microcode edac_core serio_raw sp5100_tco >>>>> edac_mce_amd k10temp snd_hda_codec_realtek i2c >>>>> _piix4 snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel >>>>> snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd >>>>> r8169 soundcore mii shpchp wmi acpi_cpufreq nfsd >>>>> auth_rpcgss nfs_acl lockd sunrpc btrfs xor raid6_pq raid1 radeon >>>>> i2c_algo_bit firewire_ohci drm_kms_helper firewire_core ttm crc_itu_t >>>>> mvsas drm libsas scsi_transport_sas i2c_core >>>>> cpufreq_stats bcache >>>>> Jul 6 00:26:32 mcp kernel: [30957.548771] Workqueue: events >>>>> write_dirty_finish [bcache] >>>>> Jul 6 00:26:32 mcp kernel: CPU: 1 PID: 15568 Comm: kworker/1:2 Not >>>>> tainted 3.14.8-200.fc20.x86_64 #1 >>>>> Jul 6 00:26:32 mcp kernel: [30957.555773] 0000000000000000 >>>>> 00000000dea1a02f ffff880007ffd8b0 ffffffff816f0502 >>>>> Jul 6 00:26:32 mcp kernel: Hardware name: Gigabyte Technology Co., Ltd. >>>>> GA-890GPA-UD3H/GA-890GPA-UD3H, BIOS F9 09/09/2011 >>>>> Jul 6 00:26:32 mcp kernel: [30957.564855] ffff880007ffd8f8 >>>>> ffff880007ffd8e8 ffffffff8108a1cd ffff88040e6b7800 >>>>> Jul 6 00:26:32 mcp kernel: Workqueue: events write_dirty_finish [bcache] >>>>> Jul 6 00:26:32 mcp kernel: 0000000000000000 00000000dea1a02f >>>>> ffff880007ffd8b0 ffffffff816f0502 >>>>> Jul 6 00:26:32 mcp kernel: ffff880007ffd8f8 ffff880007ffd8e8 >>>>> ffffffff8108a1cd ffff88040e6b7800 >>>>> Jul 6 00:26:32 mcp kernel: [30957.573943] ffffffffffffffe4 >>>>> ffff880007ffdd58 ffff880007ffd980 0000000000000000 >>>>> Jul 6 00:26:32 mcp kernel: [30957.583045] Call Trace: >>>>> Jul 6 00:26:32 mcp kernel: [30957.587031] [<ffffffff816f0502>] >>>>> dump_stack+0x45/0x56 >>>>> Jul 6 00:26:32 mcp kernel: ffffffffffffffe4 ffff880007ffdd58 >>>>> ffff880007ffd980 0000000000000000 >>>>> Jul 6 00:26:32 mcp kernel: Call Trace: >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff816f0502>] dump_stack+0x45/0x56 >>>>> Jul 6 00:26:32 mcp kernel: [30957.593729] [<ffffffff8108a1cd>] >>>>> warn_slowpath_common+0x7d/0xa0 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff8108a1cd>] >>>>> warn_slowpath_common+0x7d/0xa0 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff8108a24c>] warn_slowpath_fmt+0x5c/0x80 >>>>> Jul 6 00:26:32 mcp kernel: [30957.601287] [<ffffffff8108a24c>] >>>>> warn_slowpath_fmt+0x5c/0x80 >>>>> Jul 6 00:26:32 mcp kernel: [30957.608568] [<ffffffffa000965b>] >>>>> btree_split+0x5bb/0x630 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa000965b>] btree_split+0x5bb/0x630 >>>>> [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.616248] [<ffffffff816f3d22>] ? >>>>> __schedule+0x2e2/0x740 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff816f3d22>] ? __schedule+0x2e2/0x740 >>>>> Jul 6 00:26:32 mcp kernel: [30957.623240] [<ffffffff8136decd>] ? >>>>> list_del+0xd/0x30 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff8136decd>] ? list_del+0xd/0x30 >>>>> Jul 6 00:26:32 mcp kernel: [30957.629792] [<ffffffffa0009771>] >>>>> bch_btree_insert_node+0xa1/0x2c0 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.638293] [<ffffffffa000a724>] >>>>> btree_insert_fn+0x24/0x50 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa0009771>] >>>>> bch_btree_insert_node+0xa1/0x2c0 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa000a724>] >>>>> btree_insert_fn+0x24/0x50 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa0007861>] >>>>> bch_btree_map_nodes_recurse+0x51/0x180 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.646175] [<ffffffffa0007861>] >>>>> bch_btree_map_nodes_recurse+0x51/0x180 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.655171] [<ffffffffa000a700>] ? >>>>> bch_btree_insert_check_key+0x1b0/0x1b0 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa000a700>] ? >>>>> bch_btree_insert_check_key+0x1b0/0x1b0 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa000da62>] ? >>>>> bch_btree_ptr_invalid+0x12/0x20 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.664290] [<ffffffffa000da62>] ? >>>>> bch_btree_ptr_invalid+0x12/0x20 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.672814] [<ffffffffa000c48d>] ? >>>>> bch_btree_ptr_bad+0x4d/0x110 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa000c48d>] ? >>>>> bch_btree_ptr_bad+0x4d/0x110 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.681045] [<ffffffff816f6592>] ? >>>>> down_write+0x12/0x30 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff816f6592>] ? down_write+0x12/0x30 >>>>> Jul 6 00:26:32 mcp kernel: [30957.731889] [<ffffffff810b2d66>] ? >>>>> srcu_notifier_call_chain+0x16/0x20 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff810b2d66>] ? >>>>> srcu_notifier_call_chain+0x16/0x20 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffffa000b1e1>] >>>>> bch_btree_insert+0xf1/0x170 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.739905] [<ffffffffa000b1e1>] >>>>> bch_btree_insert+0xf1/0x170 [bcache] >>>>> Jul 6 00:26:32 mcp kernel: [30957.747908] [<ffffffff810d2180>] ? >>>>> abort_exclusive_wait+0xb0/0xb0 >>>>> Jul 6 00:26:32 mcp kernel: [<ffffffff810d2180>] ? >>>>> abort_exclusive_wait+0xb0/0xb0 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffffa0021d66>] >>>>> write_dirty_finish+0x1f6/0x260 [bcache] >>>>> Jul 6 00:26:33 mcp kernel: [30957.755602] [<ffffffffa0021d66>] >>>>> write_dirty_finish+0x1f6/0x260 [bcache] >>>>> Jul 6 00:26:33 mcp kernel: [30957.763908] [<ffffffff810c3e76>] ? >>>>> __dequeue_entity+0x26/0x40 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810c3e76>] ? >>>>> __dequeue_entity+0x26/0x40 >>>>> Jul 6 00:26:33 mcp kernel: [30957.771228] [<ffffffff810135d6>] ? >>>>> __switch_to+0x126/0x4c0 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810135d6>] ? __switch_to+0x126/0x4c0 >>>>> Jul 6 00:26:33 mcp kernel: [30957.778283] [<ffffffff810a68e6>] >>>>> process_one_work+0x176/0x430 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810a68e6>] >>>>> process_one_work+0x176/0x430 >>>>> Jul 6 00:26:33 mcp kernel: [30957.785598] [<ffffffff810a757b>] >>>>> worker_thread+0x11b/0x3a0 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810a757b>] worker_thread+0x11b/0x3a0 >>>>> Jul 6 00:26:33 mcp kernel: [30957.792656] [<ffffffff810a7460>] ? >>>>> rescuer_thread+0x3b0/0x3b0 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810a7460>] ? >>>>> rescuer_thread+0x3b0/0x3b0 >>>>> Jul 6 00:26:33 mcp kernel: [30957.799982] [<ffffffff810ae2d1>] >>>>> kthread+0xe1/0x100 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810ae2d1>] kthread+0xe1/0x100 >>>>> Jul 6 00:26:33 mcp kernel: [30957.806427] [<ffffffff810ae1f0>] ? >>>>> insert_kthread_work+0x40/0x40 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810ae1f0>] ? >>>>> insert_kthread_work+0x40/0x40 >>>>> Jul 6 00:26:33 mcp kernel: [30957.813995] [<ffffffff8170083c>] >>>>> ret_from_fork+0x7c/0xb0 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff8170083c>] ret_from_fork+0x7c/0xb0 >>>>> Jul 6 00:26:33 mcp kernel: [30957.820844] [<ffffffff810ae1f0>] ? >>>>> insert_kthread_work+0x40/0x40 >>>>> Jul 6 00:26:33 mcp kernel: [<ffffffff810ae1f0>] ? >>>>> insert_kthread_work+0x40/0x40 >>>>> Jul 6 00:26:33 mcp kernel: [30957.828357] ---[ end trace >>>>> 874ec8b4276a8f33 ]--- >>>>> Jul 6 00:26:33 mcp kernel: ---[ end trace 874ec8b4276a8f33 ]--- >>>>> Jul 6 00:26:33 mcp kernel: [30957.834379] bcache: bch_btree_insert() >>>>> error -12 >>>>> Jul 6 00:26:33 mcp kernel: bcache: bch_btree_insert() error -12 >>>>> >>>>> --Larkin >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
