Re: OSDs with btrfs are down

[Jiri Kanicky <j@xxxxxxxxxx>]

Hi.

Correction. My SWAP is 3GB on SSD disk. I dont use th nodes for client 
stuff.

Thx Jiri

On 5/01/2015 01:21, Jiri Kanicky wrote:
> Hi,
>
> Here is my memory output. I use HP Microservers with 2GB RAM. Swap is 
> 500MB on SSD disk.
>
> cephadmin@ceph1:~$ free
>              total       used       free     shared    buffers cached
> Mem:       1885720    1817860      67860          0         32 694552
> -/+ buffers/cache:    1123276     762444
> Swap:      3859452     633492    3225960
>
> More googling took me to the following post:
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-June/040279.html
>
> Linux 3.14.1 is affected by serious Btrfs regression(s) that were 
> fixed in
> later releases.
>
> Unfortunately even latest Linux can crash and corrupt Btrfs file 
> system if
> OSDs are using snapshots (which is the default). Due to kernel bugs 
> related to
> Btrfs snapshots I also lost some OSDs until I found that snapshotting 
> can be
> disabled with "filestore btrfs snap = false".
>
>
> I am wondering if this can be the problem.
>
> -Jiri
>
>
> On 5/01/2015 01:17, Dyweni - BTRFS wrote:
> > Hi,
> >
> > BTRFS crashed because the system ran out of memory...
> >
> > I see these entries in your logs:
> >
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535661] kworker/0:2: page
> > > allocation failure: order:1, mode:0x204020
> > ....
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536112] BTRFS: error (device
> > > sdb1) in create_pending_snapshot:1334: errno=-12 Out of memory
> > ....
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536135] BTRFS: error (device
> > > sdb1) in cleanup_transaction:1577: errno=-12 Out of memory
> >
> >
> > How much memory do you have in this node?  Where you using Ceph
> > (as a client) on this node?  Do you have swap configured on this
> > node?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 2015-01-04 07:12, Jiri Kanicky wrote:
> > > Hi,
> > >
> > > My OSDs with btrfs are down on one node. I found the cluster in this 
> > > state:
> > >
> > > cephadmin@ceph1:~$ ceph osd tree
> > > # id    weight  type name       up/down reweight
> > > -1      10.88   root default
> > > -2      5.44            host ceph1
> > > 0       2.72                    osd.0   down    0
> > > 1       2.72                    osd.1   down    0
> > > -3      5.44            host ceph2
> > > 2       2.72                    osd.2   up      1
> > > 3       2.72                    osd.3   up      1
> > >
> > >
> > > cephadmin@ceph1:~$ ceph status
> > >     cluster bce2ff4d-e03b-4b75-9b17-8a48ee4d7788
> > >      health HEALTH_ERR 645 pgs degraded; 29 pgs inconsistent; 14 pgs
> > > recovering; 645 pgs stuck degraded; 768 pgs stuck unclean; 631 pgs
> > > stuck undersized; 631 pgs undersized; recovery 397226/915548 objects
> > > degraded (43.387%); 72026/915548 objects misplaced (7.867%); 783 scrub
> > > errors
> > >      monmap e1: 2 mons at
> > > {ceph1=192.168.30.21:6789/0,ceph2=192.168.30.22:6789/0}, election
> > > epoch 30, quorum 0,1 ceph1,ceph2
> > >      mdsmap e30: 1/1/1 up {0=ceph1=up:active}, 1 up:standby
> > >      osdmap e242: 4 osds: 2 up, 2 in
> > >       pgmap v38318: 768 pgs, 3 pools, 1572 GB data, 447 kobjects
> > >             1811 GB used, 3764 GB / 5579 GB avail
> > >             397226/915548 objects degraded (43.387%); 72026/915548
> > > objects misplaced (7.867%)
> > >                   14 active+recovering+degraded+remapped
> > >                  122 active+remapped
> > >                    1 active+remapped+inconsistent
> > >                  603 active+undersized+degraded
> > >                   28 active+undersized+degraded+inconsistent
> > >
> > >
> > > Would you know if this is pure BTRFS issue or is there any setting I
> > > forgot to use?
> > >
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535661] kworker/0:2: page
> > > allocation failure: order:1, mode:0x204020
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535669] CPU: 0 PID: 62644 Comm:
> > > kworker/0:2 Not tainted 3.16.0-0.bpo.4-amd64 #1 Debian
> > > 3.16.7-ckt2-1~bpo70+1
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535671] Hardware name: HP
> > > ProLiant MicroServer Gen8, BIOS J06 11/09/2013
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535701] Workqueue: events
> > > do_async_commit [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535704] 0000000000000000
> > > 0000000000000001 ffffffff81541f8f 0000000000204020
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535707] ffffffff811519ed
> > > 0000000000000001 ffff880075de0c00 0000000000000002
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535710] 0000000000000000
> > > 0000000000000001 ffff880075de0c08 0000000000000096
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535713] Call Trace:
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535720] [<ffffffff81541f8f>] ?
> > > dump_stack+0x41/0x51
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535725] [<ffffffff811519ed>] ?
> > > warn_alloc_failed+0xfd/0x160
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535730] [<ffffffff8115606f>] ?
> > > __alloc_pages_nodemask+0x91f/0xbb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535734] [<ffffffff8119fe00>] ?
> > > kmem_getpages+0x60/0x110
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535737] [<ffffffff811a1648>] ?
> > > fallback_alloc+0x158/0x220
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535741] [<ffffffff811a1f44>] ?
> > > kmem_cache_alloc+0x1a4/0x1e0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535745] [<ffffffff812c8fa0>] ?
> > > ida_pre_get+0x60/0xd0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535749] [<ffffffff811bdf61>] ?
> > > get_anon_bdev+0x21/0xe0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535762] [<ffffffffa015caef>] ?
> > > btrfs_init_fs_root+0xff/0x1b0 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535774] [<ffffffffa015de23>] ?
> > > btrfs_read_fs_root+0x33/0x40 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535785] [<ffffffffa015df06>] ?
> > > btrfs_get_fs_root+0xd6/0x230 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535796] [<ffffffffa0162383>] ?
> > > create_pending_snapshot+0x793/0xa00 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535807] [<ffffffffa0162679>] ?
> > > create_pending_snapshots+0x89/0xa0 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535818] [<ffffffffa016394a>] ?
> > > btrfs_commit_transaction+0x35a/0xa10 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535824] [<ffffffff8107a88e>] ?
> > > mod_timer+0x10e/0x220
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535834] [<ffffffffa016402a>] ?
> > > do_async_commit+0x2a/0x40 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535839] [<ffffffff810878fc>] ?
> > > process_one_work+0x15c/0x450
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535843] [<ffffffff81088b52>] ?
> > > worker_thread+0x112/0x540
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535847] [<ffffffff81088a40>] ?
> > > create_and_start_worker+0x60/0x60
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535851] [<ffffffff8108f511>] ?
> > > kthread+0xc1/0xe0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535854] [<ffffffff8108f450>] ?
> > > flush_kthread_worker+0xb0/0xb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535858] [<ffffffff815483bc>] ?
> > > ret_from_fork+0x7c/0xb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535861] [<ffffffff8108f450>] ?
> > > flush_kthread_worker+0xb0/0xb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535863] Mem-Info:
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535865] Node 0 DMA per-cpu:
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535867] CPU    0: hi: 0,
> > > btch:   1 usd:   0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535869] CPU    1: hi: 0,
> > > btch:   1 usd:   0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535870] Node 0 DMA32 per-cpu:
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535872] CPU    0: hi: 186,
> > > btch:  31 usd: 216
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535874] CPU    1: hi: 186,
> > > btch:  31 usd: 150
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535879] active_anon:156968
> > > inactive_anon:52877 isolated_anon:0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535879] active_file:117459
> > > inactive_file:98113 isolated_file:0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535879]  unevictable:0
> > > dirty:2167 writeback:0 unstable:0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535879]  free:18814
> > > slab_reclaimable:12533 slab_unreclaimable:5401
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535879]  mapped:3555 shmem:2
> > > pagetables:1364 bounce:0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535879]  free_cma:0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535883] Node 0 DMA free:7440kB
> > > min:380kB low:472kB high:568kB active_anon:0kB inactive_anon:52kB
> > > active_file:896kB inactive_file:132kB unevictable:0kB
> > > isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB
> > > mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB
> > > slab_reclaimable:792kB slab_unreclaimable:272kB kernel_stack:928kB
> > > pagetables:64kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> > > pages_scanned:0 all_unreclaimable? no
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535889] lowmem_reserve[]: 0 
> > > 1823 1823 1823
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535893] Node 0 DMA32
> > > free:67816kB min:44672kB low:55840kB high:67008kB active_anon:627872kB
> > > inactive_anon:211456kB active_file:468940kB inactive_file:392320kB
> > > unevictable:0kB isolated(anon):0kB isolated(file):0kB
> > > present:1914768kB managed:1869828kB mlocked:0kB dirty:8688kB
> > > writeback:0kB mapped:14216kB shmem:8kB slab_reclaimable:49340kB
> > > slab_unreclaimable:21332kB kernel_stack:6624kB pagetables:5392kB
> > > unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
> > > pages_scanned:279 all_unreclaimable? no
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535898] lowmem_reserve[]: 0 0 0 0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535902] Node 0 DMA: 34*4kB (M)
> > > 39*8kB (UEM) 85*16kB (UEM) 48*32kB (UEM) 30*64kB (UM) 1*128kB (M)
> > > 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB = 7440kB
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535914] Node 0 DMA32: 15762*4kB
> > > (UEM) 84*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
> > > 0*2048kB 1*4096kB (R) = 67816kB
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535925] Node 0 hugepages_total=0
> > > hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535927] 222415 total pagecache 
> > > pages
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535928] 6842 pages in swap cache
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535930] Swap cache stats: add
> > > 196373, delete 189531, find 57898/74929
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535932] Free swap  = 3508472kB
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535933] Total swap = 3859452kB
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535934] 482686 pages RAM
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535936] 0 pages 
> > > HighMem/MovableOnly
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535937] 11235 pages reserved
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535938] 0 pages hwpoisoned
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535943] ------------[ cut here
> > > ]------------
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535952] WARNING: CPU: 0 PID:
> > > 62644 at /build/linux-ax4Uh1/linux-3.16.7-ckt2/fs/btrfs/super.c:259
> > > __btrfs_abort_transaction+0x5f/0x140 [btrfs]()
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535954] BTRFS: Transaction
> > > aborted (error -12)
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535955] Modules linked in: nfsd
> > > auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop evdev
> > > x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp iTCO_wdt
> > > iTCO_vendor_support kvm_intel kvm crct10dif_pclmul crc32_pclmul
> > > ipmi_si ipmi_msghandler ghash_clmulni_intel cryptd thermal tg3 button
> > > processor ptp thermal_sys pps_core ttm libphy drm_kms_helper psmouse
> > > serio_raw pcspkr drm i2c_algo_bit ehci_pci i2c_core lpc_ich
> > > acpi_power_meter ehci_hcd mfd_core hpilo hpwdt btrfs xor raid6_pq
> > > sd_mod crc_t10dif crct10dif_common ahci libahci crc32c_intel libata
> > > scsi_mod xhci_hcd uhci_hcd usbcore usb_common
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535991] CPU: 0 PID: 62644 Comm:
> > > kworker/0:2 Not tainted 3.16.0-0.bpo.4-amd64 #1 Debian
> > > 3.16.7-ckt2-1~bpo70+1
> > > Jan  4 17:11:06 ceph1 kernel: [756636.535993] Hardware name: HP
> > > ProLiant MicroServer Gen8, BIOS J06 11/09/2013
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536002] Workqueue: events
> > > do_async_commit [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536004] 0000000000000000
> > > ffffffffa01e1a28 ffffffff81541f8f ffff880004bdbbb8
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536007] ffffffff8106cecc
> > > 00000000fffffff4 ffff8800561b9800 ffff88005f2a8c00
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536009] 0000000000000536
> > > ffffffffa01de080 ffffffff8106cfba ffffffffa01e1b08
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536012] Call Trace:
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536016] [<ffffffff81541f8f>] ?
> > > dump_stack+0x41/0x51
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536021] [<ffffffff8106cecc>] ?
> > > warn_slowpath_common+0x8c/0xc0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536025] [<ffffffff8106cfba>] ?
> > > warn_slowpath_fmt+0x4a/0x50
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536034] [<ffffffffa0134e0f>] ?
> > > __btrfs_abort_transaction+0x5f/0x140 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536045] [<ffffffffa01624b5>] ?
> > > create_pending_snapshot+0x8c5/0xa00 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536056] [<ffffffffa0162679>] ?
> > > create_pending_snapshots+0x89/0xa0 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536066] [<ffffffffa016394a>] ?
> > > btrfs_commit_transaction+0x35a/0xa10 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536070] [<ffffffff8107a88e>] ?
> > > mod_timer+0x10e/0x220
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536080] [<ffffffffa016402a>] ?
> > > do_async_commit+0x2a/0x40 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536084] [<ffffffff810878fc>] ?
> > > process_one_work+0x15c/0x450
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536088] [<ffffffff81088b52>] ?
> > > worker_thread+0x112/0x540
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536092] [<ffffffff81088a40>] ?
> > > create_and_start_worker+0x60/0x60
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536095] [<ffffffff8108f511>] ?
> > > kthread+0xc1/0xe0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536098] [<ffffffff8108f450>] ?
> > > flush_kthread_worker+0xb0/0xb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536101] [<ffffffff815483bc>] ?
> > > ret_from_fork+0x7c/0xb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536105] [<ffffffff8108f450>] ?
> > > flush_kthread_worker+0xb0/0xb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536107] ---[ end trace
> > > e977935ba9bc9ec6 ]---
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536112] BTRFS: error (device
> > > sdb1) in create_pending_snapshot:1334: errno=-12 Out of memory
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536130] BUG: unable to handle
> > > kernel NULL pointer dereference
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536130] BTRFS info (device
> > > sdb1): forced readonly
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536134] BTRFS warning (device
> > > sdb1): Skipping commit of aborted transaction.
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536135] BTRFS: error (device
> > > sdb1) in cleanup_transaction:1577: errno=-12 Out of memory
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536137] BTRFS info (device
> > > sdb1): delayed_refs has NO entry
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536188]  at 000000000000045c
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536196] IP: [<ffffffffa016f90d>]
> > > btrfs_orphan_cleanup+0x1d/0x420 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536227] PGD 581c6067 PUD 
> > > 30a70067 PMD 0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536249] Oops: 0002 [#1] SMP
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536267] Modules linked in: nfsd
> > > auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc loop evdev
> > > x86_pkg_temp_thermal intel_powerclamp intel_rapl coretemp iTCO_wdt
> > > iTCO_vendor_support kvm_intel kvm crct10dif_pclmul crc32_pclmul
> > > ipmi_si ipmi_msghandler ghash_clmulni_intel cryptd thermal tg3 button
> > > processor ptp thermal_sys pps_core ttm libphy drm_kms_helper psmouse
> > > serio_raw pcspkr drm i2c_algo_bit ehci_pci i2c_core lpc_ich
> > > acpi_power_meter ehci_hcd mfd_core hpilo hpwdt btrfs xor raid6_pq
> > > sd_mod crc_t10dif crct10dif_common ahci libahci crc32c_intel libata
> > > scsi_mod xhci_hcd uhci_hcd usbcore usb_common
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536587] CPU: 1 PID: 58876 Comm:
> > > ceph-osd Tainted: G        W     3.16.0-0.bpo.4-amd64 #1 Debian
> > > 3.16.7-ckt2-1~bpo70+1
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536627] Hardware name: HP
> > > ProLiant MicroServer Gen8, BIOS J06 11/09/2013
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536652] task: ffff880000068150
> > > ti: ffff88002e904000 task.ti: ffff88002e904000
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536688] RIP:
> > > 0010:[<ffffffffa016f90d>]  [<ffffffffa016f90d>]
> > > btrfs_orphan_cleanup+0x1d/0x420 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536741] RSP:
> > > 0018:ffff88002e907bb8  EFLAGS: 00010296
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536764] RAX: 0000000000000000
> > > RBX: fffffffffffffff4 RCX: ffff880049bd0278
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536800] RDX: 0000000000000001
> > > RSI: 0000000000000283 RDI: fffffffffffffff4
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536836] RBP: 0000000000000000
> > > R08: ffff88002e904000 R09: 0000000000000001
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536872] R10: 0000000000000000
> > > R11: 0000000000000078 R12: ffff88004c224dd0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536907] R13: ffff88004c224d28
> > > R14: ffff880070639cc0 R15: ffff880070639ce8
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536944] FS:
> > > 00007f6eab2b8700(0000) GS:ffff880072e20000(0000)
> > > knlGS:0000000000000000
> > > Jan  4 17:11:06 ceph1 kernel: [756636.536981] CS:  0010 DS: 0000 ES:
> > > 0000 CR0: 0000000080050033
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537004] CR2: 000000000000045c
> > > CR3: 000000005edd8000 CR4: 00000000001407e0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537040] Stack:
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537060] ffff88004a365800
> > > ffffffffa0164b60 ffff8800561b9d20 ffff880000068150
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537111] 0000000000000000
> > > 00ffffff00000000 ffff880000068150 ffffffff810b0120
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537162] ffff88002e907bf8
> > > ffff88005f291d98 0000000000000000 ffff88004c224dd0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537213] Call Trace:
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537245] [<ffffffffa0164b60>] ?
> > > btrfs_commit_transaction_async+0xe0/0x230 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537285] [<ffffffff810b0120>] ?
> > > __wake_up_sync+0x20/0x20
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537320] [<ffffffffa0197b10>] ?
> > > btrfs_mksubvol.isra.62+0x440/0x4f0 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537357] [<ffffffff810b0120>] ?
> > > __wake_up_sync+0x20/0x20
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537391] [<ffffffffa0197d7e>] ?
> > > btrfs_ioctl_snap_create_transid+0x1be/0x1d0 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537437] [<ffffffffa0197f35>] ?
> > > btrfs_ioctl_snap_create_v2+0x115/0x150 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537484] [<ffffffffa019b0db>] ?
> > > btrfs_ioctl+0x10cb/0x1eb0 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537519] [<ffffffffa0176062>] ?
> > > btrfs_file_write_iter+0x1a2/0x560 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537559] [<ffffffff811bb4db>] ?
> > > new_sync_write+0x7b/0xb0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537584] [<ffffffff811f7fac>] ?
> > > fsnotify+0x1cc/0x260
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537609] [<ffffffff811cdee6>] ?
> > > do_vfs_ioctl+0x86/0x4e0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537633] [<ffffffff811bddcb>] ?
> > > __sb_end_write+0x2b/0x70
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537658] [<ffffffff811bc326>] ?
> > > vfs_write+0x196/0x1f0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537682] [<ffffffff811ce3e1>] ?
> > > SyS_ioctl+0xa1/0xc0
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537707] [<ffffffff8154846d>] ?
> > > system_call_fast_compare_end+0x10/0x15
> > > Jan  4 17:11:06 ceph1 kernel: [756636.537731] Code: bd 00 00 48 89 43
> > > d0 eb c0 0f 1f 44 00 00 0f 1f 44 00 00 41 57 31 c0 ba 01 00 00 00 41
> > > 56 41 55 41 54 55 53 48 89 fb 48 83 ec 48 <f0> 0f b1 97 68 04 00 00 45
> > > 31 f6 85 c0 74 14 48 83 c4 48 44 89
> > > Jan  4 17:11:06 ceph1 kernel: [756636.538053] RIP [<ffffffffa016f90d>]
> > > btrfs_orphan_cleanup+0x1d/0x420 [btrfs]
> > > Jan  4 17:11:06 ceph1 kernel: [756636.538102]  RSP <ffff88002e907bb8>
> > > Jan  4 17:11:06 ceph1 kernel: [756636.538122] CR2: 000000000000045c
> > > Jan  4 17:11:06 ceph1 kernel: [756636.538543] ---[ end trace
> > > e977935ba9bc9ec7 ]---
> > >
> > > Thanks
> > > -Jiri
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@xxxxxxxxxxxxxx
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com