Dan Merillat <dan.merillat@xxxxxxxxx> schrieb: > On Sat, Apr 11, 2015 at 3:52 AM, Kai Krakow <hurikhan77@xxxxxxxxx> wrote: >> Dan Merillat <dan.merillat@xxxxxxxxx> schrieb: >> >>> Looking through the kernel log, this may be related: I booted into >>> 4.0-rc7, and attempted to run it there at first: >>> Apr 7 12:54:08 fileserver kernel: [ 2028.533893] bcache-register: >>> page allocation failure: order:8, mode:0x >>> ... memory dump >>> Apr 7 12:54:08 fileserver kernel: [ 2028.541396] bcache: >>> register_cache() error opening sda4: cannot allocate memory >> >> Is your system under memory stress? Are you maybe using the huge memory >> page allocation policy in your kernel? If yes, could you retry without or >> at least set it to madvice mode? > > No, it's right after bootup, nothing heavy running yet. No idea why > memory is already so fragmented - it's something to do with 4.0-rc7, > since it never has had that problem on 3.18. I'm thinking of the same: something new in 4.0-*. And it is early boot, so huge memory should make no difference. Yet, I had problems with it when it was configured - memory became heavily fragmented and swapping a very slow process. So I set it to madvice policy and had no problems since. Maybe worth a try, tho still I couldn't imagine why this became a problem in early boot then. >>> Apr 7 12:55:29 fileserver kernel: [ 2109.303315] bcache: >>> run_cache_set() invalidating existing data >>> Apr 7 12:55:29 fileserver kernel: [ 2109.408255] bcache: >>> bch_cached_dev_attach() Caching md127 as bcache0 on set >>> 804d6906-fa80-40ac-9081-a71a4d595378 >> >> Why is it on md? I thought you are not using intermediate layers like >> LVM... > > The backing device is MD, the cdev is directly on sda4 Ah okay... >>> Apr 7 12:55:29 fileserver kernel: [ 2109.408443] bcache: >>> register_cache() registered cache device sda4 >>> Apr 7 12:55:33 fileserver kernel: [ 2113.307687] bcache: >>> bch_cached_dev_attach() Can't attach md127: already attached >> >> And why is it done twice? Something looks strange here... What is your >> device layout? > > 2100 seconds after boot? That's me doing it manually to try to figure > out why I can't access my filesystem. Oh, I didn't get the time difference here. >>> Apr 7 12:55:33 fileserver kernel: [ 2113.307747] bcache: >>> __cached_dev_store() Can't attach 804d6906-fa80-40ac-9081-a71a4d595378 >>> Apr 7 12:55:33 fileserver kernel: [ 2113.307747] : cache set not found >> >> My first guess would be that two different caches overlap and try to >> share the same device space. I had a similar problem after repartitioning >> because I did not "wipefs" the device first. > > I had to wipefs, it wouldn't let me create the bcache super until I did. Yes same here. But that is not what I meant. Some years back I created btrfs on raw device, then decided to prefer partitioning, did that, and created btrfs inside that partition. The kernel (or udev) saw two btrfs now. Another user here reported the same problem when creating bcache first on raw device, then in a partition. The kernel saw two bcache devices, with one being broken. The fix was to kill the superblock signature that was still lingering around in the raw device. But it would have just been easier to wipefs in the first place before partitioning. >> If you are using huge memory this may be an artifact of your initial >> finding. > > I'm not using it for anything, but it's configured. It's never given > this problem in 3.18, so something changed in 4.0. Maybe want to try without huge memory option? Just to sort things out? >>> So I rebooted to 4.0-rc7 again: >>> Apr 7 19:36:23 fileserver kernel: [ 2.145004] bcache: >>> journal_read_bucket() 157: too big, 552 bytes, offset 2047 >>> Apr 7 19:36:23 fileserver kernel: [ 2.154586] bcache: prio_read() >>> bad csum reading priorities >>> Apr 7 19:36:23 fileserver kernel: [ 2.154643] bcache: prio_read() >>> bad magic reading priorities >>> Apr 7 19:36:23 fileserver kernel: [ 2.158008] bcache: error on >>> 804d6906-fa80-40ac-9081-a71a4d595378: bad btree header at bucket >>> 65638, block 0, 0 keys, disabling caching >> >> Same here: If somehow two different caches overwrite each other, this >> could explain the problem. > > Possibly! So wipefs wasn't good enough, I should have done a discard > on the entire cdev > to make sure? See above... And I have another idea below: >>> Apr 7 19:36:23 fileserver kernel: [ 2.158408] bcache: >>> cache_set_free() Cache set 804d6906-fa80-40ac-9081-a71a4d595378 >>> unregistered >>> Apr 7 19:36:23 fileserver kernel: [ 2.158468] bcache: >>> register_cache() registered cache device sda4 >>> >>> Apr 7 19:36:23 fileserver kernel: [ 2.226581] md127: detected >>> capacity change from 0 to 12001954234368 >> >> I wonder where md127 comes from... Maybe bcache probing is running too >> early and should run after md setup. > > No, that's how udev works, it registers things as it finds them. So > on raw disks it finds > the bcache cdev, and registers it. Then it finds the raid signature > and sets it up. When the new md127 shows up, it finds the bdev > signature and registers that. Bog-standard setup, most people never > look this closely at the startup. I'd hope bcache wouldn't screw up > if its pieces get registered in a different order. I don't think that order is a problem. But I remember my md times back in kernel 2.2 when I used it to mirror two hard disks. The problem with md (and that's why I never again used it later and avoided it), at least at that time, was: Any software would see the same data through two devices: The md device and the underlying raw device. MD didn't hide it away the way bcache or lvm do it (by using a private superblock), it is simply dependent on a configuration file and some auto-detection through some partition signature. This is an artifact from the fact that you could easily migrate from single device to md raid device using md without backup/restore as outlined here: http://tldp.org/HOWTO/Software-RAID-HOWTO-7.html With this knowledge, I guess that bcache could probably detect its backing device signature twice - once through the underlying raw device and once through the md device. From your logs I'm not sure if they were complete enough to see that case. But to be sure I'd modify the udev rules to exclude the md parent devices from being run through probe-bcache. Otherwise all sorts of strange things may happen (like one process accessing the backing device through md, while bcache access it through the parent device - probably even on different mirror stripes). It's your setup, but personally I'd avoid MD for that reason and go with lvm. MD is just not modern, neither appropriate for modern system setups. It should really be just there for legacy setups and migration paths. I'm also not sure if MD is able to pass-through write barriers correctly which is needed to stay with consistent filesystems in case of crashes/reboots. Even LVM/device-mapper ignored them back some (more) kernel versions and I am not sure if they are respected for every target type as of now - why I always recommend to avoid them if not needed. I could also imagine that ignoring write-barriers (not passing them down to the hardware while the filesystem driver expected it to work) and using discard may lead to filesystem corruption upon reboots or crashes. -- Replies to list only preferred. -- To unsubscribe from this list: send the line "unsubscribe linux-bcache" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
