On Tue, Feb 22, 2022 at 08:53:02AM +1100, Daniel Black wrote: > Per references at the bottom btfs + iouring manage to corrupt the > reading of a file. > > Using podman here however docker or another container runtime will > probably work. As will any MariaDB-10.6 on a distro with a native > liburing userspace. Apologies for the older and bloated container > image. > > Reproduction of bug: > > using a btrfs: > > $ dd if=/dev/zero of=../btrfs.blk bs=1M count=2K > $ sudo losetup --direct-io=on -f ../btrfs.blk > $ sudo mkfs.btrfs /dev/loop6 > $ sudo mount /dev/loop/6 /mnt/btrfstest > $ sudo mkdir /mnt/btrfstest/noaio > $ sudo chown dan: /mnt/btrfstest/noaio > > Initialize database on directory: > > $ podman run --name mdbinit --rm -v > /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > --innodb_use_native_aio=0 > > $ podman kill mdbinit > > Switch to using uring to read: > > $ podman run --rm -v /mnt/btrfstest/noaio/:/var/lib/mysql:Z -e > MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 > quay.io/danielgblack/mariadb-test:10.6-impish-sysbench > --innodb_use_native_aio=1 > > Failure observed on startup: > > 2022-02-21 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk > or a failed read of file './ibdata1' page [page id: space=0, page > number=9]. You may have to recover from a backup. I gave it a try, but it fails setting up io_uring: 2022-02-22 11:27:13 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts) 2022-02-22 11:27:13 0 [Warning] mysqld: io_uring_queue_init() failed with errno 1 2022-02-22 11:27:13 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF 2022-02-22 11:27:13 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 2022-02-22 11:27:13 0 [Note] InnoDB: Completed initialization of buffer pool So that's why it doesn't fail here, as it fallbacks to no aio mode. Any idea why it's failing to setup io_uring? I have the liburing2 and liburing-dev packages installed on debian, and tried with a 5.17-rc4 kernel. I can run fio with io_uring as the ioengine (works perferctly so far). Thanks. > > 2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): > > len 16384; hex 00000000000000092022-02-21 14:43:31 0 [ERROR] InnoDB: > Database page corruption on disk or a failed read of file './ibdata1' > page [page id: space=0, page number=243]. You may have to recover from > a backup. > > ffffffffffffff2022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii > and hex (16384 bytes): > > len 16384; hex > 00000000000000f3ffffff0000000000009c2045bf00ffffff0000000000ffffff00000000000000020171000900000000015500020006000700000000000000090000000000000000000000000500000000000000020572000000000000000204b208010000030085690000006e66696d756d000908a0d3000300000803000073757072656d756d000000000013080000100500a000000000000000000000000000000b5359535f464f524549474e18080000180500c0000000000000000c5359ffffff535f464fff0000524549474e5f434f4c531308ffffff0000200500ff0000db000000000000000d5359535f56495254550000000000000056414c00000000000020080000280501030000ffffffff0000ffffffff0000000000000000000e6d7973000000716c2f696e6e6f64625f7461626c655f7300ffffffff0000ffffffff0000000000746174732000ffff080000300501ffff0000ffffffff000005d6692b000000000000000f6d7973716c2f696e6e6f64625f696e6465785fd2007374617473220800003805015500000000000000106d7973716c2f7472616e73616374696f6e5f72656769737472791c0800004005000000f4ffffff74000000000000ffffff00116d79ffffffffffffffffffffffffffffffffffff73716c2f67ffffffffffffffffff7469645f736c6176655fffffffffffffffffff706f7300ffffffffffffff2022-02-21 > 14:43:31 0 [ERROR] InnoDB: Database page corruption on disk or a > failed read of file './mysql/innodb_table_stats.ibd' page [page id: > space=1, page number=0]. You may have to recover from a backup. > > 002022-02-21 14:43:31 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes): > > Without --innodb_use_native_aio=0 as a container argument this starts > without error. > > $ sudo losetup --direct-io=off -f btrfs.blk also exhibits the failure > > Observed failures in: > * 5.17.0-0.rc4.96.fc36.x86_64 > * 5.16.8 (on nixos) > * 5.15.6 > > No observed failure: > * 5.15.14-200.fc35.x86_64 > * 5.10 > > references: > * https://jira.mariadb.org/browse/MDEV-27900 > * https://github.com/NixOS/nixpkgs/issues/160516 > * https://jira.mariadb.org/browse/MDEV-27449
