Hello, I am experiencing totally weird filesystem corruptions with the following setup: * Ceph infernalis on Debian8 * 10 OSDs (5 hosts) with spinning disks * 4 OSDs (1 host, with SSDs) The SSDs are new in my setup and I am trying to setup a Cache tier. Now, with the spinning disks Ceph is running since about a year without any major issues. Replacing disks and all that went fine. Ceph is used by rbd+libvirt+kvm with rbd_cache = true rbd_cache_writethrough_until_flush = true rbd_cache_size = 128M rbd_cache_max_dirty = 96M Also, in libvirt, I have cachemode=writeback enabled. So far so good. Now, I've added the SSD-Cache tier to the picture with "cache-mode writeback" The SSD-Machine also has "deadline" scheduler enabled. Suddenly VMs start to corrupt their filesystems (all ext4) with "Journal failed". Trying to reboot the machines ends in "No bootable drive" Using parted and testdisk on the image mapped via rbd reveals that the partition table is gone. testdisk finds the proper ones, e2fsck repairs the filesystem beyond usage afterwards. This does not happen to all machines, It happens to those that actually do some or most fo the IO elasticsearch, MariaDB+Galera, postgres, backup, GIT So I thought, yesterday one of my ldap-servers died, and that one is not doing IO. Could it be that rbd caching + qemu writeback cache + ceph cach tier writeback are not playing well together? I've read through some older mails on the list, where people had similar problems and suspected somehting like that. What are the proper/right settings for rdb/qemu/libvirt? libvirt: cachemode=none (writeback?) rdb: cache_mode = none SSD-tier: cachemode: writeback ? Thanks for any help, udo.
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
