On Thu, 30 Nov 2023, Eric Wheeler wrote: > On Thu, 30 Nov 2023, Joe Thornber wrote: > > On Wed, Nov 29, 2023 at 7:51 PM Eric Wheeler <dm-devel@xxxxxxxxxxxxxxxxxx> wrote: > > On Wed, 29 Nov 2023, Joe Thornber wrote: > > > On Wed, Nov 29, 2023 at 8:24 AM Joe Thornber <thornber@xxxxxxxxxx> wrote: > > > Hi Eric, > > > > > > Since integrity is below thin I would say this is likely a real issue > > > with your hardware; probably causing thinp to head down the rarely used > > > error path. > > > > While I suppose this is possible, the problem presented itself on two > > completely different NVMe mirror pairs, two NVMes of which are brand new. > > The system survived a 24 hour memory test and we have not seen any machine > > check exceptions or ECC errors, so is far as I can tell the hardware is > > working as expected. > > > > > > Have you performed any long running tests with just dm-integrity and not dm-thin? > > It ran fine for about 12 hours and then IOs hung shortly after we did a > reserve_metadata_snap. More info: Not only did it run for 12 hours just fine before IOs hung at 3:30am when we do metadata dumps, but this happened twice, on consecutive days at the same time of day (~3:30am), when cron does a reserve_metadata_snap for metadata thin_dumps of inactive snapshots. In each case we reformated dm-integrity the afternoon before, and ran an `integritysetup --integrity-recalculate-reset` to make sure it was clean. The first day we tried was on an older set of NVMe disks, but the second day we tried with brand new NVMe's that had never been used. -- Eric Wheeler
