I have seen 2 layers of partitions "work" right after you force a
scan command, but on boot it seems to fail. Using only one partition
is much after.
I have also seen the 2 layers of partitions (a partition table on a
partition like you have) work and then suddenly stop working with an
update, so it can random cause issues and typically when I have seen
it done it was server absolutely no purpose. I have also seen (when
the partition table was an old dos table) a boot suddenly start
finding the empty partition table and refusing to find the LVM volume
on a disk.
So definitely get rid of the partition of a partion since it is
serving no purpose and seems to not be 100% reliable across
reboots/updates and such.
I was suspecting that n1p1 was a partition of a parition, but I was not sure.
On Wed, Aug 7, 2024 at 5:14 PM Ryan England <ryan.england@xxxxxxxxxxx> wrote:
>
> Hello Roger,
>
> Thanks for the response. That's correct. parted was previously run in
> order to generate /dev/nvme[0, 1, 2]n1p1. I'm not running parted
> against /dev/nvme[0, 1, 2]n1p1. Looking at the results from lsdrv, it
> does appear that there are two partitions on each of the NVMe drives.
> Should I delete the p1 part of each partition and use n1 as a part of
> the array, e.g. /dev/nvme0n1?
>
> The partitions were created before I got involved with this project.
> I've just been using them as is. The steps I've been using are
> included below.
>
> I was trying to provide as much information yesterday evening but only
> had access to my phone. I'll be sure to write these up from my laptop
> going forward. This will free me up to provide as many details as I
> can.
>
> Regards,
> Ryan E.
>
> Create Raid array with 3 drives
> - mdadm --create /dev/md127 --level=5 --raid-devices=3 /dev/nvme0n1p1
> /dev/nvme1n1p1 /dev/nvme2n1p1
>
> Check results
> - cat /proc/mdstat - await recovery
>
> Format the filesystem
> - mkfs.ext4 /dev/md127
>
> Write raid to mdadm.conf
> - mdadm --detail --scan | grep md127 >> /etc/mdadm/mdadm.conf
> - update-initramfs -u
>
> Check the raid
> - mdadm -D /dev/md127
>
> Mount and write to fstab
> - mount /dev/md127 /data1
>
> Find the UUID for /dev/md127
> - blkid /dev/md127
>
> Add new entry to /etc/fstab - Be sure to include nofail option to
> prevent emergency mode
> - vim /etc/fstab
>
> Test mount in /etc/fstab
> - umount /data1
> - mount -av
>
> On Tue, Aug 6, 2024 at 9:13 PM Roger Heflin <rogerheflin@xxxxxxxxx> wrote:
> >
> > Those steps are generic and a suggestion, and some of those commands
> > if misused would produce this behavior.
> >
> > You aren't doing any parted commands against /dev/nvme2n1p1 are you?
> > You are just running parted against /dev/nvme[01]n1 right?
> >
> > You likely need to run history and dump the exact commands, when it
> > goes wrong showing someone the theoretical work instruction is not
> > that useful because if you are asking what went wrong then something
> > in that work instruction was not quite done right and/or
> > misunderstood.
> >
> > On Tue, Aug 6, 2024 at 7:28 PM Ryan England <ryan.england@xxxxxxxxxxx> wrote:
> > >
> > > Hello Roger,
> > >
> > > Thanks for the update. The process is almost exactly as follows. The only difference is the command to create the array. I used mdadm --create /dev/md127 --level=5 --raid-devices=3 /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1.
> > >
> > > Regards,
> > > Ryan E.
> > >
> > >
> > > This would all be run as root or with sudo.
> > >
> > > IMPORTANT: (to confirm which drive is which)
> > >
> > > lsblk
> > >
> > > 1. New drives will not be partitioned
> > >
> > > 2. Sizes can also be used to determine which drive
> > >
> > > (we can use 'sudo smartctl -x /dev/nvme#' to see models, etc).
> > >
> > > NOTE: If using XFS, you need to install xfsprogs
> > >
> > > sudo apt-get update && sudo apt-get install xfsprogs
> > >
> > > You can use the more common ext4 also, it all depends on inode usage concerns
> > >
> > > performance is similar.
> > >
> > > To create raid 1 in live linux system
> > >
> > > $ sudo parted -a optimal /dev/nvme1
> > >
> > > $ sudo parted -a optimal /dev/nvme2
> > >
> > > etc ...
> > >
> > > And this will have options in the tool:
> > >
> > > # Place a flag gpt or mbr
> > >
> > > (parted) mklabel gpt
> > >
> > > # Create partitions
> > >
> > > (parted) mkpart primary ext4 0% 100%
> > >
> > > # Mark partition as software raid partition
> > >
> > > (parted) set 1 raid on
> > >
> > > # Verify its alligned
> > >
> > > (parted) align-check
> > >
> > > optimal
> > >
> > > # Show results
> > >
> > > print
> > >
> > > then repeat for each drive...
> > >
> > > ====================
> > >
> > > RAID setup examples:
> > >
> > > ====================
> > >
> > > Raid 0 array
> > >
> > > mdadm --create /dev/md0 --level=0 --raid-devices=2 /dev/nvme1n1p1 /dev/nvme2n1p1
> > >
> > > Or for Raid 1 array (only 2+ drives but all mirrors)
> > >
> > > mdadm --create /dev/md0 --level=1 --raid-devices=2 --spare=0 /dev/nvme1n1p1 /dev/nvme2n1p1
> > >
> > > Check results
> > >
> > > cat /proc/mdstat
> > >
> > > ======================
> > >
> > > Format the filesystem: (performance about the same just # of files)
> > >
> > > ======================
> > >
> > > # Format array - similar performance but depending on usage of millions of small files
> > >
> > > # - you may hit inode issues
> > >
> > > $ sudo mkfs.ext4 /dev/md0
> > >
> > > or
> > >
> > > # Format for XFS: - handles larger numbers of files better (millions) no inode issues
> > >
> > > $ sudo mkfs.xfs /dev/md0
> > >
> > > ============================
> > >
> > > Finish/check the Raid config
> > >
> > > ============================
> > >
> > > # Write raid to a mdadm.conf
> > >
> > > $ sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf
> > >
> > > # Update initramfs
> > >
> > > $ sudo update-initramfs -u
> > >
> > > # Check the raid:
> > >
> > > $ sudo mdadm -D /dev/md0
> > >
> > > ========================
> > >
> > > Mount and write to fstab
> > >
> > > ========================
> > >
> > > For example for a mount point of /data:
> > >
> > > $ sudo mkdir /data
> > >
> > > $ sudo chmod 1777 /data
> > >
> > > $ sudo mount /dev/md0 /data
> > >
> > > (And setup the /etc/fstab).
> > >
> > > # Find the UUID for the /dev/md0
> > >
> > > $ blkid /dev/md0
> > >
> > > # Add the md0's UUID to the /etc/fstab
> > >
> > > $ blkid /dev/md0
> > >
> > > /dev/md0: UUID="a6d39b9b-d4e4-4fa1-b98c-4a73e7ba0f83" TYPE="xfs" (or ext4)
> > >
> > > echo "# Add RAID md0 to mount on /data" | sudo tee -a /etc/fstab
> > >
> > > echo "UUID=a6d39b9b-d4e4-4fa1-b98c-4a73e7ba0f83 /data ext4 defaults,relatime,rw 0 2" | sudo tee -a /etc/fstab
> > >
> > > or if you choose xfs:
> > >
> > > echo "# Add RAID md0 to mount on /data" >> /etc/fstab
> > >
> > > echo "UUID=a6d39b9b-d4e4-4fa1-b98c-4a73e7ba0f83 /data xfs defaults,discard,relatime,rw 0 2" | sudo tee -a /etc/fstab
> > >
> > > mount -a
> > >
> > >
> > >
> > > On Tue, Aug 6, 2024, 7:57 PM Roger Heflin <rogerheflin@xxxxxxxxx> wrote:
> > >>
> > >> Several of the parts are indicating that the partition has a partition
> > >> table on it.
> > >>
> > >> Both the examine and the wipefs show that. The aa55 is a GPT
> > >> partition table and that WILL overwrite parts of the mdadm headers.
> > >>
> > >> What are the full steps that you are using to create the raid?
> > >>
> > >> On Tue, Aug 6, 2024 at 6:20 PM Ryan England <ryan.england@xxxxxxxxxxx> wrote:
> > >> >
> > >> > Hello everyone,
> > >> >
> > >> > I've been working on a system with a software RAID for the last couple
> > >> > of weeks. I ran through the process of creating the array as RAID5
> > >> > using /dev/nvme0n1p1, /dev/nvme1n1p1, and /dev/nvme2n1p1. I then
> > >> > create the filesystem, update mdadm.conf, and run update-initramfs -u.
> > >> >
> > >> > The array and file system are created successfully. It's created as
> > >> > /dev/md127. I mount it to the system and I can write data to it.
> > >> > /etc/fstab has also been updated.
> > >> >
> > >> > After rebooting the machine, the system enters Emergency Mode.
> > >> > Commenting out the newly created device and rebooting the machine
> > >> > brings it back to Emergency Mode. I can also skip EM by adding the
> > >> > nofail option to the mount point in /etc/fstab.
> > >> >
> > >> > Today, I walked through recreating the array. Once created, I ran
> > >> > mkfs.ext4 again. This time, I noticed that the command found an ext4
> > >> > file system. To try and repair it, I ran fsck -y against /dev/md127.
> > >> > The end of the fsck noted that a resize of the inode (re)creation
> > >> > failed: Inode checksum does not match inode. Mounting failed, so we
> > >> > made the filesystem again.
> > >> >
> > >> > It's worth noting that there's NO data on this array at this time.
> > >> > Hence why we were able to go through with making the filesystem again.
> > >> > I made sure to gather all of the info noted within the mdadm wiki and
> > >> > I've included that below. The only thing not included is mdadm
> > >> > --detail of each of the partitions because the system doesn't
> > >> > recognize them as being part of an md. Also, md0 hosts the root volume
> > >> > and isn't a part of the output below.
> > >> >
> > >> > As far as troubleshooting is concerned, I've tried the following:
> > >> > 1. mdadm --manage /dev/md127 --run
> > >> > 2. echo "clean" > /sys/block/md127/md/array_state & then run command 1
> > >> > 3. mdadm --assemble --force /dev/md127 /dev/nvme0n1p1 /dev/nvme1n1p1
> > >> > /dev/nvme2n1p1 & then run command 1
> > >> >
> > >> > I've also poured over logs. Once, I noticed that nvme2n1p1 wasn't
> > >> > being recognized as a part of the kernel logs. To rule that out as the
> > >> > issue, I created a RAID1 between nvme0n1p1 & nvme1n1p1. This still
> > >> > didn't work.
> > >> >
> > >> > Looking through journalctl -xb, I found an error noting a package that
> > >> > was missing. The package is named ibblockdev-mdraid2. Installing that
> > >> > package still didn't help.
> > >> >
> > >> > Lastly, I included the output of wipefs at the behest of a colleague.
> > >> > Any support you can provide will be greatly appreciated.
> > >> >
> > >> > Regards,
> > >> > Ryan E.
> > >> >
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > Start of the mdadm bug report log file.
> > >> >
> > >> > Date: Tue Aug 6 02:42:59 PM PDT 2024
> > >> > uname: Linux REDACTED 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
> > >> > 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
> > >> > command line flags:
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > mdadm --version
> > >> >
> > >> > mdadm - v4.2 - 2021-12-30
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > cat /proc/mdstat
> > >> >
> > >> > Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
> > >> > [raid4] [raid10]
> > >> > md0 : active raid1 sdb2[1] sda2[0] 1874715648 blocks super 1.2 [2/2]
> > >> > [UU] bitmap: 8/14 pages [32KB], 65536KB chunk unused devices: <none>
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > mdadm --examine /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1
> > >> >
> > >> > /dev/nvme0n1p1: MBR Magic : aa55 Partition[0] : 4294967295 sectors at
> > >> > 1 (type ee)
> > >> > /dev/nvme1n1p1: MBR Magic : aa55 Partition[0] : 4294967295 sectors at
> > >> > 1 (type ee)
> > >> > /dev/nvme2n1p1: MBR Magic : aa55 Partition[0] : 4294967295 sectors at
> > >> > 1 (type ee)
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > mdadm --detail /dev/nvme0n1p1 /dev/nvme1n1p1 /dev/nvme2n1p1
> > >> >
> > >> > mdadm: /dev/nvme0n1p1 does not appear to be an md device
> > >> > mdadm: /dev/nvme1n1p1 does not appear to be an md device
> > >> > mdadm: /dev/nvme2n1p1 does not appear to be an md device
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > smartctl --xall /dev/nvme0n1p1
> > >> >
> > >> > smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-117-generic] (local build)
> > >> > Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
> > >> >
> > >> > === START OF INFORMATION SECTION ===
> > >> > Model Number: SAMSUNG MZQL23T8HCLS-00A07
> > >> > Serial Number: S64HNS0TC05245
> > >> > Firmware Version: GDC5602Q
> > >> > PCI Vendor/Subsystem ID: 0x144d
> > >> > IEEE OUI Identifier: 0x002538
> > >> > Total NVM Capacity: 3,840,755,982,336 [3.84 TB]
> > >> > Unallocated NVM Capacity: 0
> > >> > Controller ID: 6
> > >> > NVMe Version: 1.4
> > >> > Number of Namespaces: 32
> > >> > Namespace 1 Size/Capacity: 3,840,755,982,336 [3.84 TB]
> > >> > Namespace 1 Utilization: 71,328,116,736 [71.3 GB]
> > >> > Namespace 1 Formatted LBA Size: 512
> > >> > Local Time is: Tue Aug 6 15:16:08 2024 PDT
> > >> > Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required
> > >> > Optional Admin Commands (0x005f): Security Format Frmw_DL NS_Mngmt
> > >> > Self_Test MI_Snd/Rec
> > >> > Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero
> > >> > Sav/Sel_Feat Timestmp
> > >> > Log Page Attributes (0x0e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
> > >> > Maximum Data Transfer Size: 512 Pages
> > >> > Warning Comp. Temp. Threshold: 80 Celsius
> > >> > Critical Comp. Temp. Threshold: 83 Celsius
> > >> > Namespace 1 Features (0x1a): NA_Fields No_ID_Reuse NP_Fields
> > >> >
> > >> > Supported Power States
> > >> > St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
> > >> > 0 + 25.00W 14.00W - 0 0 0 0 70 70
> > >> > 1 + 8.00W 8.00W - 1 1 1 1 70 70
> > >> >
> > >> > Supported LBA Sizes (NSID 0x1)
> > >> > Id Fmt Data Metadt Rel_Perf
> > >> > 0 + 512 0 0
> > >> > 1 - 4096 0 0
> > >> >
> > >> > === START OF SMART DATA SECTION ===
> > >> > SMART overall-health self-assessment test result: PASSED
> > >> >
> > >> > SMART/Health Information (NVMe Log 0x02)
> > >> > Critical Warning: 0x00
> > >> > Temperature: 35 Celsius
> > >> > Available Spare: 100%
> > >> > Available Spare Threshold: 10%
> > >> > Percentage Used: 0%
> > >> > Data Units Read: 31,574,989 [16.1 TB]
> > >> > Data Units Written: 304,488 [155 GB]
> > >> > Host Read Commands: 36,420,064
> > >> > Host Write Commands: 3,472,342
> > >> > Controller Busy Time: 63
> > >> > Power Cycles: 11
> > >> > Power On Hours: 5,582
> > >> > Unsafe Shutdowns: 9
> > >> > Media and Data Integrity Errors: 0
> > >> > Error Information Log Entries: 0
> > >> > Warning Comp. Temperature Time: 0
> > >> > Critical Comp. Temperature Time: 0
> > >> > Temperature Sensor 1: 35 Celsius
> > >> > Temperature Sensor 2: 44 Celsius
> > >> >
> > >> > Error Information (NVMe Log 0x01, 16 of 64 entries)
> > >> > No Errors Logged
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > smartctl --xall /dev/nvme1n1p1
> > >> >
> > >> > smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-117-generic] (local build)
> > >> > Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
> > >> >
> > >> > === START OF INFORMATION SECTION ===
> > >> > Model Number: SAMSUNG MZQL23T8HCLS-00A07
> > >> > Serial Number: S64HNS0TC05241
> > >> > Firmware Version: GDC5602Q
> > >> > PCI Vendor/Subsystem ID: 0x144d
> > >> > IEEE OUI Identifier: 0x002538
> > >> > Total NVM Capacity: 3,840,755,982,336 [3.84 TB]
> > >> > Unallocated NVM Capacity: 0
> > >> > Controller ID: 6
> > >> > NVMe Version: 1.4
> > >> > Number of Namespaces: 32
> > >> > Namespace 1 Size/Capacity: 3,840,755,982,336 [3.84 TB]
> > >> > Namespace 1 Utilization: 71,324,651,520 [71.3 GB]
> > >> > Namespace 1 Formatted LBA Size: 512
> > >> > Local Time is: Tue Aug 6 15:16:22 2024 PDT
> > >> > Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required
> > >> > Optional Admin Commands (0x005f): Security Format Frmw_DL NS_Mngmt
> > >> > Self_Test MI_Snd/Rec
> > >> > Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero
> > >> > Sav/Sel_Feat Timestmp
> > >> > Log Page Attributes (0x0e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
> > >> > Maximum Data Transfer Size: 512 Pages
> > >> > Warning Comp. Temp. Threshold: 80 Celsius
> > >> > Critical Comp. Temp. Threshold: 83 Celsius
> > >> > Namespace 1 Features (0x1a): NA_Fields No_ID_Reuse NP_Fields
> > >> >
> > >> > Supported Power States
> > >> > St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
> > >> > 0 + 25.00W 14.00W - 0 0 0 0 70 70
> > >> > 1 + 8.00W 8.00W - 1 1 1 1 70 70
> > >> >
> > >> > Supported LBA Sizes (NSID 0x1)
> > >> > Id Fmt Data Metadt Rel_Perf
> > >> > 0 + 512 0 0
> > >> > 1 - 4096 0 0
> > >> >
> > >> > === START OF SMART DATA SECTION ===
> > >> > SMART overall-health self-assessment test result: PASSED
> > >> >
> > >> > SMART/Health Information (NVMe Log 0x02)
> > >> > Critical Warning: 0x00
> > >> > Temperature: 34 Celsius
> > >> > Available Spare: 100%
> > >> > Available Spare Threshold: 10%
> > >> > Percentage Used: 0%
> > >> > Data Units Read: 24,073,787 [12.3 TB]
> > >> > Data Units Written: 7,805,460 [3.99 TB]
> > >> > Host Read Commands: 29,506,475
> > >> > Host Write Commands: 10,354,117
> > >> > Controller Busy Time: 64
> > >> > Power Cycles: 11
> > >> > Power On Hours: 5,582
> > >> > Unsafe Shutdowns: 9
> > >> > Media and Data Integrity Errors: 0
> > >> > Error Information Log Entries: 0
> > >> > Warning Comp. Temperature Time: 0
> > >> > Critical Comp. Temperature Time: 0
> > >> > Temperature Sensor 1: 34 Celsius
> > >> > Temperature Sensor 2: 44 Celsius
> > >> >
> > >> > Error Information (NVMe Log 0x01, 16 of 64 entries)
> > >> > No Errors Logged
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > smartctl --xall /dev/nvme2n1p1
> > >> >
> > >> > smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-117-generic] (local build)
> > >> > Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
> > >> >
> > >> > === START OF INFORMATION SECTION ===
> > >> > Model Number: SAMSUNG MZQL23T8HCLS-00A07
> > >> > Serial Number: S64HNS0TC05244
> > >> > Firmware Version: GDC5602Q
> > >> > PCI Vendor/Subsystem ID: 0x144d
> > >> > IEEE OUI Identifier: 0x002538
> > >> > Total NVM Capacity: 3,840,755,982,336 [3.84 TB]
> > >> > Unallocated NVM Capacity: 0
> > >> > Controller ID: 6
> > >> > NVMe Version: 1.4
> > >> > Number of Namespaces: 32
> > >> > Namespace 1 Size/Capacity: 3,840,755,982,336 [3.84 TB]
> > >> > Namespace 1 Utilization: 3,840,514,523,136 [3.84 TB]
> > >> > Namespace 1 Formatted LBA Size: 512
> > >> > Local Time is: Tue Aug 6 15:16:33 2024 PDT
> > >> > Firmware Updates (0x17): 3 Slots, Slot 1 R/O, no Reset required
> > >> > Optional Admin Commands (0x005f): Security Format Frmw_DL NS_Mngmt
> > >> > Self_Test MI_Snd/Rec
> > >> > Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero
> > >> > Sav/Sel_Feat Timestmp
> > >> > Log Page Attributes (0x0e): Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg
> > >> > Maximum Data Transfer Size: 512 Pages
> > >> > Warning Comp. Temp. Threshold: 80 Celsius
> > >> > Critical Comp. Temp. Threshold: 83 Celsius
> > >> > Namespace 1 Features (0x1a): NA_Fields No_ID_Reuse NP_Fields
> > >> >
> > >> > Supported Power States
> > >> > St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
> > >> > 0 + 25.00W 14.00W - 0 0 0 0 70 70
> > >> > 1 + 8.00W 8.00W - 1 1 1 1 70 70
> > >> >
> > >> > Supported LBA Sizes (NSID 0x1)
> > >> > Id Fmt Data Metadt Rel_Perf
> > >> > 0 + 512 0 0
> > >> > 1 - 4096 0 0
> > >> >
> > >> > === START OF SMART DATA SECTION ===
> > >> > SMART overall-health self-assessment test result: PASSED
> > >> >
> > >> > SMART/Health Information (NVMe Log 0x02)
> > >> > Critical Warning: 0x00
> > >> > Temperature: 33 Celsius
> > >> > Available Spare: 100%
> > >> > Available Spare Threshold: 10%
> > >> > Percentage Used: 0%
> > >> > Data Units Read: 33,340 [17.0 GB]
> > >> > Data Units Written: 24,215,921 [12.3 TB]
> > >> > Host Read Commands: 812,460
> > >> > Host Write Commands: 31,463,496
> > >> > Controller Busy Time: 50
> > >> > Power Cycles: 12
> > >> > Power On Hours: 5,582
> > >> > Unsafe Shutdowns: 9
> > >> > Media and Data Integrity Errors: 0
> > >> > Error Information Log Entries: 0
> > >> > Warning Comp. Temperature Time: 0
> > >> > Critical Comp. Temperature Time: 0
> > >> > Temperature Sensor 1: 33 Celsius
> > >> > Temperature Sensor 2: 42 Celsius
> > >> >
> > >> > Error Information (NVMe Log 0x01, 16 of 64 entries)
> > >> > No Errors Logged
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > lsdrv
> > >> >
> > >> > PCI [nvme] 22:00.0 Non-Volatile memory controller: Samsung Electronics
> > >> > Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
> > >> > └nvme nvme0 SAMSUNG MZQL23T8HCLS-00A07 {S64HNS0TC05245}
> > >> > └nvme0n1 3.49t [259:0] Partitioned (gpt)
> > >> > └nvme0n1p1 3.49t [259:1] Partitioned (gpt)
> > >> > PCI [nvme] 23:00.0 Non-Volatile memory controller: Samsung Electronics
> > >> > Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
> > >> > └nvme nvme1 SAMSUNG MZQL23T8HCLS-00A07 {S64HNS0TC05241}
> > >> > └nvme1n1 3.49t [259:2] Partitioned (gpt)
> > >> > └nvme1n1p1 3.49t [259:3] Partitioned (gpt)
> > >> > PCI [nvme] 24:00.0 Non-Volatile memory controller: Samsung Electronics
> > >> > Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
> > >> > └nvme nvme2 SAMSUNG MZQL23T8HCLS-00A07 {S64HNS0TC05244}
> > >> > └nvme2n1 3.49t [259:4] Partitioned (gpt)
> > >> > └nvme2n1p1 3.49t [259:5] Partitioned (gpt)
> > >> > PCI [ahci] 64:00.0 SATA controller: ASMedia Technology Inc. ASM1062
> > >> > Serial ATA Controller (rev 02)
> > >> > ├scsi 0:0:0:0 ATA SAMSUNG MZ7L31T9 {S6ESNS0W416204}
> > >> > │└sda 1.75t [8:0] Partitioned (gpt)
> > >> > │ ├sda1 512.00m [8:1] vfat {B0FD-2869}
> > >> > │ │└Mounted as /dev/sda1 @ /boot/efi
> > >> > │ └sda2 1.75t [8:2] MD raid1 (0/2) (w/ sdb2) in_sync 'ubuntu-server:0'
> > >> > {2bcfa20a-e221-299c-d3e6-f4cf8124e265}
> > >> > │ └md0 1.75t [9:0] MD v1.2 raid1 (2) active
> > >> > {2bcfa20a:-e221-29:9c-d3e6-:f4cf8124e265}
> > >> > │ │ Partitioned (gpt)
> > >> > │ └md0p1 1.75t [259:6] ext4 {81b5ccee-9c72-4cac-8579-3b9627a8c1b6}
> > >> > │ └Mounted as /dev/md0p1 @ /
> > >> > └scsi 1:0:0:0 ATA SAMSUNG MZ7L31T9 {S6ESNS0W416208}
> > >> > └sdb 1.75t [8:16] Partitioned (gpt)
> > >> > ├sdb1 512.00m [8:17] vfat {B11F-39A7}
> > >> > └sdb2 1.75t [8:18] MD raid1 (1/2) (w/ sda2) in_sync
> > >> > 'ubuntu-server:0' {2bcfa20a-e221-299c-d3e6-f4cf8124e265}
> > >> > └md0 1.75t [9:0] MD v1.2 raid1 (2) active
> > >> > {2bcfa20a:-e221-29:9c-d3e6-:f4cf8124e265}
> > >> > Partitioned (gpt)
> > >> > PCI [ahci] 66:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD]
> > >> > FCH SATA Controller [AHCI mode] (rev 91)
> > >> > └scsi 2:x:x:x [Empty]
> > >> > PCI [ahci] 66:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD]
> > >> > FCH SATA Controller [AHCI mode] (rev 91)
> > >> > └scsi 10:x:x:x [Empty]
> > >> > PCI [ahci] 04:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD]
> > >> > FCH SATA Controller [AHCI mode] (rev 91)
> > >> > └scsi 18:x:x:x [Empty]
> > >> > PCI [ahci] 04:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD]
> > >> > FCH SATA Controller [AHCI mode] (rev 91)
> > >> > └scsi 26:x:x:x [Empty]
> > >> > Other Block Devices
> > >> > ├loop0 0.00k [7:0] Empty/Unknown
> > >> > ├loop1 0.00k [7:1] Empty/Unknown
> > >> > ├loop2 0.00k [7:2] Empty/Unknown
> > >> > ├loop3 0.00k [7:3] Empty/Unknown
> > >> > ├loop4 0.00k [7:4] Empty/Unknown
> > >> > ├loop5 0.00k [7:5] Empty/Unknown
> > >> > ├loop6 0.00k [7:6] Empty/Unknown
> > >> > └loop7 0.00k [7:7] Empty/Unknown
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > wipefs /dev/nvme0n1p1
> > >> >
> > >> > DEVICE OFFSET TYPE UUID LABEL
> > >> > nvme0n1p1 0x200 gpt
> > >> > nvme0n1p1 0x37e38900000 gpt
> > >> > nvme0n1p1 0x1fe PMBR
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > wipefs /dev/nvme1n1p1
> > >> >
> > >> > DEVICE OFFSET TYPE UUID LABEL
> > >> > nvme1n1p1 0x200 gpt
> > >> > nvme1n1p1 0x37e38900000 gpt
> > >> > nvme1n1p1 0x1fe PMBR
> > >> >
> > >> > ____________________________________________
> > >> >
> > >> > wipefs /dev/nvme2n1p1
> > >> >
> > >> > DEVICE OFFSET TYPE UUID LABEL
> > >> > nvme2n1p1 0x200 gpt
> > >> > nvme2n1p1 0x37e38900000 gpt
> > >> > nvme2n1p1 0x1fe PMBR
> > >> >
