On Fri Aug18'23 01:39:08PM, Roger Heflin wrote:
> From: Roger Heflin <rogerheflin@xxxxxxxxx>
> Date: Fri, 18 Aug 2023 13:39:08 -0500
> To: Community support for Fedora users <users@xxxxxxxxxxxxxxxxxxxxxxx>
> Reply-To: Community support for Fedora users <users@xxxxxxxxxxxxxxxxxxxxxxx>
> Subject: Re: slowness with kernel 6.4.10 and software raid
>
> The above makes it very clear what is happening. What kind of disks
> are these? And did you set the scterc timeout? You can see it via
> smartctl -l scterc /dev/sda and then repeat on the other disk.
>
> Setting the timeout as low as you can will improve this situation
> some, but it appears that sda has a number of bad sectors on it.
>
> a full output of "smartclt --xall /dev/sda" would be useful also to
> see how bad it is.
>
> Short answer is you probably need a new device for sda.
>
Thanks!
I tried:
# smartctl -l scterc /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.4.10-200.fc38.x86_64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
SCT Error Recovery Control command not supported
# smartctl --xall /dev/sda
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.4.10-200.fc38.x86_64] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST2000DM001-1ER164
Serial Number: Z4Z5F3LE
LU WWN Device Id: 5 000c50 091167f04
Firmware Version: CC27
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5528
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Aug 18 14:01:28 2023 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM level is: 128 (minimum power consumption without standby)
Rd look-ahead is: Enabled
Write cache is: Enabled
DSN feature is: Unavailable
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Unavailable
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 80) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 212) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x1085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 116 092 006 - 106200704
3 Spin_Up_Time PO---- 096 096 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 97
5 Reallocated_Sector_Ct PO--CK 097 097 010 - 3960
7 Seek_Error_Rate POSR-- 084 060 030 - 333268033
9 Power_On_Hours -O--CK 062 062 000 - 34085
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 96
183 Runtime_Bad_Block -O--CK 100 100 000 - 0
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 001 001 000 - 384
188 Command_Timeout -O--CK 100 098 000 - 3 71 72
189 High_Fly_Writes -O-RCK 065 065 000 - 35
190 Airflow_Temperature_Cel -O---K 063 055 045 - 37 (Min/Max 37/42)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 19
193 Load_Cycle_Count -O--CK 001 001 000 - 294513
194 Temperature_Celsius -O---K 037 045 000 - 37 (0 18 0 0 0)
197 Current_Pending_Sector -O--C- 094 080 000 - 1064
198 Offline_Uncorrectable ----C- 094 080 000 - 1064
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 31366h+32m+19.252s
241 Total_LBAs_Written ------ 100 253 000 - 22394883074
242 Total_LBAs_Read ------ 100 253 000 - 258335971674
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 5 Comprehensive SMART error log
0x03 GPL R/O 5 Ext. Comprehensive SMART error log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x09 SL R/W 1 Selective self-test log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xa1 GPL,SL VS 20 Device vendor specific log
0xa2 GPL VS 4496 Device vendor specific log
0xa8 GPL,SL VS 129 Device vendor specific log
0xa9 GPL,SL VS 1 Device vendor specific log
0xab GPL VS 1 Device vendor specific log
0xb0 GPL VS 5176 Device vendor specific log
0xbe-0xbf GPL VS 65535 Device vendor specific log
0xc0 GPL,SL VS 1 Device vendor specific log
0xc1 GPL,SL VS 10 Device vendor specific log
0xc3 GPL,SL VS 8 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
Device Error Count: 384 (device log contains only the most recent 20 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 384 [3] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b9 20 00 00 Error: UNC at LBA = 0xa312b920 = 2735913248
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b9 20 40 00 16d+06:35:59.162 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b9 18 40 00 16d+06:35:59.154 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b9 10 40 00 16d+06:35:59.154 READ FPDMA QUEUED
61 00 00 00 08 00 00 a3 12 b9 10 40 00 16d+06:35:59.154 WRITE FPDMA QUEUED
ef 00 10 00 02 00 00 00 00 00 00 a0 00 16d+06:35:59.154 SET FEATURES [Enable SATA feature]
Error 383 [2] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b9 10 00 00 Error: UNC at LBA = 0xa312b910 = 2735913232
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b9 10 40 00 16d+06:35:53.336 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b9 08 40 00 16d+06:35:53.335 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b9 00 40 00 16d+06:35:53.335 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 f8 40 00 16d+06:35:53.335 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 f0 40 00 16d+06:35:53.331 READ FPDMA QUEUED
Error 382 [1] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b8 e8 00 00 Error: UNC at LBA = 0xa312b8e8 = 2735913192
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b8 e8 40 00 16d+06:35:49.468 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 e0 40 00 16d+06:35:49.460 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 d8 40 00 16d+06:35:49.460 READ FPDMA QUEUED
61 00 00 00 08 00 00 a3 12 b8 d8 40 00 16d+06:35:49.460 WRITE FPDMA QUEUED
ef 00 10 00 02 00 00 00 00 00 00 a0 00 16d+06:35:49.459 SET FEATURES [Enable SATA feature]
Error 381 [0] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b8 d8 00 00 Error: UNC at LBA = 0xa312b8d8 = 2735913176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b8 d8 40 00 16d+06:35:45.676 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 d0 40 00 16d+06:35:45.673 READ FPDMA QUEUED
ef 00 10 00 02 00 00 00 00 00 00 a0 00 16d+06:35:45.673 SET FEATURES [Enable SATA feature]
27 00 00 00 00 00 00 00 00 00 00 e0 00 16d+06:35:45.673 READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
ec 00 00 00 00 00 00 00 00 00 00 a0 00 16d+06:35:45.672 IDENTIFY DEVICE
Error 380 [19] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b8 c8 00 00 Error: UNC at LBA = 0xa312b8c8 = 2735913160
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b8 c8 40 00 16d+06:35:39.283 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 c0 40 00 16d+06:35:39.282 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 b8 40 00 16d+06:35:39.282 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 b0 40 00 16d+06:35:39.270 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 a8 40 00 16d+06:35:39.270 READ FPDMA QUEUED
Error 379 [18] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b8 a8 00 00 Error: UNC at LBA = 0xa312b8a8 = 2735913128
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b8 a8 40 00 16d+06:35:35.558 READ FPDMA QUEUED
61 00 00 05 78 00 00 65 ac 20 00 40 00 16d+06:35:35.557 WRITE FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 a0 40 00 16d+06:35:35.540 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 98 40 00 16d+06:35:35.532 READ FPDMA QUEUED
ef 00 10 00 02 00 00 00 00 00 00 a0 00 16d+06:35:35.532 SET FEATURES [Enable SATA feature]
Error 378 [17] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b8 90 00 00 Error: UNC at LBA = 0xa312b890 = 2735913104
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b8 90 40 00 16d+06:35:31.406 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 88 40 00 16d+06:35:31.406 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 80 40 00 16d+06:35:31.405 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 78 40 00 16d+06:35:31.398 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 70 40 00 16d+06:35:31.397 READ FPDMA QUEUED
Error 377 [16] occurred at disk power-on lifetime: 34042 hours (1418 days + 10 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 53 00 00 00 00 a3 12 b8 70 00 00 Error: UNC at LBA = 0xa312b870 = 2735913072
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
60 00 00 00 08 00 00 a3 12 b8 70 40 00 16d+06:35:27.414 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 68 40 00 16d+06:35:27.413 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 60 40 00 16d+06:35:27.402 READ FPDMA QUEUED
60 00 00 00 08 00 00 a3 12 b8 58 40 00 16d+06:35:27.401 READ FPDMA QUEUED
61 00 00 00 08 00 00 a3 12 b8 58 40 00 16d+06:35:27.401 WRITE FPDMA QUEUED
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed: read failure 90% 29204 771754056
# 2 Short offline Completed without error 00% 19 -
# 3 Short offline Completed without error 00% 0 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 522 (0x020a)
Device State: Active (0)
Current Temperature: 37 Celsius
Power Cycle Min/Max Temperature: 37/41 Celsius
Lifetime Min/Max Temperature: 18/45 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Data Table command not supported
SCT Error Recovery Control command not supported
Device Statistics (GP/SMART Log 0x04) not supported
Pending Defects log (GP Log 0x0c) not supported
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x000a 2 102 Device-to-host register FISes sent due to a COMRESET
0x0001 2 0 Command failed due to ICRC error
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
Many thanks,
Ranjan
> On Fri, Aug 18, 2023 at 1:30 PM Ranjan Maitra <mlmaitra@xxxxxxx> wrote:
> >
> > Thanks, Roger!
> >
> >
> > On Fri Aug18'23 12:23:23PM, Roger Heflin wrote:
> > > From: Roger Heflin <rogerheflin@xxxxxxxxx>
> > > Date: Fri, 18 Aug 2023 12:23:23 -0500
> > > To: Community support for Fedora users <users@xxxxxxxxxxxxxxxxxxxxxxx>
> > > Reply-To: Community support for Fedora users <users@xxxxxxxxxxxxxxxxxxxxxxx>
> > > Subject: Re: slowness with kernel 6.4.10 and software raid
> > >
> > > Is it moving at all or just stopped? If just stopped it appears that
> > > md126 is using external:/md127 for something and md127 looks wrong
> > > (both disks are spare) but I don't know in this external case what
> > > md127 should look like.
> >
> > It is moving, slowly. It is a 2 TB drive, but this is weird.
> >
> > >
> > > I would suggest checking messages with grep md12[67] /var/log/messages
> > > (and older messages files if the reboot was not this week) to see what
> > > is going on.
> >
> > Good idea! Here is the result from
> >
> > $ grep md126 /var/log/messages
> >
> >
> > Aug 14 15:02:30 localhost mdadm[1035]: Rebuild60 event detected on md device /dev/md126
> > Aug 16 14:21:20 localhost kernel: md/raid1:md126: active with 2 out of 2 mirrors
> > Aug 16 14:21:20 localhost kernel: md126: detected capacity change from 0 to 3711741952
> > Aug 16 14:21:20 localhost kernel: md126: p1
> > Aug 16 14:21:23 localhost systemd[1]: Condition check resulted in dev-md126p1.device - /dev/md126p1 being skipped.
> > Aug 16 14:21:28 localhost systemd-fsck[942]: /dev/md126p1: clean, 7345384/115998720 files, 409971205/463967488 blocks
> > Aug 16 14:21:31 localhost kernel: EXT4-fs (md126p1): mounted filesystem 932eb81c-2ab4-4e6e-b093-46e43dbd6c28 r/w with ordered data mode. Quota mode: none.
> > Aug 16 14:21:31 localhost mdadm[1033]: NewArray event detected on md device /dev/md126
> > Aug 16 14:21:31 localhost mdadm[1033]: RebuildStarted event detected on md device /dev/md126
> > Aug 16 14:21:31 localhost kernel: md: data-check of RAID array md126
> > Aug 16 19:33:18 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735900352
> > Aug 16 19:33:22 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735900864
> > Aug 16 19:33:28 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900496 on sda)
> > Aug 16 19:33:36 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900568 on sda)
> > Aug 16 19:33:41 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900576 on sda)
> > Aug 16 19:33:50 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900624 on sda)
> > Aug 16 19:34:00 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900640 on sda)
> > Aug 16 19:34:10 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900688 on sda)
> > Aug 16 19:34:18 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900712 on sda)
> > Aug 16 19:34:28 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900792 on sda)
> > Aug 16 19:34:32 localhost kernel: md/raid1:md126: redirecting sector 2735900352 to other mirror: sdc
> > Aug 16 19:34:37 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900872 on sda)
> > Aug 16 19:34:45 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900920 on sda)
> > Aug 16 19:34:54 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735900992 on sda)
> > Aug 16 19:34:54 localhost kernel: md/raid1:md126: redirecting sector 2735900864 to other mirror: sdc
> > Aug 16 19:35:07 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735905704
> > Aug 16 19:35:11 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735905960
> > Aug 16 19:35:18 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735905768 on sda)
> > Aug 16 19:35:19 localhost kernel: md/raid1:md126: redirecting sector 2735905704 to other mirror: sdc
> > Aug 16 19:35:24 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735906120 on sda)
> > Aug 16 19:35:33 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735906192 on sda)
> > Aug 16 19:35:39 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735906448 on sda)
> > Aug 16 19:35:40 localhost kernel: md/raid1:md126: redirecting sector 2735905960 to other mirror: sdc
> > Aug 16 19:35:45 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735906472
> > Aug 16 19:35:49 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735906504 on sda)
> > Aug 16 19:35:52 localhost kernel: md/raid1:md126: redirecting sector 2735906472 to other mirror: sdc
> > Aug 16 19:36:03 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735908008
> > Aug 16 19:36:08 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735908232 on sda)
> > Aug 16 19:36:16 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735908344 on sda)
> > Aug 16 19:36:21 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735908424 on sda)
> > Aug 16 19:36:21 localhost kernel: md/raid1:md126: redirecting sector 2735908008 to other mirror: sda
> > Aug 16 19:36:30 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735908008
> > Aug 16 19:36:37 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735908296 on sda)
> > Aug 16 19:36:38 localhost kernel: md/raid1:md126: redirecting sector 2735908008 to other mirror: sdc
> > Aug 16 19:36:42 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735908776
> > Aug 16 19:36:42 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735909032
> > Aug 16 19:36:46 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735908784 on sda)
> > Aug 16 19:36:50 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735908944 on sda)
> > Aug 16 19:36:50 localhost kernel: md/raid1:md126: redirecting sector 2735908776 to other mirror: sdc
> > Aug 16 19:36:55 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735909312 on sda)
> > Aug 16 19:37:00 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735909360 on sda)
> > Aug 16 19:37:04 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735909400 on sda)
> > Aug 16 19:37:11 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735909520 on sda)
> > Aug 16 19:37:11 localhost kernel: md/raid1:md126: redirecting sector 2735909032 to other mirror: sdc
> > Aug 16 19:37:21 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735910056
> > Aug 16 19:37:21 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735910568
> > Aug 16 19:37:25 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735910064 on sda)
> > Aug 16 19:37:31 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735910080 on sda)
> > Aug 16 19:38:00 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735910128 on sda)
> > Aug 16 19:38:08 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735910240 on sda)
> > Aug 16 19:38:12 localhost kernel: md/raid1:md126: redirecting sector 2735910056 to other mirror: sdc
> > Aug 16 19:38:15 localhost kernel: md/raid1:md126: redirecting sector 2735910568 to other mirror: sdc
> > Aug 16 19:38:23 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735911080
> > Aug 16 19:38:23 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735911592
> > Aug 16 19:38:27 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735911520 on sda)
> > Aug 16 19:38:27 localhost kernel: md/raid1:md126: redirecting sector 2735911080 to other mirror: sdc
> > Aug 16 19:38:28 localhost kernel: md/raid1:md126: redirecting sector 2735911592 to other mirror: sdc
> > Aug 16 19:38:33 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735912104
> > Aug 16 19:38:37 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735912184 on sda)
> > Aug 16 19:38:45 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735912240 on sda)
> > Aug 16 19:38:49 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735912248 on sda)
> > Aug 16 19:38:59 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735912288 on sda)
> > Aug 16 19:39:05 localhost kernel: md/raid1:md126: redirecting sector 2735912104 to other mirror: sdc
> > Aug 16 19:39:10 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735912872
> > Aug 16 19:39:14 localhost kernel: md/raid1:md126: sda: rescheduling sector 2735913128
> > Aug 16 19:39:25 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735912976 on sda)
> > Aug 16 19:39:33 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735913048 on sda)
> > Aug 16 19:39:37 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735913072 on sda)
> > Aug 16 19:39:41 localhost kernel: md/raid1:md126: redirecting sector 2735912872 to other mirror: sdc
> > Aug 16 19:39:45 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735913128 on sda)
> > Aug 16 19:39:55 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735913176 on sda)
> > Aug 16 19:40:05 localhost kernel: md/raid1:md126: read error corrected (8 sectors at 2735913232 on sda)
> >
> >
> > And here is what I get from:
> >
> > $ grep md127 /var/log/messages
> >
> >
> > Aug 16 14:16:38 localhost systemd[1]: mdmon@md127.service: Deactivated successfully.
> > Aug 16 14:16:38 localhost systemd[1]: mdmon@md127.service: Unit process 884 (mdmon) remains running after unit stopped.
> > Aug 16 14:16:38 localhost systemd[1]: Stopped mdmon@md127.service - MD Metadata Monitor on /dev/md127.
> > Aug 16 14:16:38 localhost audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=mdmon@md127 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> > Aug 16 14:16:38 localhost systemd[1]: mdmon@md127.service: Consumed 41.719s CPU time.
> > Aug 16 14:21:20 localhost systemd[1]: Starting mdmon@md127.service - MD Metadata Monitor on /dev/md127...
> > Aug 16 14:21:20 localhost systemd[1]: Started mdmon@md127.service - MD Metadata Monitor on /dev/md127.
> > Aug 16 14:21:20 localhost audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=mdmon@md127 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> >
> > >
> > > Maybe also if you have a prior good reboot in messages file include
> > > that and see what happened differently between the 2.
> >
> > Yeah, I do not know where to find this. I looked into /var/log/messages, but it looks like it starts on August 13, which was a surprise to me, and the last non-responsive instance for me was last week (August 10, I think, when I booted into the 6.4 kernel). I did reboot in frustration on August 16.
> >
> > Thanks,
> > Ranjan
> >
> >
> > >
> > > On Fri, Aug 18, 2023 at 7:46 AM Ranjan Maitra <mlmaitra@xxxxxxx> wrote:
> > > >
> > > > On Thu Aug17'23 10:37:29PM, Samuel Sieb wrote:
> > > > > From: Samuel Sieb <samuel@xxxxxxxx>
> > > > > Date: Thu, 17 Aug 2023 22:37:29 -0700
> > > > > To: users@xxxxxxxxxxxxxxxxxxxxxxx
> > > > > Reply-To: Community support for Fedora users <users@xxxxxxxxxxxxxxxxxxxxxxx>
> > > > > Subject: Re: slowness with kernel 6.4.10 and software raid
> > > > >
> > > > > On 8/17/23 21:38, Ranjan Maitra wrote:
> > > > > > $ cat /proc/mdstat
> > > > > > Personalities : [raid1]
> > > > > > md126 : active raid1 sda[1] sdc[0]
> > > > > > 1855870976 blocks super external:/md127/0 [2/2] [UU]
> > > > > > [=>...................] check = 8.8% (165001216/1855870976) finish=45465.2min speed=619K/sec
> > > > > >
> > > > > > md127 : inactive sda[1](S) sdc[0](S)
> > > > > > 10402 blocks super external:imsm
> > > > > >
> > > > > > unused devices: <none>
> > > > > >
> > > > > > I am not sure what it is doing, and I am a bit concerned that this will go on at this rate for about 20 days. No knowing what will happen after that, and also if this problem will recur with another reboot.
> > > > >
> > > > > After a certain amount of time, mdraid will do a verification of the data
> > > > > where it scans the entire array. If you reboot, it will continue from where
> > > > > it left off. But that is *really* slow, so you should find out what's going
> > > > > on there.
> > > >
> > > > Yes, I know, just not sure what to do. Thanks very much!
> > > >
> > > > Any suggestion is appreciated!
> > > >
> > > > Best wishes,
> > > > Ranjan
> > > > _______________________________________________
> > > > users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
> > > > To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
> > > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
> > > > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
> > > _______________________________________________
> > > users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
> > > To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
> > > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
> > _______________________________________________
> > users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
> > To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
> > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
> _______________________________________________
> users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
