Running ffsb (the create workload - the description file is appended)
and then checking with fsck produces errors. This is on an 88TB fs (I
had some hardware (?) problems and had to use a slightly smaller set of
disks than the usual 96TB fs), consisting of 44 2TB LVM volumes (each of those
consisting of 7 300GB disks).
Doing a spot check with debugfs on the first error that e2fsck reports
however, shows that there is no problem there - I think.
I attach the e2fsck report, the spot-check I did with debugfs and the
ffsb workload description file.
This was run on an x86_64 system with 16 CPUs and 128GB of memory,
running a 2.6.32-rc5 kernel and the pu branch of e2fsprogs.
I'm checking things out but if you have any suggestions, please let me
know.
Thanks,
Nick
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 272k/1372220k (67k/206k), time: 16.23/11.18/ 4.44
Pass 1: I/O read: 1MB, write: 0MB, rate: 0.06MB/s
Pass 2: Checking directory structure
First entry '' (inode=739442688) in directory inode 1374757121 (/data) should be '.'
Fix? no
Invalid inode number for '.' in directory inode 1374757121.
Fix? no
Directory entry for '.' in /data (1374757121) is big.
Split? no
Entry '' in /data (1374757121) references inode 739442688 in group 118805 where _INODE_UNINIT is set.
Fix? no
Entry '' in /data (1374757121) has deleted/unused inode 739442688. Clear? no
Missing '.' in directory inode 4217481985.
Fix? no
Invalid inode number for '.' in directory inode 4217481985.
Fix? no
Directory entry for '.' in /meta (4217481985) is big.
Split? no
Pass 2: Memory used: 272k/18014398507852448k (66k/207k), time: 1.85/ 0.90/ 0.91
Pass 2: I/O read: 1MB, write: 0MB, rate: 0.54MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 272k/18014398508375496k (66k/207k), time: 27.06/19.05/ 7.36
'..' in /data (1374757121) is <The NULL inode> (0), should be / (2).
Fix? no
'..' in /meta (4217481985) is <The NULL inode> (0), should be / (2).
Fix? no
Pass 3: Memory used: 272k/18014398507852448k (66k/207k), time: 0.57/ 0.26/ 0.32
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 2 ref count is 5, should be 3. Fix? no
Inode 1374757121 ref count is 2, should be 1. Fix? no
Unattached inode 1374757122
Connect to /lost+found? no
Unattached inode 1374757123
Connect to /lost+found? no
Unattached inode 1374757124
Connect to /lost+found? no
Unattached inode 1374757125
Connect to /lost+found? no
Unattached inode 1374757126
Connect to /lost+found? no
Unattached inode 1374757127
Connect to /lost+found? no
Unattached inode 1374757128
Connect to /lost+found? no
Unattached inode 1374757129
Connect to /lost+found? no
Unattached inode 1374757130
Connect to /lost+found? no
Unattached inode 1374757131
Connect to /lost+found? no
Unattached inode 1374757132
Connect to /lost+found? no
Unattached inode 1374757133
Connect to /lost+found? no
Unattached inode 1374757134
Connect to /lost+found? no
Unattached inode 1374757135
Connect to /lost+found? no
Unattached inode 1374757136
Connect to /lost+found? no
Unattached inode 1374757137
Connect to /lost+found? no
Unattached inode 1374757138
Connect to /lost+found? no
Unattached inode 1374757139
Connect to /lost+found? no
Unattached inode 1374757140
Connect to /lost+found? no
Unattached inode 1374757141
Connect to /lost+found? no
Unattached inode 1374757142
Connect to /lost+found? no
Unattached inode 1374757143
Connect to /lost+found? no
Unattached inode 1374757144
Connect to /lost+found? no
Unattached inode 1374757145
Connect to /lost+found? no
Unattached inode 1374757146
Connect to /lost+found? no
Unattached inode 1374757147
Connect to /lost+found? no
Unattached inode 1374757148
Connect to /lost+found? no
Unattached inode 1374757149
Connect to /lost+found? no
Unattached inode 1374757150
Connect to /lost+found? no
Unattached inode 1374757151
Connect to /lost+found? no
Unattached inode 1374757152
Connect to /lost+found? no
Unattached inode 1374757153
Connect to /lost+found? no
Unattached inode 1374757154
Connect to /lost+found? no
Unattached inode 1374757155
Connect to /lost+found? no
Unattached inode 1374757156
Connect to /lost+found? no
Unattached inode 1374757157
Connect to /lost+found? no
Unattached inode 1374757158
Connect to /lost+found? no
Unattached inode 1374757159
Connect to /lost+found? no
Unattached inode 1374757160
Connect to /lost+found? no
Unattached inode 1374757161
Connect to /lost+found? no
Unattached inode 1374757162
Connect to /lost+found? no
Unattached inode 1374757163
Connect to /lost+found? no
Unattached inode 1374757164
Connect to /lost+found? no
Unattached inode 1374757165
Connect to /lost+found? no
Unattached inode 1374757166
Connect to /lost+found? no
Unattached inode 1374757167
Connect to /lost+found? no
Unattached inode 1374757168
Connect to /lost+found? no
Unattached inode 1374757169
Connect to /lost+found? no
Unattached inode 1374757170
Connect to /lost+found? no
Unattached inode 1374757171
Connect to /lost+found? no
Unattached inode 1374757172
Connect to /lost+found? no
Unattached inode 1374757173
Connect to /lost+found? no
Unattached inode 1374757174
Connect to /lost+found? no
Unattached inode 1374757175
Connect to /lost+found? no
Unattached inode 1374757176
Connect to /lost+found? no
Unattached inode 1374757177
Connect to /lost+found? no
Unattached inode 1374757178
Connect to /lost+found? no
Unattached inode 1374757179
Connect to /lost+found? no
Unattached inode 1374757180
Connect to /lost+found? no
Unattached inode 1374757181
Connect to /lost+found? no
Unattached inode 1374757182
Connect to /lost+found? no
Unattached inode 1374757183
Connect to /lost+found? no
Unattached inode 1374757184
Connect to /lost+found? no
Unattached inode 1374757185
Connect to /lost+found? no
Unattached inode 1374757186
Connect to /lost+found? no
Unattached inode 1374757187
Connect to /lost+found? no
Unattached inode 1374757188
Connect to /lost+found? no
Unattached inode 1374757189
Connect to /lost+found? no
Unattached inode 1374757190
Connect to /lost+found? no
Unattached inode 1374757191
Connect to /lost+found? no
Unattached inode 1374757192
Connect to /lost+found? no
Unattached inode 1374757193
Connect to /lost+found? no
Unattached inode 1374757194
Connect to /lost+found? no
Unattached inode 1374757195
Connect to /lost+found? no
Unattached inode 1374757196
Connect to /lost+found? no
Unattached inode 1374757197
Connect to /lost+found? no
Unattached inode 1374757198
Connect to /lost+found? no
Unattached inode 1374757199
Connect to /lost+found? no
Unattached inode 1374757200
Connect to /lost+found? no
Unattached inode 1374757201
Connect to /lost+found? no
Unattached inode 1374757202
Connect to /lost+found? no
Unattached inode 1374757203
Connect to /lost+found? no
Unattached inode 1374757204
Connect to /lost+found? no
Unattached inode 1374757205
Connect to /lost+found? no
Unattached inode 1374757206
Connect to /lost+found? no
Unattached inode 1374757207
Connect to /lost+found? no
Unattached inode 1374757208
Connect to /lost+found? no
Unattached inode 1374757209
Connect to /lost+found? no
Unattached inode 1374757210
Connect to /lost+found? no
Unattached inode 1374757211
Connect to /lost+found? no
Unattached inode 1374757212
Connect to /lost+found? no
Unattached inode 1374757213
Connect to /lost+found? no
Unattached inode 1374757214
Connect to /lost+found? no
Unattached inode 1374757215
Connect to /lost+found? no
Unattached inode 1374757216
Connect to /lost+found? no
Unattached inode 1374757217
Connect to /lost+found? no
Unattached inode 1374757218
Connect to /lost+found? no
Unattached inode 1374757219
Connect to /lost+found? no
Unattached inode 1374757220
Connect to /lost+found? no
Unattached inode 1374757221
Connect to /lost+found? no
Unattached inode 1374757222
Connect to /lost+found? no
Unattached inode 1374757223
Connect to /lost+found? no
Unattached inode 1374757224
Connect to /lost+found? no
Unattached inode 1374757225
Connect to /lost+found? no
Unattached inode 1374757226
Connect to /lost+found? no
Unattached inode 1374757227
Connect to /lost+found? no
Unattached inode 1374757228
Connect to /lost+found? no
Unattached inode 1374757229
Connect to /lost+found? no
Unattached inode 1374757230
Connect to /lost+found? no
Unattached inode 1374757231
Connect to /lost+found? no
Unattached inode 1374757232
Connect to /lost+found? no
Unattached inode 1374757233
Connect to /lost+found? no
Unattached inode 1374757234
Connect to /lost+found? no
Unattached inode 1374757235
Connect to /lost+found? no
Unattached inode 1374757236
Connect to /lost+found? no
Unattached inode 1374757237
Connect to /lost+found? no
Unattached inode 1374757238
Connect to /lost+found? no
Unattached inode 1374757239
Connect to /lost+found? no
Unattached inode 1374757240
Connect to /lost+found? no
Unattached inode 1374757241
Connect to /lost+found? no
Unattached inode 1374757242
Connect to /lost+found? no
Unattached inode 1374757243
Connect to /lost+found? no
Unattached inode 1374757244
Connect to /lost+found? no
Unattached inode 1374757245
Connect to /lost+found? no
Unattached inode 1374757246
Connect to /lost+found? no
Unattached inode 1374757247
Connect to /lost+found? no
Unattached inode 1374757248
Connect to /lost+found? no
Unattached inode 1374757249
Connect to /lost+found? no
Unattached inode 1374757250
Connect to /lost+found? no
Unattached inode 1374757251
Connect to /lost+found? no
Inode 4217481985 ref count is 2, should be 1. Fix? no
Pass 4: Memory used: 272k/18014398509138608k (66k/207k), time: 145.87/145.61/ 0.24
Pass 4: I/O read: 1MB, write: 0MB, rate: 0.01MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 272k/18014398508615560k (66k/207k), time: 238.95/54.46/ 2.69
Pass 5: I/O read: 173MB, write: 0MB, rate: 0.72MB/s
/dev/mapper/bigvg-bigvol: ********** WARNING: Filesystem still has errors **********
143 inodes used (0.00%)
0 non-contiguous files (0.0%)
0 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 3/130
303588979 blocks used (1.35%)
0 bad blocks
0 large files
130 regular files
4 directories
0 character device files
0 block device files
0 fifos
4294967292 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
1 file
Memory used: 272k/18014398508615560k (66k/207k), time: 411.93/219.12/10.34
I/O read: 215MB, write: 0MB, rate: 0.52MB/s
Pass 1: Checking inodes, blocks, and sizes
Pass 1: Memory used: 272k/1372220k (67k/206k), time: 16.23/11.18/ 4.44
Pass 1: I/O read: 1MB, write: 0MB, rate: 0.06MB/s
Pass 2: Checking directory structure
First entry '' (inode=739442688) in directory inode 1374757121 (/data) should be '.'
Fix? no
Invalid inode number for '.' in directory inode 1374757121.
Fix? no
Directory entry for '.' in /data (1374757121) is big.
Split? no
Entry '' in /data (1374757121) references inode 739442688 in group 118805 where _INODE_UNINIT is set.
Fix? no
Entry '' in /data (1374757121) has deleted/unused inode 739442688. Clear? no
Missing '.' in directory inode 4217481985.
Fix? no
Invalid inode number for '.' in directory inode 4217481985.
Fix? no
Directory entry for '.' in /meta (4217481985) is big.
Split? no
Pass 2: Memory used: 272k/18014398507852448k (66k/207k), time: 1.85/ 0.90/ 0.91
Pass 2: I/O read: 1MB, write: 0MB, rate: 0.54MB/s
Pass 3: Checking directory connectivity
Peak memory: Memory used: 272k/18014398508375496k (66k/207k), time: 27.06/19.05/ 7.36
'..' in /data (1374757121) is <The NULL inode> (0), should be / (2).
Fix? no
'..' in /meta (4217481985) is <The NULL inode> (0), should be / (2).
Fix? no
Pass 3: Memory used: 272k/18014398507852448k (66k/207k), time: 0.57/ 0.26/ 0.32
Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s
Pass 4: Checking reference counts
Inode 2 ref count is 5, should be 3. Fix? no
Inode 1374757121 ref count is 2, should be 1. Fix? no
Unattached inode 1374757122
Connect to /lost+found? no
Unattached inode 1374757123
Connect to /lost+found? no
Unattached inode 1374757124
Connect to /lost+found? no
Unattached inode 1374757125
Connect to /lost+found? no
Unattached inode 1374757126
Connect to /lost+found? no
Unattached inode 1374757127
Connect to /lost+found? no
Unattached inode 1374757128
Connect to /lost+found? no
Unattached inode 1374757129
Connect to /lost+found? no
Unattached inode 1374757130
Connect to /lost+found? no
Unattached inode 1374757131
Connect to /lost+found? no
Unattached inode 1374757132
Connect to /lost+found? no
Unattached inode 1374757133
Connect to /lost+found? no
Unattached inode 1374757134
Connect to /lost+found? no
Unattached inode 1374757135
Connect to /lost+found? no
Unattached inode 1374757136
Connect to /lost+found? no
Unattached inode 1374757137
Connect to /lost+found? no
Unattached inode 1374757138
Connect to /lost+found? no
Unattached inode 1374757139
Connect to /lost+found? no
Unattached inode 1374757140
Connect to /lost+found? no
Unattached inode 1374757141
Connect to /lost+found? no
Unattached inode 1374757142
Connect to /lost+found? no
Unattached inode 1374757143
Connect to /lost+found? no
Unattached inode 1374757144
Connect to /lost+found? no
Unattached inode 1374757145
Connect to /lost+found? no
Unattached inode 1374757146
Connect to /lost+found? no
Unattached inode 1374757147
Connect to /lost+found? no
Unattached inode 1374757148
Connect to /lost+found? no
Unattached inode 1374757149
Connect to /lost+found? no
Unattached inode 1374757150
Connect to /lost+found? no
Unattached inode 1374757151
Connect to /lost+found? no
Unattached inode 1374757152
Connect to /lost+found? no
Unattached inode 1374757153
Connect to /lost+found? no
Unattached inode 1374757154
Connect to /lost+found? no
Unattached inode 1374757155
Connect to /lost+found? no
Unattached inode 1374757156
Connect to /lost+found? no
Unattached inode 1374757157
Connect to /lost+found? no
Unattached inode 1374757158
Connect to /lost+found? no
Unattached inode 1374757159
Connect to /lost+found? no
Unattached inode 1374757160
Connect to /lost+found? no
Unattached inode 1374757161
Connect to /lost+found? no
Unattached inode 1374757162
Connect to /lost+found? no
Unattached inode 1374757163
Connect to /lost+found? no
Unattached inode 1374757164
Connect to /lost+found? no
Unattached inode 1374757165
Connect to /lost+found? no
Unattached inode 1374757166
Connect to /lost+found? no
Unattached inode 1374757167
Connect to /lost+found? no
Unattached inode 1374757168
Connect to /lost+found? no
Unattached inode 1374757169
Connect to /lost+found? no
Unattached inode 1374757170
Connect to /lost+found? no
Unattached inode 1374757171
Connect to /lost+found? no
Unattached inode 1374757172
Connect to /lost+found? no
Unattached inode 1374757173
Connect to /lost+found? no
Unattached inode 1374757174
Connect to /lost+found? no
Unattached inode 1374757175
Connect to /lost+found? no
Unattached inode 1374757176
Connect to /lost+found? no
Unattached inode 1374757177
Connect to /lost+found? no
Unattached inode 1374757178
Connect to /lost+found? no
Unattached inode 1374757179
Connect to /lost+found? no
Unattached inode 1374757180
Connect to /lost+found? no
Unattached inode 1374757181
Connect to /lost+found? no
Unattached inode 1374757182
Connect to /lost+found? no
Unattached inode 1374757183
Connect to /lost+found? no
Unattached inode 1374757184
Connect to /lost+found? no
Unattached inode 1374757185
Connect to /lost+found? no
Unattached inode 1374757186
Connect to /lost+found? no
Unattached inode 1374757187
Connect to /lost+found? no
Unattached inode 1374757188
Connect to /lost+found? no
Unattached inode 1374757189
Connect to /lost+found? no
Unattached inode 1374757190
Connect to /lost+found? no
Unattached inode 1374757191
Connect to /lost+found? no
Unattached inode 1374757192
Connect to /lost+found? no
Unattached inode 1374757193
Connect to /lost+found? no
Unattached inode 1374757194
Connect to /lost+found? no
Unattached inode 1374757195
Connect to /lost+found? no
Unattached inode 1374757196
Connect to /lost+found? no
Unattached inode 1374757197
Connect to /lost+found? no
Unattached inode 1374757198
Connect to /lost+found? no
Unattached inode 1374757199
Connect to /lost+found? no
Unattached inode 1374757200
Connect to /lost+found? no
Unattached inode 1374757201
Connect to /lost+found? no
Unattached inode 1374757202
Connect to /lost+found? no
Unattached inode 1374757203
Connect to /lost+found? no
Unattached inode 1374757204
Connect to /lost+found? no
Unattached inode 1374757205
Connect to /lost+found? no
Unattached inode 1374757206
Connect to /lost+found? no
Unattached inode 1374757207
Connect to /lost+found? no
Unattached inode 1374757208
Connect to /lost+found? no
Unattached inode 1374757209
Connect to /lost+found? no
Unattached inode 1374757210
Connect to /lost+found? no
Unattached inode 1374757211
Connect to /lost+found? no
Unattached inode 1374757212
Connect to /lost+found? no
Unattached inode 1374757213
Connect to /lost+found? no
Unattached inode 1374757214
Connect to /lost+found? no
Unattached inode 1374757215
Connect to /lost+found? no
Unattached inode 1374757216
Connect to /lost+found? no
Unattached inode 1374757217
Connect to /lost+found? no
Unattached inode 1374757218
Connect to /lost+found? no
Unattached inode 1374757219
Connect to /lost+found? no
Unattached inode 1374757220
Connect to /lost+found? no
Unattached inode 1374757221
Connect to /lost+found? no
Unattached inode 1374757222
Connect to /lost+found? no
Unattached inode 1374757223
Connect to /lost+found? no
Unattached inode 1374757224
Connect to /lost+found? no
Unattached inode 1374757225
Connect to /lost+found? no
Unattached inode 1374757226
Connect to /lost+found? no
Unattached inode 1374757227
Connect to /lost+found? no
Unattached inode 1374757228
Connect to /lost+found? no
Unattached inode 1374757229
Connect to /lost+found? no
Unattached inode 1374757230
Connect to /lost+found? no
Unattached inode 1374757231
Connect to /lost+found? no
Unattached inode 1374757232
Connect to /lost+found? no
Unattached inode 1374757233
Connect to /lost+found? no
Unattached inode 1374757234
Connect to /lost+found? no
Unattached inode 1374757235
Connect to /lost+found? no
Unattached inode 1374757236
Connect to /lost+found? no
Unattached inode 1374757237
Connect to /lost+found? no
Unattached inode 1374757238
Connect to /lost+found? no
Unattached inode 1374757239
Connect to /lost+found? no
Unattached inode 1374757240
Connect to /lost+found? no
Unattached inode 1374757241
Connect to /lost+found? no
Unattached inode 1374757242
Connect to /lost+found? no
Unattached inode 1374757243
Connect to /lost+found? no
Unattached inode 1374757244
Connect to /lost+found? no
Unattached inode 1374757245
Connect to /lost+found? no
Unattached inode 1374757246
Connect to /lost+found? no
Unattached inode 1374757247
Connect to /lost+found? no
Unattached inode 1374757248
Connect to /lost+found? no
Unattached inode 1374757249
Connect to /lost+found? no
Unattached inode 1374757250
Connect to /lost+found? no
Unattached inode 1374757251
Connect to /lost+found? no
Inode 4217481985 ref count is 2, should be 1. Fix? no
Pass 4: Memory used: 272k/18014398509138608k (66k/207k), time: 145.87/145.61/ 0.24
Pass 4: I/O read: 1MB, write: 0MB, rate: 0.01MB/s
Pass 5: Checking group summary information
Pass 5: Memory used: 272k/18014398508615560k (66k/207k), time: 238.95/54.46/ 2.69
Pass 5: I/O read: 173MB, write: 0MB, rate: 0.72MB/s
/dev/mapper/bigvg-bigvol: ********** WARNING: Filesystem still has errors **********
143 inodes used (0.00%)
0 non-contiguous files (0.0%)
0 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 3/130
303588979 blocks used (1.35%)
0 bad blocks
0 large files
130 regular files
4 directories
0 character device files
0 block device files
0 fifos
4294967292 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
1 file
Memory used: 272k/18014398508615560k (66k/207k), time: 411.93/219.12/10.34
I/O read: 215MB, write: 0MB, rate: 0.52MB/s
# debugfs -c /dev/mapper/bigvg-bigvol
debugfs 1.41.9 (22-Aug-2009)
/dev/mapper/bigvg-bigvol: catastrophic mode - not reading inode or group bitmaps
debugfs: stat <1374757121>
Inode: 1374757121 Type: directory Mode: 0700 Flags: 0x80000
Generation: 3074310533 Version: 0x00000000:00000083
User: 0 Group: 0 Size: 4096
File ACL: 0 Directory ACL: 0
Links: 2 Blockcount: 8
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x4adfdda6:9fdcba8c -- Thu Oct 22 00:20:54 2009
atime: 0x4adfdc7a:66134bb4 -- Thu Oct 22 00:15:54 2009
mtime: 0x4adfdda6:9fdcba8c -- Thu Oct 22 00:20:54 2009
crtime: 0x4adfdc7a:66134bb4 -- Thu Oct 22 00:15:54 2009
Size of extra inode fields: 28
EXTENTS:
(0): 7237802096
debugfs: ls <1374757121>
1374757121 (12) . 2 (12) .. 1374757122 (20) datafile0
1374757123 (20) datafile1 1374757124 (20) datafile2
1374757125 (20) datafile3 1374757126 (20) datafile4
1374757127 (20) datafile5 1374757128 (20) datafile6
...
# Large file creates
# Creating 1 GB files.
time=300
alignio=1
[filesystem0]
location=/ffsb/disk1
# All created files will be 1 GB.
min_filesize=1073741824
max_filesize=1073741824
[end0]
[threadgroup0]
num_threads=1
create_weight=1
write_blocksize=4096 # 4 kB
[stats]
enable_stats=1
enable_range=1
msec_range 0.00 0.01
msec_range 0.01 0.02
msec_range 0.02 0.05
msec_range 0.05 0.10
msec_range 0.10 0.20
msec_range 0.20 0.50
msec_range 0.50 1.00
msec_range 1.00 2.00
msec_range 2.00 5.00
msec_range 5.00 10.00
msec_range 10.00 20.00
msec_range 20.00 50.00
msec_range 50.00 100.00
msec_range 100.00 200.00
msec_range 200.00 500.00
msec_range 500.00 1000.00
msec_range 1000.00 2000.00
msec_range 2000.00 5000.00
msec_range 5000.00 10000.00
[end]
[end0]
