Hi amir and all,
On 2/6/23 3:06 AM, Amir Goldstein wrote:
>>>>> Apart from that, I still fail to get some thoughts (apart from
>>>>> unprivileged
>>>>> mounts) how EROFS + overlayfs combination fails on automative real
>>>>> workloads
>>>>> aside from "ls -lR" (readdir + stat).
>>>>>
>>>>> And eventually we still need overlayfs for most use cases to do
>>>>> writable
>>>>> stuffs, anyway, it needs some words to describe why such < 1s
>>>>> difference is
>>>>> very very important to the real workload as you already mentioned
>>>>> before.
>>>>>
>>>>> And with overlayfs lazy lookup, I think it can be close to ~100ms or
>>>>> better.
>>>>>
>>>>
>>>> If we had an overlay.fs-verity xattr, then I think there are no
>>>> individual features lacking for it to work for the automotive usecase
>>>> I'm working on. Nor for the OCI container usecase. However, the
>>>> possibility of doing something doesn't mean it is the better technical
>>>> solution.
>>>>
>>>> The container usecase is very important in real world Linux use today,
>>>> and as such it makes sense to have a technically excellent solution for
>>>> it, not just a workable solution. Obviously we all have different
>>>> viewpoints of what that is, but these are the reasons why I think a
>>>> composefs solution is better:
>>>>
>>>> * It is faster than all other approaches for the one thing it actually
>>>> needs to do (lookup and readdir performance). Other kinds of
>>>> performance (file i/o speed, etc) is up to the backing filesystem
>>>> anyway.
>>>>
>>>> Even if there are possible approaches to make overlayfs perform better
>>>> here (the "lazy lookup" idea) it will not reach the performance of
>>>> composefs, while further complicating the overlayfs codebase. (btw, did
>>>> someone ask Miklos what he thinks of that idea?)
>>>>
>>>
>>> Well, Miklos was CCed (now in TO:)
>>> I did ask him specifically about relaxing -ouserxarr,metacopy,redirect:
>>> https://lore.kernel.org/linux-unionfs/20230126082228.rweg75ztaexykejv@wittgenstein/T/#mc375df4c74c0d41aa1a2251c97509c6522487f96
>>> but no response on that yet.
>>>
>>> TBH, in the end, Miklos really is the one who is going to have the most
>>> weight on the outcome.
>>>
>>> If Miklos is interested in adding this functionality to overlayfs, you are going
>>> to have a VERY hard sell, trying to merge composefs as an independent
>>> expert filesystem. The community simply does not approve of this sort of
>>> fragmentation unless there is a very good reason to do that.
>>>
>>>> For the automotive usecase we have strict cold-boot time requirements
>>>> that make cold-cache performance very important to us. Of course, there
>>>> is no simple time requirements for the specific case of listing files
>>>> in an image, but any improvement in cold-cache performance for both the
>>>> ostree rootfs and the containers started during boot will be worth its
>>>> weight in gold trying to reach these hard KPIs.
>>>>
>>>> * It uses less memory, as we don't need the extra inodes that comes
>>>> with the overlayfs mount. (See profiling data in giuseppes mail[1]).
>>>
>>> Understood, but we will need profiling data with the optimized ovl
>>> (or with the single blob hack) to compare the relevant alternatives.
>>
>> My little request again, could you help benchmark on your real workload
>> rather than "ls -lR" stuff? If your hard KPI is really what as you
>> said, why not just benchmark the real workload now and write a detailed
>> analysis to everyone to explain it's a _must_ that we should upstream
>> a new stacked fs for this?
>>
>
> I agree that benchmarking the actual KPI (boot time) will have
> a much stronger impact and help to build a much stronger case
> for composefs if you can prove that the boot time difference really matters.
>
> In order to test boot time on fair grounds, I prepared for you a POC
> branch with overlayfs lazy lookup:
> https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata
>
> It is very lightly tested, but should be sufficient for the benchmark.
> Note that:
> 1. You need to opt-in with redirect_dir=lazyfollow,metacopy=on
> 2. The lazyfollow POC only works with read-only overlay that
> has two lower dirs (1 metadata layer and one data blobs layer)
> 3. The data layer must be a local blockdev fs (i.e. not a network fs)
> 4. Only absolute path redirects are lazy (e.g. "/objects/cc/3da...")
>
> These limitations could be easily lifted with a bit more work.
> If any of those limitations stand in your way for running the benchmark
> let me know and I'll see what I can do.
>
Thanks for the lazyfollow POC, I updated the perf test with overlayfs
lazyfollow enabled.
| uncached(ms)| cached(ms)
----------------------------------------------------+-----+----
composefs | 404 | 181
composefs (readahead of manifest disabled) | 523 | 178
erofs (loop BUFFER) | 300 | 188
erofs (loop DIRECT) | 486 | 188
erofs (loop DIRECT + ra manifest) | 292 | 190
erofs (loop BUFFER) +overlayfs(lazyfollowup) | 502 | 286
erofs (loop DIRECT) +overlayfs(lazyfollowup) | 686 | 285
erofs (loop DIRECT+ra manifest)+overlayfs(lazyfollowup) | 484 | 300
I find that composefs behaves better than purely erofs (loop DIRECT),
e.g. 404ms vs 486ms in uncached situation, somewhat because composefs
reads part of metadata by buffered kernel_read() and thus the builtin
readahead is performed on the manifest file. With the readahead for the
manifest disabled, the performance gets much worse.
Erofs can also use similar optimization of readahead the manifest file
when accessing the metadata if really needed. An example POC
implementation is inlined in the bottom of this mail. Considering the
workload of "ls -lR" will read basically the full content of the
manifest file, plusing the manifest file size is just ~10MB, the POC
implementation just performs async readahead upon the manifest with a
fixed step of 128KB. With this opt-in, erofs performs somewhat better
in uncached situation. I have to admit that this much depends on the
range and the step size of the readahead, but at least it indicates that
erofs can get comparable performance with similar optimization.
Besides, as mentioned in [1], in composefs the on-disk inode under one
directory is arranged closer than erofs, which means the submitted IO
when doing "ls -l" in erofs is more random than that in composefs,
somewhat affecting the performance. It can be possibly fixed by
improving mkfs.erofs if the gap (~80ms) really matters.
The inode id arrangement under the root directory of tested rootfs is
shown as an attachment, and the tested erofs image of erofs+overlayfs is
constructed from the script (mkhack.sh) attached in [2] offered by
Alexander.
To summarize:
For composefs and erofs, they are quite similar and the performance is
also comparable (with the same optimization).
But when comparing composefs and erofs+overlayfs(lazyfollowup), at least
in the workload of "ls -lR", the combination of erofs and overlayfs
costs ~100ms more in both cached and uncached situation. If such ~100ms
diff really matters, erofs could resolve "redirect" xattr itself, in
which case overlayfs is not involved and then the performance shall be
comparable with composefs, but i'm not sure if it's worthwhile
considering the results are already close. Besides the rootfs is
read-only in this case, and if we need writable layer anyway, overlayfs
still needs to be introduced.
[1]
https://lore.kernel.org/lkml/1d65be2f-6d3a-13c6-4982-66bbb0f9b530@xxxxxxxxxxxxxxxxx/
[2]
https://lore.kernel.org/linux-fsdevel/5fb32a1297821040edd8c19ce796fc0540101653.camel@xxxxxxxxxx/
diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c
index d3b8736fa124..e74e24e00b49 100644
--- a/fs/erofs/inode.c
+++ b/fs/erofs/inode.c
@@ -21,6 +21,10 @@ static void *erofs_read_inode(struct erofs_buf *buf,
struct erofs_inode_compact *dic;
struct erofs_inode_extended *die, *copied = NULL;
unsigned int ifmt;
+ struct folio *folio;
+ struct inode *binode = sb->s_bdev->bd_inode;
+ struct address_space *mapping = binode->i_mapping;
+ pgoff_t index = inode_loc >> PAGE_SHIFT;
int err;
blkaddr = erofs_blknr(inode_loc);
@@ -29,6 +33,16 @@ static void *erofs_read_inode(struct erofs_buf *buf,
erofs_dbg("%s, reading inode nid %llu at %u of blkaddr %u",
__func__, vi->nid, *ofs, blkaddr);
+ folio = filemap_get_folio(mapping, index);
+ if (!folio || !folio_test_uptodate(folio)) {
+ loff_t isize = i_size_read(binode);
+ pgoff_t end_index = (isize - 1) >> PAGE_SHIFT;
+ unsigned long nr_to_read = min_t(unsigned long,
end_index - index, 32);
+ DEFINE_READAHEAD(ractl, NULL, NULL, mapping, index);
+
+ page_cache_ra_unbounded(&ractl, nr_to_read, 0);
+ }
+
kaddr = erofs_read_metabuf(buf, sb, blkaddr, EROFS_KMAP);
if (IS_ERR(kaddr)) {
erofs_err(sb, "failed to get inode (nid: %llu) page, err
%ld",
--
Thanks,
Jingbocomposefs:
=========
# ls -il /mnt/cps
/mnt/cps:
1 dr-xr-xr-x 2 root root 6 8月 10 2021 afs
2 lrwxrwxrwx 1 root root 7 8月 10 2021 bin -> usr/bin
3 dr-xr-xr-x 5 root root 270 1月 17 18:15 boot
4 drwxr-xr-x 2 root root 6 1月 17 18:14 dev
5 drwxr-xr-x 87 root root 4096 1月 17 18:15 etc
6 drwxr-xr-x 3 root root 19 1月 17 18:15 home
7 lrwxrwxrwx 1 root root 7 8月 10 2021 lib -> usr/lib
8 lrwxrwxrwx 1 root root 9 8月 10 2021 lib64 -> usr/lib64
9 drwxr-xr-x 2 root root 6 8月 10 2021 media
10 drwxr-xr-x 2 root root 6 8月 10 2021 mnt
11 drwxr-xr-x 3 root root 16 1月 17 18:14 opt
12 drwxr-xr-x 2 root root 6 1月 17 18:14 proc
13 dr-xr-x--- 3 root root 103 1月 17 18:15 root
14 drwxr-xr-x 14 root root 187 1月 17 18:15 run
15 lrwxrwxrwx 1 root root 8 8月 10 2021 sbin -> usr/sbin
16 drwxr-xr-x 2 root root 6 8月 10 2021 srv
17 drwxr-xr-x 2 root root 6 1月 17 18:14 sys
18 drwxrwxrwt 3 root root 29 1月 17 18:15 tmp
19 drwxr-xr-x 12 root root 144 1月 17 18:14 usr
20 drwxr-xr-x 18 root root 235 1月 17 18:14 var
erofs:
====
# ls -il /mnt/erofs-raw/bootstrap
/mnt/erofs-raw/bootstrap:
1149 crw-r--r-- 1 root root 0, 0 2月 2 11:00 00
1016 crw-r--r-- 1 root root 0, 0 2月 2 11:00 01
1018 crw-r--r-- 1 root root 0, 0 2月 2 11:00 02
1020 crw-r--r-- 1 root root 0, 0 2月 2 11:00 03
1022 crw-r--r-- 1 root root 0, 0 2月 2 11:00 04
1152 crw-r--r-- 1 root root 0, 0 2月 2 11:00 05
1154 crw-r--r-- 1 root root 0, 0 2月 2 11:00 06
1156 crw-r--r-- 1 root root 0, 0 2月 2 11:00 07
1158 crw-r--r-- 1 root root 0, 0 2月 2 11:00 08
1160 crw-r--r-- 1 root root 0, 0 2月 2 11:00 09
1162 crw-r--r-- 1 root root 0, 0 2月 2 11:00 0a
1164 crw-r--r-- 1 root root 0, 0 2月 2 11:00 0b
1166 crw-r--r-- 1 root root 0, 0 2月 2 11:00 0c
1168 crw-r--r-- 1 root root 0, 0 2月 2 11:00 0d
1170 crw-r--r-- 1 root root 0, 0 2月 2 11:00 0e
1172 crw-r--r-- 1 root root 0, 0 2月 2 11:00 0f
1174 crw-r--r-- 1 root root 0, 0 2月 2 11:00 10
1176 crw-r--r-- 1 root root 0, 0 2月 2 11:00 11
1178 crw-r--r-- 1 root root 0, 0 2月 2 11:00 12
1180 crw-r--r-- 1 root root 0, 0 2月 2 11:00 13
1182 crw-r--r-- 1 root root 0, 0 2月 2 11:00 14
1184 crw-r--r-- 1 root root 0, 0 2月 2 11:00 15
1186 crw-r--r-- 1 root root 0, 0 2月 2 11:00 16
1188 crw-r--r-- 1 root root 0, 0 2月 2 11:00 17
1190 crw-r--r-- 1 root root 0, 0 2月 2 11:00 18
1192 crw-r--r-- 1 root root 0, 0 2月 2 11:00 19
1194 crw-r--r-- 1 root root 0, 0 2月 2 11:00 1a
1196 crw-r--r-- 1 root root 0, 0 2月 2 11:00 1b
1198 crw-r--r-- 1 root root 0, 0 2月 2 11:00 1c
1200 crw-r--r-- 1 root root 0, 0 2月 2 11:00 1d
1202 crw-r--r-- 1 root root 0, 0 2月 2 11:00 1e
1204 crw-r--r-- 1 root root 0, 0 2月 2 11:00 1f
1206 crw-r--r-- 1 root root 0, 0 2月 2 11:00 20
1208 crw-r--r-- 1 root root 0, 0 2月 2 11:00 21
1210 crw-r--r-- 1 root root 0, 0 2月 2 11:00 22
1212 crw-r--r-- 1 root root 0, 0 2月 2 11:00 23
1214 crw-r--r-- 1 root root 0, 0 2月 2 11:00 24
1216 crw-r--r-- 1 root root 0, 0 2月 2 11:00 25
1218 crw-r--r-- 1 root root 0, 0 2月 2 11:00 26
1220 crw-r--r-- 1 root root 0, 0 2月 2 11:00 27
1222 crw-r--r-- 1 root root 0, 0 2月 2 11:00 28
1224 crw-r--r-- 1 root root 0, 0 2月 2 11:00 29
1226 crw-r--r-- 1 root root 0, 0 2月 2 11:00 2a
1228 crw-r--r-- 1 root root 0, 0 2月 2 11:00 2b
1230 crw-r--r-- 1 root root 0, 0 2月 2 11:00 2c
1232 crw-r--r-- 1 root root 0, 0 2月 2 11:00 2d
1234 crw-r--r-- 1 root root 0, 0 2月 2 11:00 2e
1236 crw-r--r-- 1 root root 0, 0 2月 2 11:00 2f
1238 crw-r--r-- 1 root root 0, 0 2月 2 11:00 30
1240 crw-r--r-- 1 root root 0, 0 2月 2 11:00 31
1242 crw-r--r-- 1 root root 0, 0 2月 2 11:00 32
1244 crw-r--r-- 1 root root 0, 0 2月 2 11:00 33
1246 crw-r--r-- 1 root root 0, 0 2月 2 11:00 34
1248 crw-r--r-- 1 root root 0, 0 2月 2 11:00 35
1250 crw-r--r-- 1 root root 0, 0 2月 2 11:00 36
1252 crw-r--r-- 1 root root 0, 0 2月 2 11:00 37
1254 crw-r--r-- 1 root root 0, 0 2月 2 11:00 38
1256 crw-r--r-- 1 root root 0, 0 2月 2 11:00 39
1258 crw-r--r-- 1 root root 0, 0 2月 2 11:00 3a
1260 crw-r--r-- 1 root root 0, 0 2月 2 11:00 3b
1262 crw-r--r-- 1 root root 0, 0 2月 2 11:00 3c
1264 crw-r--r-- 1 root root 0, 0 2月 2 11:00 3d
1266 crw-r--r-- 1 root root 0, 0 2月 2 11:00 3e
1268 crw-r--r-- 1 root root 0, 0 2月 2 11:00 3f
1270 crw-r--r-- 1 root root 0, 0 2月 2 11:00 40
1272 crw-r--r-- 1 root root 0, 0 2月 2 11:00 41
1274 crw-r--r-- 1 root root 0, 0 2月 2 11:00 42
1276 crw-r--r-- 1 root root 0, 0 2月 2 11:00 43
1280 crw-r--r-- 1 root root 0, 0 2月 2 11:00 44
1278 crw-r--r-- 1 root root 0, 0 2月 2 11:00 45
1282 crw-r--r-- 1 root root 0, 0 2月 2 11:00 46
1284 crw-r--r-- 1 root root 0, 0 2月 2 11:00 47
1286 crw-r--r-- 1 root root 0, 0 2月 2 11:00 48
1288 crw-r--r-- 1 root root 0, 0 2月 2 11:00 49
1290 crw-r--r-- 1 root root 0, 0 2月 2 11:00 4a
1292 crw-r--r-- 1 root root 0, 0 2月 2 11:00 4b
1294 crw-r--r-- 1 root root 0, 0 2月 2 11:00 4c
1296 crw-r--r-- 1 root root 0, 0 2月 2 11:00 4d
1298 crw-r--r-- 1 root root 0, 0 2月 2 11:00 4e
1300 crw-r--r-- 1 root root 0, 0 2月 2 11:00 4f
1302 crw-r--r-- 1 root root 0, 0 2月 2 11:00 50
1304 crw-r--r-- 1 root root 0, 0 2月 2 11:00 51
1306 crw-r--r-- 1 root root 0, 0 2月 2 11:00 52
1308 crw-r--r-- 1 root root 0, 0 2月 2 11:00 53
1310 crw-r--r-- 1 root root 0, 0 2月 2 11:00 54
1312 crw-r--r-- 1 root root 0, 0 2月 2 11:00 55
1314 crw-r--r-- 1 root root 0, 0 2月 2 11:00 56
1316 crw-r--r-- 1 root root 0, 0 2月 2 11:00 57
1318 crw-r--r-- 1 root root 0, 0 2月 2 11:00 58
1320 crw-r--r-- 1 root root 0, 0 2月 2 11:00 59
1322 crw-r--r-- 1 root root 0, 0 2月 2 11:00 5a
1324 crw-r--r-- 1 root root 0, 0 2月 2 11:00 5b
1326 crw-r--r-- 1 root root 0, 0 2月 2 11:00 5c
1328 crw-r--r-- 1 root root 0, 0 2月 2 11:00 5d
1330 crw-r--r-- 1 root root 0, 0 2月 2 11:00 5e
1332 crw-r--r-- 1 root root 0, 0 2月 2 11:00 5f
1334 crw-r--r-- 1 root root 0, 0 2月 2 11:00 60
1336 crw-r--r-- 1 root root 0, 0 2月 2 11:00 61
1338 crw-r--r-- 1 root root 0, 0 2月 2 11:00 62
1340 crw-r--r-- 1 root root 0, 0 2月 2 11:00 63
1342 crw-r--r-- 1 root root 0, 0 2月 2 11:00 64
1344 crw-r--r-- 1 root root 0, 0 2月 2 11:00 65
1346 crw-r--r-- 1 root root 0, 0 2月 2 11:00 66
1348 crw-r--r-- 1 root root 0, 0 2月 2 11:00 67
1350 crw-r--r-- 1 root root 0, 0 2月 2 11:00 68
1352 crw-r--r-- 1 root root 0, 0 2月 2 11:00 69
1354 crw-r--r-- 1 root root 0, 0 2月 2 11:00 6a
1356 crw-r--r-- 1 root root 0, 0 2月 2 11:00 6b
1358 crw-r--r-- 1 root root 0, 0 2月 2 11:00 6c
1360 crw-r--r-- 1 root root 0, 0 2月 2 11:00 6d
1362 crw-r--r-- 1 root root 0, 0 2月 2 11:00 6e
1364 crw-r--r-- 1 root root 0, 0 2月 2 11:00 6f
1366 crw-r--r-- 1 root root 0, 0 2月 2 11:00 70
1368 crw-r--r-- 1 root root 0, 0 2月 2 11:00 71
1370 crw-r--r-- 1 root root 0, 0 2月 2 11:00 72
1372 crw-r--r-- 1 root root 0, 0 2月 2 11:00 73
1374 crw-r--r-- 1 root root 0, 0 2月 2 11:00 74
1376 crw-r--r-- 1 root root 0, 0 2月 2 11:00 75
1378 crw-r--r-- 1 root root 0, 0 2月 2 11:00 76
1380 crw-r--r-- 1 root root 0, 0 2月 2 11:00 77
1382 crw-r--r-- 1 root root 0, 0 2月 2 11:00 78
1384 crw-r--r-- 1 root root 0, 0 2月 2 11:00 79
1386 crw-r--r-- 1 root root 0, 0 2月 2 11:00 7a
1388 crw-r--r-- 1 root root 0, 0 2月 2 11:00 7b
1390 crw-r--r-- 1 root root 0, 0 2月 2 11:00 7c
1392 crw-r--r-- 1 root root 0, 0 2月 2 11:00 7d
1394 crw-r--r-- 1 root root 0, 0 2月 2 11:00 7e
1396 crw-r--r-- 1 root root 0, 0 2月 2 11:00 7f
1398 crw-r--r-- 1 root root 0, 0 2月 2 11:00 80
1400 crw-r--r-- 1 root root 0, 0 2月 2 11:00 81
1402 crw-r--r-- 1 root root 0, 0 2月 2 11:00 82
1404 crw-r--r-- 1 root root 0, 0 2月 2 11:00 83
1408 crw-r--r-- 1 root root 0, 0 2月 2 11:00 84
1406 crw-r--r-- 1 root root 0, 0 2月 2 11:00 85
1410 crw-r--r-- 1 root root 0, 0 2月 2 11:00 86
1412 crw-r--r-- 1 root root 0, 0 2月 2 11:00 87
1414 crw-r--r-- 1 root root 0, 0 2月 2 11:00 88
1416 crw-r--r-- 1 root root 0, 0 2月 2 11:00 89
1418 crw-r--r-- 1 root root 0, 0 2月 2 11:00 8a
1420 crw-r--r-- 1 root root 0, 0 2月 2 11:00 8b
1422 crw-r--r-- 1 root root 0, 0 2月 2 11:00 8c
1424 crw-r--r-- 1 root root 0, 0 2月 2 11:00 8d
1426 crw-r--r-- 1 root root 0, 0 2月 2 11:00 8e
1428 crw-r--r-- 1 root root 0, 0 2月 2 11:00 8f
1430 crw-r--r-- 1 root root 0, 0 2月 2 11:00 90
1432 crw-r--r-- 1 root root 0, 0 2月 2 11:00 91
1434 crw-r--r-- 1 root root 0, 0 2月 2 11:00 92
1436 crw-r--r-- 1 root root 0, 0 2月 2 11:00 93
1438 crw-r--r-- 1 root root 0, 0 2月 2 11:00 94
1440 crw-r--r-- 1 root root 0, 0 2月 2 11:00 95
1442 crw-r--r-- 1 root root 0, 0 2月 2 11:00 96
1444 crw-r--r-- 1 root root 0, 0 2月 2 11:00 97
1446 crw-r--r-- 1 root root 0, 0 2月 2 11:00 98
1448 crw-r--r-- 1 root root 0, 0 2月 2 11:00 99
1450 crw-r--r-- 1 root root 0, 0 2月 2 11:00 9a
1452 crw-r--r-- 1 root root 0, 0 2月 2 11:00 9b
1454 crw-r--r-- 1 root root 0, 0 2月 2 11:00 9c
1456 crw-r--r-- 1 root root 0, 0 2月 2 11:00 9d
1458 crw-r--r-- 1 root root 0, 0 2月 2 11:00 9e
1460 crw-r--r-- 1 root root 0, 0 2月 2 11:00 9f
1462 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a0
1464 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a1
1466 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a2
1468 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a3
1470 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a4
1472 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a5
1474 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a6
1476 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a7
1478 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a8
1480 crw-r--r-- 1 root root 0, 0 2月 2 11:00 a9
1482 crw-r--r-- 1 root root 0, 0 2月 2 11:00 aa
1484 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ab
1486 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ac
1488 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ad
1490 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ae
1492 crw-r--r-- 1 root root 0, 0 2月 2 11:00 af
1494 dr-xr-xr-x 2 root root 27 8月 10 2021 afs
1497 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b0
1499 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b1
1501 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b2
1503 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b3
1505 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b4
1507 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b5
1509 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b6
1511 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b7
1513 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b8
1515 crw-r--r-- 1 root root 0, 0 2月 2 11:00 b9
1517 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ba
1519 crw-r--r-- 1 root root 0, 0 2月 2 11:00 bb
1521 crw-r--r-- 1 root root 0, 0 2月 2 11:00 bc
1523 crw-r--r-- 1 root root 0, 0 2月 2 11:00 bd
1525 crw-r--r-- 1 root root 0, 0 2月 2 11:00 be
1527 crw-r--r-- 1 root root 0, 0 2月 2 11:00 bf
1529 lrwxrwxrwx 1 root root 7 8月 10 2021 bin -> usr/bin
1536 dr-xr-xr-x 5 root root 323 1月 17 18:15 boot
1661 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c0
1674 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c1
1676 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c2
1678 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c3
1680 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c4
1682 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c5
1684 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c6
1686 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c7
1688 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c8
1690 crw-r--r-- 1 root root 0, 0 2月 2 11:00 c9
1692 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ca
1694 crw-r--r-- 1 root root 0, 0 2月 2 11:00 cb
1696 crw-r--r-- 1 root root 0, 0 2月 2 11:00 cc
1698 crw-r--r-- 1 root root 0, 0 2月 2 11:00 cd
1700 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ce
1702 crw-r--r-- 1 root root 0, 0 2月 2 11:00 cf
1704 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d0
1706 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d1
1708 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d2
1710 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d3
1712 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d4
1714 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d5
1716 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d6
1718 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d7
1720 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d8
1722 crw-r--r-- 1 root root 0, 0 2月 2 11:00 d9
1724 crw-r--r-- 1 root root 0, 0 2月 2 11:00 da
1726 crw-r--r-- 1 root root 0, 0 2月 2 11:00 db
1728 crw-r--r-- 1 root root 0, 0 2月 2 11:00 dc
1730 crw-r--r-- 1 root root 0, 0 2月 2 11:00 dd
1732 crw-r--r-- 1 root root 0, 0 2月 2 11:00 de
1734 drwxr-xr-x 2 root root 27 1月 17 18:14 dev
1737 crw-r--r-- 1 root root 0, 0 2月 2 11:00 df
1739 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e0
1741 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e1
1743 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e2
1745 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e3
1747 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e4
1749 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e5
1751 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e6
1753 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e7
1755 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e8
1757 crw-r--r-- 1 root root 0, 0 2月 2 11:00 e9
1759 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ea
1761 crw-r--r-- 1 root root 0, 0 2月 2 11:00 eb
1763 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ec
1765 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ed
1767 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ee
1769 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ef
1792 drwxr-xr-x 87 root root 3255 1月 17 18:15 etc
4862 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f0
4734 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f1
2942 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f2
4990 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f3
4350 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f4
3710 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f5
2174 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f6
6114 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f7
6116 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f8
6118 crw-r--r-- 1 root root 0, 0 2月 2 11:00 f9
6120 crw-r--r-- 1 root root 0, 0 2月 2 11:00 fa
6122 crw-r--r-- 1 root root 0, 0 2月 2 11:00 fb
6124 crw-r--r-- 1 root root 0, 0 2月 2 11:00 fc
6126 crw-r--r-- 1 root root 0, 0 2月 2 11:00 fd
6128 crw-r--r-- 1 root root 0, 0 2月 2 11:00 fe
6130 crw-r--r-- 1 root root 0, 0 2月 2 11:00 ff
6132 drwxr-xr-x 3 root root 44 1月 17 18:15 home
6162 lrwxrwxrwx 1 root root 7 8月 10 2021 lib -> usr/lib
6165 lrwxrwxrwx 1 root root 9 8月 10 2021 lib64 -> usr/lib64
6168 drwxr-xr-x 2 root root 27 8月 10 2021 media
6171 drwxr-xr-x 2 root root 27 8月 10 2021 mnt
6174 drwxr-xr-x 3 root root 41 1月 17 18:14 opt
15221 drwxr-xr-x 2 root root 27 1月 17 18:14 proc
15224 dr-xr-x--- 3 root root 148 1月 17 18:15 root
15259 drwxr-xr-x 14 root root 260 1月 17 18:15 run
15339 lrwxrwxrwx 1 root root 8 8月 10 2021 sbin -> usr/sbin
15342 drwxr-xr-x 2 root root 27 8月 10 2021 srv
15345 drwxr-xr-x 2 root root 27 1月 17 18:14 sys
15348 drwxrwxrwt 3 root root 54 1月 17 18:15 tmp
15366 drwxr-xr-x 12 root root 209 1月 17 18:14 usr
280740 drwxr-xr-x 18 root root 332 1月 17 18:14 var
