On 10/2/17 8:14 AM, R. Jason Adams wrote:
> Hello,
>
> I have a use case where I'm writing ~500Kb (avg size) files to a 10TB XFS file systems. Each of system has 36 of these 10TB drives.
On what version of kernel & what version of xfsprogs?
> The application opens the file, writes the data (single call), and closes the file. In addition there are a few lines added to the extended attributes. The filesystem ends up with 18 to 20 million files when the drive is full. The files are currently spread over 128x128 directories using a hash of the filename.
It's not uncommon for application filename hashing like this to be less efficient than the internal xfs directory algorithms, FWIW.
> The format command I'm using:
>
> mkfs.xfs -f -i size=1024 ${DRIVE}
Why 1k inodes?
> Mount options:
>
> rw,noatime,attr2,inode64,allocsize=2048k,logbufs=8,logbsize=256k,noquota
Why all these options?
> As the drive is filling, the first few % of the drive seems fine. Using iostat the avgrq-sz is close to the average file size. What I'm noticing is as the drive starts to fill (say around 5-10%) the reads start increasing (r/s in iostat). In addition, the avgrq-sz starts to decrease. Pretty soon the r/s can be 1/3 to 1/2 as many as our w/s. At first we thought this was related to using extended attributes, but disabling that didn’t make a difference at all.
>
> Considering I know the app isn’t making any read request, I’m guessing this is related to updating metadata etc. Any guidance on how to resolve/reduce/etc? For example, would a different directory structure help (more files in less directories)?
Perhaps it's taking time reading through a large custom-hashed directory tree? I don't know what that custom directory layout might look like.
http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
Have you tried starting with defaults, and working your way up from there (if needed?)
Thanks,
-Eric
> Thanks,
> R. Jason Adams
--
To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
