Robert,
xfs on the rbd image as well:
/dev/rbd0 on /mnt/ceph-block-device-archive type xfs (rw)
However looking at the mount options...it does not look like I've enabled anything special in terms of mount options.
Thanks,
Shain
Shain Miley | Manager of Systems and Infrastructure, Digital Media | smiley@xxxxxxx | 202.513.3649
________________________________________
From: Robert LeBlanc [robert@xxxxxxxxxxxxx]
Sent: Tuesday, January 06, 2015 1:27 PM
To: Shain Miley
Cc: ceph-users@xxxxxxxx
Subject: Re: rbd directory listing performance issues
What fs are you running inside the RBD?
On Tue, Jan 6, 2015 at 8:29 AM, Shain Miley <SMiley@xxxxxxx> wrote:
> Hello,
>
> We currently have a 12 node (3 monitor+9 OSD) ceph cluster, made up of 107 x
> 4TB drives formatted with xfs. The cluster is running ceph version 0.80.7:
>
> Cluster health:
> cluster 504b5794-34bd-44e7-a8c3-0494cf800c23
> health HEALTH_WARN crush map has legacy tunables
> monmap e1: 3 mons at
> {hqceph1=10.35.1.201:6789/0,hqceph2=10.35.1.203:6789/0,hqceph3=10.35.1.205:6789/0},
> election epoch 156, quorum 0,1,2 hqceph1,hqceph2,hqceph3
> osdmap e19568: 107 osds: 107 up, 107 in
> pgmap v10117422: 2952 pgs, 15 pools, 77202 GB data, 19532 kobjects
> 226 TB used, 161 TB / 388 TB avail
>
> Relevant ceph.conf entries:
> osd_journal_size = 10240
> filestore_xattr_use_omap = true
> osd_mount_options_xfs =
> "rw,noatime,nodiratime,logbsize=256k,logbufs=8,inode64"
> osd_mkfs_options_xfs = "-f -i size=2048"
>
>
> A while back I created an 80 TB rbd image to be used as an archive
> repository for some of our audio and video files. We are still seeing good
> rados and rbd read and write throughput performance, however we seem to be
> having quite a long delay in response times when we try to list out the
> files in directories with a large number of folders, files, etc.
>
> Subsequent directory listing times seem to run a lot faster (but I am not
> sure for long that is the case before we see another instance of slowness),
> however the initial directory listings can take 20 to 45 seconds.
>
> The rbd kernel client is running on ubuntu 14.04 using kernel version
> '3.18.0-031800-generic'.
>
> Benchmarks:
>
> root@rbdmount1:/mnt/rbd/music_library/D#time ls (file names removed):
> real 0m18.045s
> user 0m0.000s
> sys 0m0.011s
>
> root@rbdmount1:/mnt/rbd# dd bs=1M count=1024 if=/dev/zero of=test
> conv=fdatasync
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 9.94287 s, 108 MB/s
>
>
> My questions are:
>
> 1) Is there anything inherent in our setup/configuration that would prevent
> us from having fast directory listings on these larger directories (using an
> rbd image of that size for example)?
>
> 2) Have there been any changes made in Giant that would warrant upgrading
> the cluster a a fix to resolve this issue?
>
> Any suggestions would be greatly appreciated.
>
> Thanks,
>
> Shain
>
>
> Shain Miley | Manager of Systems and Infrastructure, Digital Media |
> smiley@xxxxxxx | 202.513.3649
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
