On Tue, May 9, 2017 at 4:40 PM, Brett Niver <bniver@xxxxxxxxxx> wrote: What is your workload like? Do you have a single or multiple active User traffic is heavy. I can't really say in terms of mb/s or iops but it's an email server with 25k+ users, usually about 6k simultaneously connected receiving and reading emails. | |||
| I have only one active MDS configured. The others are Stand-by. | |||
On Tue, May 9, 2017 at 7:18 PM, Wido den Hollander <wido@xxxxxxxx> wrote:
> Op 9 mei 2017 om 20:26 schreef Brady Deetz <bdeetz@xxxxxxxxx>:
>
>
> If I'm reading your cluster diagram correctly, I'm seeing a 1gbps
> interconnect, presumably cat6. Due to the additional latency of performing
> metadata operations, I could see cephfs performing at those speeds. Are you
> using jumbo frames? Also are you routing?
>
> If you're routing, the router will introduce additional latency that an l2
> network wouldn't experience.
>
Partially true. I am running various Ceph clusters using L3 routing and with a decent router the latency for routing a packet is minimal, like 0.02 ms or so.
Ceph spends much more time in the CPU then it will take the network to forward that IP-packet.
I wouldn't be too afraid to run Ceph over a L3 network.
Wido
> On May 9, 2017 12:01 PM, "Webert de Souza Lima" <webert.boss@xxxxxxxxx>
> wrote:
>
> > Hello all,
> >
> > I'm been using cephfs for a while but never really evaluated its
> > performance.
> > As I put up a new ceph cluster, I though that I should run a benchmark to
> > see if I'm going the right way.
> >
> > By the results I got, I see that RBD performs *a lot* better in
> > comparison to cephfs.
> >
> > The cluster is like this:
> > - 2 hosts with one SSD OSD each.
> > this hosts have 2 pools: cephfs_metadata and cephfs_cache (for
> > cache tiering).
> > - 3 hosts with 5 HDD OSDs each.
> > this hosts have 1 pool: cephfs_data.
> >
> > all details, cluster set up and results can be seen here:
> > https://justpaste.it/167fr
> >
> > I created the RBD pools the same way as the CEPHFS pools except for the
> > number of PGs in the data pool.
> >
> > I wonder why that difference or if I'm doing something wrong.
> >
> > Regards,
> >
> > Webert Lima
> > DevOps Engineer at MAV Tecnologia
> > *Belo Horizonte - Brasil*
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
> >
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
