Re: question: the number of connections in ceph

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Mmm, I did the math wrong (I think I left out a factor of pool_size).
For 32 pgs over 3 hosts with 12 osds each, each osd should have ~3 pgs
(check ceph pg dump), I think.  Each osd should therefore have ~6
peers on the other two hosts (again, ceph pg dump).  At 3 connections
per peer, that would be 18 connections.  You should be able to observe
this empirically though -- I've probably got the number of
connections/pair wrong.

Btw, 32 pgs for 36 osds is *way* too few.  I think there are some
tuning guides around you'll probably want to look at.
-Sam

On Sun, Aug 30, 2020 at 10:26 PM Yin, Congmin <congmin.yin@xxxxxxxxx> wrote:
>
> Hi Anthony,
> My main question is the number of connections between OSDs. Of course, CEPH cluster will have other additional connections.
> For example,  if I have three cluster servers with 3 replication, and there are 12 OSDs on them. Then I test with FIO that there are 8 images in a pool. And the pool pgs num is 32. What is the theoretical connection tree between OSDs
>
> @Sam Just  For the example above, per OSD connection number: (num_pgs/num_osds * (pool_size - 1))= (32/12*(3-1))=5.3? Does this have nothing to do with the number of threads and images started by the test?
>
>
> Best regards.
> Congmin Yin
>
>
> -----Original Message-----
> From: Anthony D'Atri <anthony.datri@xxxxxxxxx>
> Sent: Saturday, August 29, 2020 4:24 AM
> To: Sam Just <sjust@xxxxxxxxxx>
> Cc: Yin, Congmin <congmin.yin@xxxxxxxxx>; dev@xxxxxxx
> Subject: Re: question: the number of connections in ceph
>
> Last time I read through the code, there were a limited number of “extra” peerings added to the acting set OSDs too.
>
> But it’s a bit unclear what a “connection” means. And if that includes clients.
>
> > On Aug 28, 2020, at 12:56 PM, Sam Just <sjust@xxxxxxxxxx> wrote:
> >
> > I you're asking about the osd<->osd connection count for a particular
> > osd, it'll be proportional (two hearbeat connections and a cluster
> > connection per pair I think) to the number of osds that osd shares a
> > pg with (num_pgs/num_osds * (pool_size - 1)).
> > -Sam
> >
> > On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin@xxxxxxxxx> wrote:
> >>
> >> Hi ,
> >>
> >>
> >>
> >> I want to know the number of connections in CEPH. I think the connection is mainly OSD connection.
> >>
> >> Is the following statement correct?
> >>
> >> Each OSD is connected with other OSDs, and there may be more than one connection between two OSDs.
> >>
> >> If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If there are k connections per OSD, the number of the connection is kN(N-1)/2.
> >>
> >>
> >>
> >> Thanks for your help.
> >>
> >>
> >>
> >> Best regards.
> >>
> >> Congmin Yin
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Dev mailing list -- dev@xxxxxxx
> >> To unsubscribe send an email to dev-leave@xxxxxxx
> > _______________________________________________
> > Dev mailing list -- dev@xxxxxxx
> > To unsubscribe send an email to dev-leave@xxxxxxx
>
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx




[Index of Archives]     [CEPH Users]     [Ceph Devel]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux