Mmm, I did the math wrong (I think I left out a factor of pool_size). For 32 pgs over 3 hosts with 12 osds each, each osd should have ~3 pgs (check ceph pg dump), I think. Each osd should therefore have ~6 peers on the other two hosts (again, ceph pg dump). At 3 connections per peer, that would be 18 connections. You should be able to observe this empirically though -- I've probably got the number of connections/pair wrong. Btw, 32 pgs for 36 osds is *way* too few. I think there are some tuning guides around you'll probably want to look at. -Sam On Sun, Aug 30, 2020 at 10:26 PM Yin, Congmin <congmin.yin@xxxxxxxxx> wrote: > > Hi Anthony, > My main question is the number of connections between OSDs. Of course, CEPH cluster will have other additional connections. > For example, if I have three cluster servers with 3 replication, and there are 12 OSDs on them. Then I test with FIO that there are 8 images in a pool. And the pool pgs num is 32. What is the theoretical connection tree between OSDs > > @Sam Just For the example above, per OSD connection number: (num_pgs/num_osds * (pool_size - 1))= (32/12*(3-1))=5.3? Does this have nothing to do with the number of threads and images started by the test? > > > Best regards. > Congmin Yin > > > -----Original Message----- > From: Anthony D'Atri <anthony.datri@xxxxxxxxx> > Sent: Saturday, August 29, 2020 4:24 AM > To: Sam Just <sjust@xxxxxxxxxx> > Cc: Yin, Congmin <congmin.yin@xxxxxxxxx>; dev@xxxxxxx > Subject: Re: question: the number of connections in ceph > > Last time I read through the code, there were a limited number of “extra” peerings added to the acting set OSDs too. > > But it’s a bit unclear what a “connection” means. And if that includes clients. > > > On Aug 28, 2020, at 12:56 PM, Sam Just <sjust@xxxxxxxxxx> wrote: > > > > I you're asking about the osd<->osd connection count for a particular > > osd, it'll be proportional (two hearbeat connections and a cluster > > connection per pair I think) to the number of osds that osd shares a > > pg with (num_pgs/num_osds * (pool_size - 1)). > > -Sam > > > > On Thu, Aug 27, 2020 at 1:38 AM Yin, Congmin <congmin.yin@xxxxxxxxx> wrote: > >> > >> Hi , > >> > >> > >> > >> I want to know the number of connections in CEPH. I think the connection is mainly OSD connection. > >> > >> Is the following statement correct? > >> > >> Each OSD is connected with other OSDs, and there may be more than one connection between two OSDs. > >> > >> If there is only one connection per OSD, the number of the connection is N(N-1)/2,. If there are k connections per OSD, the number of the connection is kN(N-1)/2. > >> > >> > >> > >> Thanks for your help. > >> > >> > >> > >> Best regards. > >> > >> Congmin Yin > >> > >> > >> > >> > >> > >> _______________________________________________ > >> Dev mailing list -- dev@xxxxxxx > >> To unsubscribe send an email to dev-leave@xxxxxxx > > _______________________________________________ > > Dev mailing list -- dev@xxxxxxx > > To unsubscribe send an email to dev-leave@xxxxxxx > _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
