The osd data directories are on /mnt/osd$id which are btrfs. The mon data directory is /mnt/local/mon$id which is xfs. (/mnt/local is the mount point) Here is the ceph.conf file (note: secure authentication is turned off at the moment): ; global [global] ; enable secure authentication ; auth supported = cephx ; monitors ; You need at least one. You need at least three if you want to ; tolerate any node failures. Always create an odd number. [mon] mon data = /mnt/local/mon$id ; some minimal logging (just message traffic) to aid debugging debug ms = 0 [mon.0] host = 172.16.0.130 mon addr = 172.16.0.130:6789 [mon.1] host = 172.16.0.128 mon addr = 172.16.0.128:6789 [mon.2] host = 172.16.0.129 mon addr = 172.16.0.129:6789 ; mds ; You need at least one. Define two to get a standby. [mds] ; where the mds keeps its secret encryption keys keyring = /mnt/local/keyring.$name [mds.node-172-16-0-130] host = 172.16.0.130 [mds.node-172-16-0-128] host = 172.16.0.128 [mds.node-172-16-0-129] host = 172.16.0.129 mds standby replay = true mds standby for name = node-172-16-0-130 [osd] ; This is where the btrfs volume will be mounted. osd data = /mnt/osd$id ; Ideally, make this a separate disk or partition. A few GB ; is usually enough; more if you have fast disks. You can use ; a file under the osd data dir if need be ; (e.g. /mnt/osd$id/journal), but it will be slower than a ; separate disk or partition. osd journal = /mnt/osd$id/journal osd class dir = /lib/rados-classes ; If the OSD journal is a file, you need to specify the size. This is specified in MB. osd journal size = 512 [osd.0] host = 172.16.0.130 [osd.1] host = 172.16.0.130 [osd.2] host = 172.16.0.130 [osd.3] host = 172.16.0.130 [osd.24] host = 172.16.0.128 [osd.25] host = 172.16.0.128 [osd.26] host = 172.16.0.128 [osd.27] host = 172.16.0.128 [osd.48] host = 172.16.0.129 [osd.49] host = 172.16.0.129 [osd.50] host = 172.16.0.129 [osd.51] host = 172.16.0.129 On Thu, Nov 3, 2011 at 12:23 PM, Tommi Virtanen <tommi.virtanen@xxxxxxxxxxxxx> wrote: > On Thu, Nov 3, 2011 at 12:09, Mandell Degerness <mandell@xxxxxxxxxxxxxxx> wrote: >> We are currently running all services on each of the three nodes. >> (mon, mds, and several osd processes). > > Is the osd data directory on the same filesystem as the mon data dir, > or are they perhaps on different disks? (Hoping for the former, for > the sake of xfs credibility ;) > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html