Jon; For all intents and purposes, an RBD device is treated by the system/OS as a physical disk, albeit attached via the network where multiple servers store the data (ceph cluster). Once the RBD device is created, one needs to format the device using any one of a number of file systems (ext4, btrfs, xfs, etc.). If one wants/needs to share that RBD device, one will need to use a cluster-aware file system in place of ext4, xfs, etc., like gfs, ocfs2, etc. This, however, adds a layer of complexity that you may not be looking for. If you need to share the file system amongst several hosts, with concurrent reading/writing, then you’ll probably need to look into using CephFS if you want to continue with the notion of using a Ceph cluster. CephFS, however, isn’t currently considered “production ready”, although many are using it in this manner. Brad From: Jon [mailto:three18ti@xxxxxxxxx] Hello Bradley, Please excuse my ignorance, I am new to CEPH and what I thought was a good understanding of file systems has clearly been shown to be inadequate. Maybe I'm asking the question wrong because I keep getting the same answer. I guess I don't understand what a clustered filesystem is. I thought CEPH was a clustered file system, and the Wikipedia article on clustered file systems doesn't offer any disambiguation as CEPH is listed as a clustered filesystem, albeit under the distributed heading. There is a shared disk / storage area network section, but, with the exception of GFS, these look like SAN technologies, not just file systems. I guess my question boils down to: "what format should I format my rbd so that I can read and write to it from multiple diverse hosts"? Can I format my rbd as gfs2 and call it a day? Or do I need to use a technology like glusterFS, or mount the rbd on one host and export it as an NFS Mount? (Would really like to avoid NFS if at all possible, but if that's the solution, then that's the solution). Thanks for your patience with me. I really feel like an idiot asking, but I really have no where else to turn. Thanks, On May 29, 2013 12:01 PM, "McNamara, Bradley" <Bradley.McNamara@xxxxxxxxxxx> wrote: Instead of using ext4 for the file system, you need to use a clustered file system on the RBD device. From: ceph-users-bounces@xxxxxxxxxxxxxx [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Jon Hell Igor, Thanks for getting back to me. > > You can map it to multiple hosts, but before doing dd if=/dev/zero of=/media/tmp/test you have created file system, right Correct, I can't mount /dev/rbd/rbd/test-device without first creating a file system on the device. Now, I am creating an ext4 filesystem with the -m0 flag so the metadata is contained within the filesystem. I did not create a partition on the device instead opting to format the whole device (this is how the rbd guide does it). >> This file system MUST be distributed, thus multiple hosts can read and write files on it. I'm not sure I understand. Are you implying I need to use some OTHER network filesystem (e.g. glusterfs) -on top- of ceph? Or are you saying my ext4 file system should be distributed so multiple hosts should be able to read and write to/from the ext4 filesystem? If it's the former, this seems counterintuitive, but if that's what nerds to happen I guess I'll make it so. If it's the latter, then something is not right as my hosts are not all able to read and write to the ext4 filesystem. I'm not sure how else I can test / prove it other than writing a file from each of my hosts and that file not being accessible from all of my hosts, can you provide some further troubleshooting steps? Could it be the filesystem type? Do I need to use xfs or btrfs if I want to map the block device to multiple hosts? Does CEPH not work as a client where it is running as a service? Thanks for your help, On May 29, 2013 12:47 AM, "Igor Laskovy" <igor.laskovy@xxxxxxxxx> wrote: Hi Jon, I already mentioned multiple times here - RBD just a block device. You can map it to multiple hosts, but before doing dd if=/dev/zero of=/media/tmp/test you have created file system, right? This file system MUST be distributed, thus multiple hosts can read and write files on it. On Wed, May 29, 2013 at 4:24 AM, Jon <three18ti@xxxxxxxxx> wrote: Hello, I would like to mount a single RBD on multiple hosts to be able to share the block device. Is this possible? I understand that it's not possible to share data between the different interfaces, e.g. CephFS and RBDs, but I don't see anywhere it's declared that sharing an RBD between hosts is or is not possible. I have followed the instructions on the github page of ceph-deploy (I was following the 5 minute quick start http://ceph.com/docs/next/start/quick-start/ but when I got to the step with mkcephfs it erred out and pointed me to the github page), as I only have three servers I am running the osds and monitors on all of the hosts, I realize this isn't ideal but I'm hoping it will work for testing purposes. This is what my cluster looks like: >> root@red6:~# ceph -s >> health HEALTH_OK >> monmap e2: 3 mons at {kitt=192.168.0.35:6789/0,red6=192.168.0.40:6789/0,shepard=192.168.0.2:6789/0}, election epoch 10, quorum 0,1,2 kitt,red6,shepard >> osdmap e29: 5 osds: 5 up, 5 in >> pgmap v1692: 192 pgs: 192 active+clean; 19935 MB data, 40441 MB used, 2581 GB / 2620 GB avail; 73B/s rd, 0op/s >> mdsmap e1: 0/0/1 up To test, what I have done is created a 20GB RBD mapped it and mounted it to /media/tmp on all the hosts in my cluster, so all of the hosts are also clients. Then I use dd to create a 1MB file named test-$hostname >> dd if=/dev/zero of=/media/tmp/test-`hostname` bs=1024 count=1024; after the file is created, I wait for the writes to finish in `ceph -w`, then on each host when I list /media/tmp I see the results of /media/tmp/test-`hostname`, if I unmount then remount the RBD, I get mixed results. Typically, I see the file that was created on the host that is at the front of the line in the quorum. e.g. the test I did while typing this e-mail "kitt" is listed first quorum 0,1,2 kitt,red6,shepard, this is the file I see created when I unmount then mount the rbd on shepard. Where this is going is, I would like to use CEPH as my back end storage solution for my virtualization cluster. The general idea is the hypervisors will all have a shared mountpoint that holds images and vms so vms can easily be migrated between hypervisors. Actually, I was thinking I would create one mountpoint each for images and vms for performance reasons, am I likely to see performance gains using more smaller RBDs vs fewer larger RBDs? Thanks for any feedback, Jon A
-- Igor Laskovy |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
