Great! Thanks, that was it. I did see mention of that param in the mailing list, and thought that might be it. But I failed to find that option in the docs here: http://ceph.com/docs/master/config-ref/osd-config/ So I wasn't sure where to put it. =) Thanks again. On Tue, Jun 19, 2012 at 1:08 PM, Sage Weil <sage@xxxxxxxxxxx> wrote: > On Tue, 19 Jun 2012, Travis Rhoden wrote: >> I almost posted this to http://tracker.newdream.net/issues/2595, but >> didn't want to piggy-back on an issue marked resolved. >> >> When I run mkcephfs, I get: >> >> 2012-06-19 09:36:29.211737 7fc7021d7780 -1 journal FileJournal::_open: >> unable to open journal: open() failed: (22) Invalid argument >> 2012-06-19 09:36:29.211827 7fc7021d7780 -1 OSD::mkfs: FileStore::mkfs >> failed with error -22 >> 2012-06-19 09:36:29.211852 7fc7021d7780 -1 ** ERROR: error creating >> empty object store in /srv/osd.0: (22) Invalid argument > > By default the journal code uses O_DIRECT, which tmpfs does not support. > Try adding > > journal dio = false > > to the [osd] section and see if that helps! > > sage > > >> >> I'm running 0.47.2 on Ubuntu 12.04. Here is the osd section of my ceph.conf: >> >> [osd] >> osd data = /srv/osd.$id >> osd journal = /tmpfs/osd.$id.journal >> osd journal size = 1000 >> filestore fiemap = false >> ; uncomment the following line if you are mounting with ext4 >> ; filestore xattr use omap = true >> keyring = /etc/ceph/keyring.$name >> >> >> The problem arose when I was trying to move my journals onto tmpfs. I >> blew away my old cluster (I'm just testing different configs), and >> then hit this when I tried to make a new one. The reason I think this >> is differen than issue # 2595 is that my mkcephfs does NOT succeed, >> whereas the other user's did. The journal file at >> /tmpfs/osd.$id/journal does get created but it is 0 bytes. >> >> # ls -al /tmpfs >> total 4 >> drwxrwxrwt 2 root root 60 Jun 19 09:36 . >> drwxr-xr-x 24 root root 4096 Jun 19 09:08 .. >> -rw-r--r-- 1 root root 0 Jun 19 09:36 osd.0.journal >> >> # df -h /tmpfs >> Filesystem Size Used Avail Use% Mounted on >> tmpfs 2.5G 0 2.5G 0% /tmpfs >> >> My tmpfs looks like this: >> >> # mount | grep tmpfs >> tmpfs on /tmpfs type tmpfs (rw,size=2560m) >> >> I threw debug filestore = 20 into ceph.conf and capture this in the log file: >> >> 2012-06-19 09:36:28.888220 7fc7021d7780 1 filestore(/srv/osd.0) mkfs >> in /srv/osd.0 >> 2012-06-19 09:36:28.888379 7fc7021d7780 0 filestore(/srv/osd.0) mkfs >> removing old directory current >> 2012-06-19 09:36:28.890923 7fc7021d7780 0 filestore(/srv/osd.0) mkfs >> removing old file store_version >> 2012-06-19 09:36:28.911798 7fc7021d7780 10 filestore(/srv/osd.0) mkfs >> fsid is 52ab0ff0-bb36-472d-93d5-7a02f0dfa5e5 >> 2012-06-19 09:36:28.911892 7fc7021d7780 2 filestore(/srv/osd.0) >> BTRFS_IOC_SUBVOL_CREATE ioctl failed, trying mkdir /srv/osd.0/current >> 2012-06-19 09:36:29.211668 7fc7021d7780 -1 filestore(/srv/osd.0) >> leveldb db created >> 2012-06-19 09:36:29.211700 7fc7021d7780 10 filestore(/srv/osd.0) >> open_journal at /tmpfs/osd.0.journal >> 2012-06-19 09:36:29.211737 7fc7021d7780 -1 journal FileJournal::_open: >> unable to open journal: open() failed: (22) Invalid argument >> 2012-06-19 09:36:29.211814 7fc7021d7780 0 filestore(/srv/osd.0) >> mkjournal error creating journal on /tmpfs/osd.0.journal >> 2012-06-19 09:36:29.211827 7fc7021d7780 -1 OSD::mkfs: FileStore::mkfs >> failed with error -22 >> 2012-06-19 09:36:29.211852 7fc7021d7780 -1 ^[[0;31m ** ERROR: error >> creating empty object store in /srv/osd.0: (22) Invalid argument^[[0m >> >> >> The OSD that I am working with was used before in my old cluster, but >> I blew it away first by unmounting and reformatting the partition ( >> just a simple 'mkfs.xfs /dev/md0' ). The device is software RAID 0. >> Again, I'm just playing with configs to see what the fastest >> performance i can get out of my hardware is. It seemed like maybe the >> journal was holding me back, so I'm trying to move it to /tmpfs. >> >> I feel like I'm missing something obvious. >> >> - Travis >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
