Hi, On Tue, 12 Oct 2010, DongJin Lee wrote: > Thanks a lot, the mount works smoothly without a problem. > So I'm with ceph version 0.23~rc (a7ed2ee05dc7453942018d7876401c28d3918214) > All testings are done in a single high-end machine. > > For btrfs method, I get a freeze/hang while reading/writing to it. > It's the step that I've never gone passed :( > A few read/write copies are okay, I can drag/drop from the ubuntu > desktop, cp, and read files back. so far so good.. > However, whenever I tried to do anything large e.g., copying a > directory (total 2.3GB consisting 52,000 files), running large dd > (>1GB+), dbench or any heav IOs,, things just lock up, once at this > point, I can't read the directory and I can't unmount, basically > leading me to restart the system. It is reproducible and can provide > some debugging info needed, just tell me which one :) Can you cat the /sys/kernel/debug/ceph/*/mdsc and osdc files? That will show in flight MDS and OSD requests. Also, the output from 'ceph -s' will summarize cluster status/health. If you see pending OSD requests, can you also enable logging on the osds with debug ms = 1 debug osd = 20 in the [osd] section, restart the osd daemons, and reproduce the hang? > Some other issues. > > when I start the ceph, how can I really make sure every btrfs contents > are removed? The cosd --mkfs step will clear out old data. > > sudo mkcephfs -c /etc/ceph/ceph.conf --allhosts --clobber_old_data --mkbtrfs -v > > after this, when I look at data/osd0, 1, 2, etc. I see 'current' > directory but filled with lots of x.x_head directory. > I'd thought the freshly-started shouldn't have those head directories. > I don't think it is normal?. This is normal. Those are the empty placement group directories. > To get rid of these, I format the disk to ext3/4 and then use the > above command too create btrfs.. and the heads are gone. The --mkbtrfs option to mkcephfs won't do the mkfs.btrfs if there is no btrfs devs defined. But you should still see the current and placement group dirs after running mkcephfs, regardless of whether you're using btrfs or ext3/4. > > When I tried to mkcephfs without btrfs, (i.e., using ext3/4 fs), I get > a superblock error. > [osd] > sudo = true > osd journal = /data/osd$id/journal > osd journal size = 1000 > osd data = /data/osd$id > [osd0] > host = ss1 > ; I've removed btrfs > > mkcephfs -c /etc/ceph/ceph.conf --allhosts -v > > ... > === osd.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd journal" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "keyring" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs options" "noatime" > --- ss1# test -d /data/osd0 || mkdir -p /data/osd0 > --- ss1# test -d /data/osd0/journal || mkdir -p /data/osd0 > --- ss1# /usr/bin/cosd -c /etc/ceph/ceph.conf --monmap > /tmp/monmap.5472 -i 0 --mkfs --osd-data /data/osd0 > ** WARNING: Ceph is still under heavy development, and is only suitable for ** > ** testing and review. Do not trust it with important data. ** > ** ERROR: error creating empty object store in /data/osd0: Operation > not supported > failed: '/usr/bin/cosd -c /etc/ceph/ceph.conf --monmap > /tmp/monmap.5472 -i 0 --mkfs --osd-data /data/osd0 > > I am unsure why it sets btrfs path to /data/osd0 when I clearly > commented out ; and no btrfs was mentioned in mkcephfs and conf, The 'btrfs path' is the mount point, which defaults to the same as 'osd data'. It doesn't affect anythign in your case, though. The real question is where the EOPNOTSUPP is coming from. Is /data/osd0 a btrfs or ext3/4 file system at this point? Can you enable the osd debug options mentioned above, and then attach the osd log fragment from /var/log/ceph/osd.0.log generated by the mkcephfs attempt? Thanks! sage > In the data/osd0 (Ext3/4), I see current directory, fsid, and the journal > > Thanks > > On Tue, Oct 12, 2010 at 5:40 AM, Colin McCabe <cmccabe@xxxxxxxxxxxxxx> wrote: > > On Sun, Oct 10, 2010 at 4:49 PM, DongJin Lee <dongjin.lee@xxxxxxxxxxxxxx> wrote: > >> make[1]: Entering directory `/usr/src/linux-headers-2.6.35-02063507-generic' > >> INSTALL /home/aaa/ceph/ceph-client-standalone/ceph.ko > >> DEPMOD 2.6.35-02063507-generic > >> make[1]: Leaving directory `/usr/src/linux-headers-2.6.35-02063507-generic' > >> aaa@ss1:~/ceph/ceph-client-standalone$ sudo depmod > >> aaa@ss1:~/ceph/ceph-client-standalone$ sudo modprobe ceph > >> aaa@ss1:~/ceph/ceph-client-standalone$ lsmod | grep ceph > >> ceph 228785 0 > >> libcrc32c 1300 2 ceph,btrfs > >> aaa@ss1:~/ceph/ceph-client-standalone$ sudo mount -t ceph > >> 192.168.1.2:/ /media/ceph > >> failed to parse ceph_options > >> > >> aaa@ss1:~/ceph/ceph-client-standalone$ sudo rmmod ceph > >> aaa@ss1:~/ceph/ceph-client-standalone$ lsmod | grep ceph > >> aaa@ss1:~/ceph/ceph-client-standalone$ sudo mount -t ceph > >> 192.168.1.2:/ /media/ceph > >> failed to parse ceph_options > >> > >> Again, now nothing works :( everything I do I get 'failed to parse ceph_options' > >> > > > > Hi Dongjin, > > > > There was a bug where not supplying any options to mount.ceph caused > > this error message to be displayed. I can see here that you are not > > supplying any optinos (using -o) so you are probably hitting this bug. > > > > This should be fixed by commit > > 566292a5871686e612b30bee58481db489b27bfb. Try checking out the latest > > in unstable and this will be resolved. > > > > regards, > > Colin McCabe > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > >
