Okay. As a quick guess you probably used a CRUSH placement option with your new pools that wasn't supported by the old kernel, although it might have been something else. I suspect that you'll find FUSE works better for you anyway as long as you can use it — faster updates from us to you. ;) -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Thu, Oct 10, 2013 at 10:53 AM, Frerot, Jean-Sébastien <jsfrerot@xxxxxxxxxxxxxxxx> wrote: > Hi, > Thx for your reply :) > > kernel: Linux compute01 3.8.0-31-generic #46-Ubuntu SMP Tue Sep 10 20:03:44 > UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > > So yes I'm using cephfs and was also using rdb at the same time using > different pools. My ceph fs was setup 3 months ago and I upgraded it a > couple of days ago. I move VM images from rdb to the cephfs by copying the > file from rdb to local FS then to cephfs. > > I create pools like this: > ceph osd pool create volumes 128 > ceph osd pool create images 128 > ceph osd pool create live_migration 128 > > Yes I had checked dmesg but didn't find anything relevant. > > However, as a last resort I decided to mount my FS using fuse. And it works > like a charm. So for now I'm sticking with fuse :) > > Let me know if you want me to do some explicit testing. It may take some > time for me to do them as I'm using ceph but I can manage to have some time > for maintenances. > > Regards, > > > -- > Jean-Sébastien Frerot > jsfrerot@xxxxxxxxxxxxxxxx > > > 2013/10/10 Gregory Farnum <greg@xxxxxxxxxxx> >> >> (Sorry for the delayed response, this was in my spam folder!) >> >> Has this issue persisted? Are you using the stock 13.04 kernel? >> >> Can you describe your setup a little more clearly? It sounds like >> maybe you're using CephFS now and were using rbd before; is that >> right? What data did you move, when, and how did you set up your >> CephFS to use the pools? >> The socket errors are often a slightly spammy notification that the >> socket isn't in use but has shut down; here they look to be an >> indicator of something actually gone wrong — perhaps you've >> inadvertently activated features incompatible with your kernel client, >> but let's see what's going on more before we jump to that conclusion. >> have you checked dmesg for anything else at those points? >> -Greg >> Software Engineer #42 @ http://inktank.com | http://ceph.com >> >> On Sat, Oct 5, 2013 at 6:42 PM, Frerot, Jean-Sébastien >> <jsfrerot@xxxxxxxxxxxxxxxx> wrote: >> > Hi, >> > I have a ceph cluster running with 3 physical servers, >> > >> > Here is how my setup is configured >> > server1: mon, osd, mds >> > server2: mon, osd, mds >> > server3: mon >> > OS ubuntu 13.04 >> > ceph version: 0.67.4-1raring (recentrly upgrade to see if my problem >> > still >> > persisted with the new version) >> > >> > So I was running version CUTTLEFISH until yesterday. And I was using >> > ceph >> > with openstack (using rdb) but I simplified my setup and removed >> > openstack >> > to simply use kvm with virtmanager. >> > >> > So I created a new pool to be able to do live migration of kvm instances >> > #ceph osd lspools >> > 0 data,1 metadata,2 rbd,3 volumes,4 images,6 live_migration, >> > >> > I've been running VMs for some days without problems, but then I notice >> > that >> > I couldn't use the full disk size of my first VM (web01 which was 160G >> > big >> > originaly) but now is only 119G stored in ceph. I also have a windows >> > instance running on a 300G raw file located in ceph too. So trying to >> > fix >> > the issue I decided to do a local backup of my file in cause something >> > goes >> > wrong and guess what, i wasn't able to copy the file from ceph to my >> > local >> > drive. The moment I tried to do that "cp live_migration/web01 /mnt/" the >> > OS >> > hangs, and syslog show this >30 lines/s: >> > >> > Oct 5 15:25:45 server2 kernel: [ 8773.432358] libceph: osd1 >> > 192.168.0.131:6803 socket error on read >> > >> > i couldn't kill my cp neither normally reboot my server. So I had to >> > reset >> > it. >> > >> > I tried to copy my other file "win2012" also stored in the ceph cluster >> > and >> > get the same issue and now I can't read anything from it nor start my VM >> > again >> > >> > [root@server1 ~]# ceph status >> > cluster 50dc0404-c081-4c43-ac3f-872ba5494bd7 >> > health HEALTH_OK >> > monmap e4: 3 mons at >> > >> > {server1=192.168.0.130:6789/0,server2=192.168.0.131:6789/0,server3=192.168.0.132:6789/0}, >> > election epoch 120, quorum 0,1,2 server1,server2,server3 >> > osdmap e275: 2 osds: 2 up, 2 in >> > pgmap v1508209: 576 pgs: 576 active+clean; 108 GB data, 214 GB used, >> > 785 >> > GB / 999 GB avail >> > mdsmap e181: 1/1/1 up {0=server2=up:active}, 1 up:standby >> > >> > I mount the FS with fstab like this: >> > 192.168.0.131:6789,192.168.0.130:6789:/live_migration /var/lib/instances >> > ceph name=live_migration,secret=mysecret==,noatime 0 2 >> > >> > I get this log in ceph-osd.0.log as spammy as "socket error on read" >> > error i >> > get in syslog >> > 2013-10-05 23:07:23.586807 7f24731cc700 0 -- 192.168.0.130:6801/19182 >> > >> >> > 192.168.0.130:0/4212596483 pipe(0x128d8500 sd=115 :6801 s=0 pgs=0 cs=0 >> > l=0 >> > c=0x14ac09a0).accept peer addr is rea >> > lly 192.168.0.130:0/4212596483 (socket is 192.168.0.130:35078/0) >> > >> > other infos: >> > df -h >> > /dev/mapper/server1--vg-ceph 500G 108G 393G >> > 22% >> > /opt/data/ceph >> > 192.168.0.131:6789,192.168.0.130:6789:/live_migration 1000G 215G 786G >> > 22% >> > /var/lib/instances >> > ... >> > >> > mount >> > /dev/mapper/server1--vg-ceph on /opt/data/ceph type xfs (rw,noatime) >> > 192.168.0.131:6789,192.168.0.130:6789:/live_migration on >> > /var/lib/instances >> > type ceph (name=live_migration,key=client.live_migration) >> > ... >> > >> > >> > How can I recover from this ? >> > >> > Thank you, >> > -- >> > Jean-Sébastien Frerot >> > jsfrerot@xxxxxxxxxxxxxxxx >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@xxxxxxxxxxxxxx >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |