Hi ,
Are your OSDs disk file system btrfs ? or ext4 ,ext3 , or ??
i use ceph 0.23 , ubuntu 10.10 x86_64,2.6.35-22 ,mount btrfs osd is
successful, but mount ext4 osd fail, with its logs , it seems that it
should set "osd journal = /PATH/TO/JOURNAL" at ceph.conf and set
"user_xattr" at /etc/fstab. you can try it.
jeff
å 2010-11-11åç 16:19 +0800ïWido den Hollanderåéï
> Hi,
>
> Right now only the mount process is in status D, not the cosd process,
> do you still get the same messages in your dmesg?
>
> Wido
>
> On Wed, 2010-11-10 at 23:16 +0530, srimugunthan dhandapani wrote:
> > Hi
> > The process with the D state is:
> > root 5167 0.0 0.0 1804 240 pts/0 D+ 23:09 0:00
> > /sbin/mount.ceph 192.168.1.79 / /ceph -o rw
> >
> > ps aux | grep ceph output is:
> >
> > 1000 1419 3.1 1.0 89684 21016 ? S 23:04 0:15 gedit
> > /home/srimugunthan/Desktop/ceph_checklist
> > root 4072 0.2 0.1 49604 3228 ? Ssl 23:08 0:00
> > /usr/bin/cmon -i 0 -c /tmp/ceph.conf.3544
> > root 4589 0.4 0.1 73652 2988 ? Ssl 23:08 0:00
> > /usr/bin/cmds -i node0 -c /tmp/ceph.conf.3544
> > root 5119 0.2 0.7 198356 15756 ? Ssl 23:08 0:00
> > /usr/bin/cosd -i 1 -c /tmp/ceph.conf.3544
> > root 5157 0.0 0.0 0 0 ? S< 23:09 0:00 [ceph-msgr]
> > root 5166 0.0 0.0 3788 608 pts/0 S+ 23:09 0:00 mount
> > -t ceph 192.168.1.79:/ /ceph
> > root 5167 0.0 0.0 1804 240 pts/0 D+ 23:09 0:00
> > /sbin/mount.ceph 192.168.1.79 / /ceph -o rw
> > root 5168 0.0 0.0 0 0 ? S< 23:09 0:00
> > [ceph-writeback]
> > root 5169 0.0 0.0 0 0 ? S< 23:09 0:00
> > [ceph-pg-invalid]
> > root 5170 0.0 0.0 0 0 ? S< 23:09 0:00 [ceph-trunc]
> > root 5225 0.0 0.0 3372 732 pts/1 S+ 23:12 0:00 grep
> > --color=auto ceph
> >
> > ceph -s output is :
> > 2010-11-10 23:11:50.345617 b74296d0 -- :/5216 messenger.start
> > 2010-11-10 23:11:50.346006 b74296d0 -- :/5216 --> mon0
> > 192.168.1.79:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x9b4f828
> > 2010-11-10 23:11:50.346464 b6426b70 -- 192.168.1.79:0/5216 learned my
> > addr 192.168.1.79:0/5216
> > 2010-11-10 23:11:50.347320 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 1 ==== auth_reply(proto 1 0 Success) v1 ====
> > 24+0+0 (1462504351 0 0) 0x9b51ae0
> > 2010-11-10 23:11:50.347475 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- mon_subscribe({monmap=0+}) v1 -- ?+0 0x9b519c8
> > 2010-11-10 23:11:50.347529 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(0 v0) v1 -- ?+0 0x9b51c40
> > 2010-11-10 23:11:50.347582 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(1 v0) v1 -- ?+0 0x9b51df8
> > 2010-11-10 23:11:50.347675 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(2 v0) v1 -- ?+0 0x9b4f9f0
> > 2010-11-10 23:11:50.347734 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(3 v0) v1 -- ?+0 0x9b4fb10
> > 2010-11-10 23:11:50.347790 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(4 v0) v1 -- ?+0 0x9b4fc40
> > 2010-11-10 23:11:50.347840 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(5 v0) v1 -- ?+0 0x9b4fd70
> > 2010-11-10 23:11:50.347890 b7428b70 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(6 v0) v1 -- ?+0 0x9b4fea0
> > 2010-11-10 23:11:50.348445 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 2 ==== mon_map v1 ==== 187+0+0 (2881423474 0 0)
> > 0x9b51998
> > 2010-11-10 23:11:50.349157 b74296d0 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(0 v0) v1 -- ?+0 0x9b51a98
> > 2010-11-10 23:11:50.349407 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 3 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0
> > (20232142 0 0) 0x9b51998
> > 2010-11-10 23:11:50.349628 b74296d0 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(1 v0) v1 -- ?+0 0x9b51bb8
> > 2010-11-10 23:11:50.349708 b74296d0 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(2 v0) v1 -- ?+0 0x9b51df8
> > 2010-11-10 23:11:50.349760 b74296d0 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(3 v0) v1 -- ?+0 0x9b4f9f0
> > 2010-11-10 23:11:50.349812 b74296d0 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(4 v0) v1 -- ?+0 0x9b4fb10
> > 2010-11-10 23:11:50.349861 b74296d0 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(5 v0) v1 -- ?+0 0x9b4fc40
> > 2010-11-10 23:11:50.349912 b74296d0 -- 192.168.1.79:0/5216 --> mon0
> > 192.168.1.79:6789/0 -- observe(6 v0) v1 -- ?+0 0x9b51998
> > 2010-11-10 23:11:50.350574 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 4 ==== mon_observe_notify(v8 114646 bytes latest
> > v8) v1 ==== 114697+0+0 (1311084701 0 0) 0x9b4f9f0
> > 2010-11-10 23:11:50.350674 pg v8: 528 pgs: 528
> > active+clean+degraded; 11 KB data, 89846 MB used, 30508 MB / 123 GB
> > avail; 10/20 degraded (50.000%)
> > 2010-11-10 23:11:50.354159 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 5 ==== mon_observe_notify(v4 459 bytes latest v4)
> > v1 ==== 510+0+0 (3032970034 0 0) 0x9b7dcf8
> > 2010-11-10 23:11:50.354200 mds e4: 1/1/1 up {0=up:active}
> > 2010-11-10 23:11:50.354250 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 6 ==== mon_observe_notify(v3 1638 bytes latest v3)
> > v1 ==== 1689+0+0 (4040025353 0 0) 0x9b7e568
> > 2010-11-10 23:11:50.354281 osd e3: 1 osds: 1 up, 1 in
> > 2010-11-10 23:11:50.354375 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 10 ==== mon_observe_notify(v5 1002 bytes latest
> > v5) v1 ==== 1053+0+0 (180097538 0 0) 0x9b7f080
> > 2010-11-10 23:11:50.354409 log 2010-11-10 23:09:06.232712 mon0
> > 192.168.1.79:6789/0 4 : [INF] mds0 192.168.1.79:6800/4588 up:active
> > 2010-11-10 23:11:50.354469 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 12 ==== mon_observe_notify(v1 13 bytes latest v1)
> > v1 ==== 64+0+0 (3469238695 0 0) 0x9b7f408
> > 2010-11-10 23:11:50.354508 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 13 ==== mon_observe_notify(v1 183 bytes latest v1)
> > v1 ==== 234+0+0 (1598745335 0 0) 0x9b7f6a0
> > 2010-11-10 23:11:50.354540 mon e1: 1 mons at {0=192.168.1.79:6789/0}
> > 2010-11-10 23:11:50.354573 b7428b70 -- 192.168.1.79:0/5216 <== mon0
> > 192.168.1.79:6789/0 14 ==== mon_observe_notify(v3 1025 bytes latest
> > v3) v1 ==== 1076+0+0 (4171918925 0 0) 0x9b7fc80
> > 2010-11-10 23:11:50.355243 b74296d0 -- 192.168.1.79:0/5216 shutdown complete.
> >
> > the new ceph.conf is attached.
> > Thanks,
> > mugunthan
> >
> >
> > On Wed, Nov 10, 2010 at 7:25 PM, Wido den Hollander <wido@xxxxxxxxx> wrote:
> > > Hi,
> > >
> > > It seems you OSD is stalling.
> > >
> > > When you do "ps aux", do you see any processes in the "D" state (beside
> > > cosd)
> > >
> > > And what does "ceph -s" show?
> > >
> > > Btw, in your ceph.conf, use "host = <hostname>" instead of the
> > > IP-Address of the machine.
> > >
> > > Wido
> > >
> > > On Wed, 2010-11-10 at 18:45 +0530, srimugunthan dhandapani wrote:
> > >> Hi all,
> > >> I used v0.22.2 ceph with latest git "ceph-standalone-client kernel"
> > >> (2.6.36-rc8) and when i try to mount i get the following error
> > >> mount error 5 = Input/output error
> > >> the dmesg call trace is as follows:
> > >> [ 199.099511] ceph: loaded (mds proto 32)
> > >> [ 227.730778] libceph: client4099 fsid 2bf77910-49e4-89cb-8ef0-b74ce321acc4
> > >> [ 227.730999] libceph: mon0 192.168.1.79:6789 session established
> > >> [ 360.236062] INFO: task cosd:5004 blocked for more than 120 seconds.
> > >> [ 360.236068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > >> disables this message.
> > >> [ 360.236073] cosd D f4f7df20 0 5004 1 0x00000000
> > >> [ 360.236080] f4f7df30 00200086 00000002 f4f7df20 f4f7dee8 c01673d0
> > >> ffffffff 00000000
> > >> [ 360.236092] c02383f0 f4f758d0 c085c940 f4f75b58 f4f75b5c 5adef48d
> > >> 00000035 c085c940
> > >> [ 360.236104] c085c940 c085c940 00000000 c085c940 5ade69ed 00000035
> > >> f4f758d0 00000001
> > >> [ 360.236115] Call Trace:
> > >> [ 360.236128] [<c01673d0>] ? autoremove_wake_function+0x20/0x50
> > >> [ 360.236135] [<c02383f0>] ? blkdev_writepage+0x0/0x20
> > >> [ 360.236143] [<c05a21e5>] rwsem_down_failed_common+0x95/0xf0
> > >> [ 360.236148] [<c05a2272>] rwsem_down_read_failed+0x12/0x20
> > >> [ 360.236153] [<c05a22c7>] call_rwsem_down_read_failed+0x7/0x10
> > >> [ 360.236157] [<c05a16ec>] ? down_read+0x1c/0x20
> > >> [ 360.236163] [<c020f51f>] iterate_supers+0x5f/0xc0
> > >> [ 360.236167] [<c0230400>] ? sync_one_sb+0x0/0x30
> > >> [ 360.236171] [<c0230459>] sys_sync+0x29/0x60
> > >> [ 360.236176] [<c0102fe3>] sysenter_do_call+0x12/0x28
> > >>
> > >> my ceph.conf is attached. i was able to mount with the same ceph.conf
> > >> file in ceph0.21.1 and kernel 2.6.35 and also with ceph0.21.3 and
> > >> 2.6.35 kernel.
> > >> I got the same error with v0.22.2 and kernel 2.6.35 . So i updated
> > >> the kernel but it didn't help.
> > >> In ceph code , in src/cmds.cc and src/cosd.cc i commented out the line
> > >> // g_conf.profiler_running = IsHeapProfilerRunning;
> > >> for the compilation to pass.
> > >>
> > >> the steps i used the following commands as root:
> > >>
> > >> mkcephfs -c /etc/ceph/ceph.conf --allhosts -v
> > >> /etc/init.d/ceph -a start
> > >> modprobe ceph
> > >> mount -t ceph 192.168.1.79:/ /ceph
> > >>
> > >> Kindly point out what i am missing.
> > >> thanks,
> > >> mugunthan
> > >
> > >
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
