RBD stale on VM, and RBD cache enable problem

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I have two questions. My newly created cluster with xfs on all osd,
ubuntu precise, kernel 3.2.0-23-generic. Ceph 0.47.2-1precise

pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num
64 pgp_num 64 last_change 1228 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins
pg_num 64 pgp_num 64 last_change 1226 owner 0
pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 64
pgp_num 64 last_change 1232 owner 0
pool 3 '.rgw' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8
pgp_num 8 last_change 3878 owner 18446744073709551615

1. After i stop all daemons on 1 machine in my 3 node cluster with 3
replicas, rbd image operations on vm, staling. DD on this device in VM
freezing, and after ceph start on this machine everything goes online.
Is there any problem with my config ?? in this situation ceph should
go from another copies with reads, and writes into another osd in
replica chain, yes ??

Another test iozone on device, and it's stop after daemons stop on 1
machine, and after osd up, iozone go forward, how can i tune this to
work without freeze ??

2012-06-11 21:38:49.583133    pg v88173: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:38:50.582257    pg v88174: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
.....
2012-06-11 21:39:49.991893    pg v88197: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:39:50.992755    pg v88198: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:39:51.993533    pg v88199: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:39:52.994397    pg v88200: 200 pgs: 60 active+clean, 1
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)

After boot all osd on stoped machine:

2012-06-11 21:40:37.826619   osd e4162: 72 osds: 53 up, 72 in
2012-06-11 21:40:37.825706 mon.0 10.177.66.4:6790/0 348 : [INF] osd.24
10.177.66.6:6800/21597 boot
2012-06-11 21:40:38.825297    pg v88202: 200 pgs: 54 active+clean, 7
stale+active+clean, 139 active+degraded; 783 GB data, 1928 GB used,
18111 GB / 20040 GB avail; 78169/254952 degraded (30.660%)
2012-06-11 21:40:38.826517   osd e4163: 72 osds: 54 up, 72 in
2012-06-11 21:40:38.825250 mon.0 10.177.66.4:6790/0 349 : [INF] osd.25
10.177.66.6:6803/21712 boot
2012-06-11 21:40:38.825655 mon.0 10.177.66.4:6790/0 350 : [INF] osd.28
10.177.66.6:6812/26210 boot
2012-06-11 21:40:38.825907 mon.0 10.177.66.4:6790/0 351 : [INF] osd.29
10.177.66.6:6815/26327 boot
2012-06-11 21:40:39.826738    pg v88203: 200 pgs: 56 active+clean, 4
stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
2012-06-11 21:40:39.830098   osd e4164: 72 osds: 59 up, 72 in
2012-06-11 21:40:39.826570 mon.0 10.177.66.4:6790/0 352 : [INF] osd.26
10.177.66.6:6806/21835 boot
2012-06-11 21:40:39.826961 mon.0 10.177.66.4:6790/0 353 : [INF] osd.27
10.177.66.6:6809/21953 boot
2012-06-11 21:40:39.828147 mon.0 10.177.66.4:6790/0 354 : [INF] osd.30
10.177.66.6:6818/26511 boot
2012-06-11 21:40:39.828418 mon.0 10.177.66.4:6790/0 355 : [INF] osd.31
10.177.66.6:6821/26583 boot
2012-06-11 21:40:39.828935 mon.0 10.177.66.4:6790/0 356 : [INF] osd.33
10.177.66.6:6827/26859 boot
2012-06-11 21:40:39.829274 mon.0 10.177.66.4:6790/0 357 : [INF] osd.34
10.177.66.6:6830/26979 boot
2012-06-11 21:40:40.827935    pg v88204: 200 pgs: 56 active+clean, 4
stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
2012-06-11 21:40:40.830059   osd e4165: 72 osds: 62 up, 72 in
2012-06-11 21:40:40.827798 mon.0 10.177.66.4:6790/0 358 : [INF] osd.32
10.177.66.6:6824/26701 boot
2012-06-11 21:40:40.829043 mon.0 10.177.66.4:6790/0 359 : [INF] osd.35
10.177.66.6:6833/27165 boot
2012-06-11 21:40:40.829316 mon.0 10.177.66.4:6790/0 360 : [INF] osd.36
10.177.66.6:6836/27280 boot
2012-06-11 21:40:40.829602 mon.0 10.177.66.4:6790/0 361 : [INF] osd.37
10.177.66.6:6839/27397 boot
2012-06-11 21:40:41.828776    pg v88205: 200 pgs: 56 active+clean, 4
stale+active+clean, 3 peering, 137 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 76921/254952 degraded (30.171%)
2012-06-11 21:40:41.831823   osd e4166: 72 osds: 68 up, 72 in
2012-06-11 21:40:41.828713 mon.0 10.177.66.4:6790/0 362 : [INF] osd.38
10.177.66.6:6842/27513 boot
2012-06-11 21:40:41.829440 mon.0 10.177.66.4:6790/0 363 : [INF] osd.39
10.177.66.6:6845/27628 boot
2012-06-11 21:40:41.830226 mon.0 10.177.66.4:6790/0 364 : [INF] osd.40
10.177.66.6:6848/27835 boot
2012-06-11 21:40:41.830531 mon.0 10.177.66.4:6790/0 365 : [INF] osd.41
10.177.66.6:6851/27950 boot
2012-06-11 21:40:41.830778 mon.0 10.177.66.4:6790/0 366 : [INF] osd.42
10.177.66.6:6854/28065 boot
2012-06-11 21:40:41.831249 mon.0 10.177.66.4:6790/0 367 : [INF] osd.43
10.177.66.6:6857/28181 boot
2012-06-11 21:40:42.830440    pg v88206: 200 pgs: 57 active+clean, 4
stale+active+clean, 7 peering, 132 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 75543/254952 degraded (29.630%)
2012-06-11 21:40:42.833294   osd e4167: 72 osds: 72 up, 72 in
2012-06-11 21:40:42.831046 mon.0 10.177.66.4:6790/0 368 : [INF] osd.44
10.177.66.6:6860/28373 boot
2012-06-11 21:40:42.832004 mon.0 10.177.66.4:6790/0 369 : [INF] osd.45
10.177.66.6:6863/28489 boot
2012-06-11 21:40:42.832314 mon.0 10.177.66.4:6790/0 370 : [INF] osd.46
10.177.66.6:6866/28607 boot
2012-06-11 21:40:42.832545 mon.0 10.177.66.4:6790/0 371 : [INF] osd.47
10.177.66.6:6869/28731 boot
2012-06-11 21:40:43.830481    pg v88207: 200 pgs: 64 active+clean, 4
stale+active+clean, 7 peering, 125 active+degraded; 783 GB data, 1928
GB used, 18111 GB / 20040 GB avail; 72874/254952 degraded (28.583%)
2012-06-11 21:40:43.831113   osd e4168: 72 osds: 72 up, 72 in
2012-06-11 21:40:44.832521    pg v88208: 200 pgs: 79 active+clean, 1
stale+active+clean, 4 peering, 113 active+degraded, 3
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 66185/254952 degraded (25.960%)
2012-06-11 21:40:45.834077    pg v88209: 200 pgs: 104 active+clean, 1
stale+active+clean, 4 peering, 85 active+degraded, 6
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 50399/254952 degraded (19.768%)
2012-06-11 21:40:46.835367    pg v88210: 200 pgs: 125 active+clean, 1
stale+active+clean, 4 peering, 59 active+degraded, 11
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 38563/254952 degraded (15.126%)
2012-06-11 21:40:47.836516    pg v88211: 200 pgs: 158 active+clean, 1
stale+active+clean, 26 active+degraded, 15 active+recovering; 783 GB
data, 1928 GB used, 18111 GB / 20040 GB avail; 18542/254952 degraded
(7.273%)
2012-06-11 21:40:48.853560    pg v88212: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 1/254952 degraded (0.000%)
2012-06-11 21:40:49.868514    pg v88213: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 1/254952 degraded (0.000%)
2012-06-11 21:40:50.858244    pg v88214: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail; 1/254952 degraded (0.000%)
2012-06-11 21:40:51.845622    pg v88215: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:52.857823    pg v88216: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:53.858281    pg v88217: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:54.855602    pg v88218: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:55.857241    pg v88219: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:56.857631    pg v88220: 200 pgs: 184 active+clean, 16
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:57.858987    pg v88221: 200 pgs: 185 active+clean, 15
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:58.880252    pg v88222: 200 pgs: 185 active+clean, 15
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:40:59.861910    pg v88223: 200 pgs: 188 active+clean, 12
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:00.902582    pg v88224: 200 pgs: 191 active+clean, 9
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:01.907767    pg v88225: 200 pgs: 196 active+clean, 4
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:02.876377    pg v88226: 200 pgs: 199 active+clean, 1
active+recovering; 783 GB data, 1928 GB used, 18111 GB / 20040 GB
avail
2012-06-11 21:41:03.876929    pg v88227: 200 pgs: 200 active+clean;
783 GB data, 1928 GB used, 18111 GB / 20040 GB avail

<disk type="network" device="disk">
         <driver name="qemu" type="raw"/>
         <source protocol="rbd" name="rbd/foo4">
         </source>
         <target dev="vdf" bus="virtio"/>
</disk>

2. When i use rbd_cache=1, or true in my xml, for libvirt i get:

<disk type="network" device="disk">
         <driver name="qemu" type="raw"/>
         <source protocol="rbd" name="rbd/foo5:rbd_cache=1">
         </source>
         <target dev="vdf" bus="virtio"/>
</disk>

libvirtd.log
2012-06-11 18:50:36.992+0000: 1751: error :
qemuMonitorTextAddDrive:2820 : operation failed: open disk image file
failed

Libvirt version 0.9.8-2ubuntu17 with some additional patch set before
ceph 0.46 version appears. Qemu-kvm 1.0+noroms-0ubuntu13.

Do i need any other patch for libvirt ?? Without rbd_cache attaching is ok.

-- 
-----
Pozdrawiam

Sławek "sZiBis" Skowron
; global
[global]
        ; enable secure authentication

        auth supported = cephx
        keyring = /etc/ceph/$cluster.keyring
 
        rgw_cache_enabled = false ;rgw cache enabled
        rgw_cache_lru_size = 131072 ;num of entries in rgw cache
        rgw_thread_pool_size = 4096 
        rgw print continue = false ;enable if 100-Continue works
        rgw_enable_ops_log = true ;enable logging every rgw operation
        rgw socket path = /var/run/radosgw.sock
        debug rgw = 1

        rbd cache = true
        rbd cache max dirty = 0

        admin_socket = /var/run/ceph/$cluster-$name.asok

; radosgw client list

[client.radosgw.obs-10-177-66-4]
        host = obs-10-177-66-4
        keyring = /etc/ceph/client.radosgw.obs-10-177-66-4.bin
        log file = /var/log/radosgw/$name.log
        debug ms = 1

[client.radosgw.obs-10-177-66-6]
        host = obs-10-177-66-6
        keyring = /etc/ceph/client.radosgw.obs-10-177-66-6.bin
        log file = /var/log/radosgw/$name.log
        debug ms = 1

[client.radosgw.obs-10-177-66-8]
        host = obs-10-177-66-8
        keyring = /etc/ceph/client.radosgw.obs-10-177-66-8.bin
        log file = /var/log/radosgw/$name.log
        debug ms = 1


; monitors
;  You need at least one.  You need at least three if you want to
;  tolerate any node failures.  Always create an odd number.
[mon]
        mon data = /vol0/data/mon.$id
 
        ; some minimal logging (just message traffic) to aid debugging
 
        debug ms = 0     ; see message traffic
        debug mon = 0   ; monitor
        debug paxos = 0 ; monitor replication
        debug auth = 0  ;
 
        mon allowed clock drift = 2
 

[mon.0]
        host = obs-10-177-66-4
        mon addr = 10.177.66.4:6790


; osd
;  You need at least one.  Two if you want data to be replicated.
;  Define as many as you like.
[osd]
        ; This is where the btrfs volume will be mounted.

        osd data = /vol0/data/osd.$id
 
        ; Ideally, make this a separate disk or partition.  A few GB
        ; is usually enough; more if you have fast disks.  You can use
        ; a file under the osd data dir if need be
        ; (e.g. /data/osd$id/journal), but it will be slower than a
        ; separate disk or partition.

        osd journal = /vol0/data/osd.$id/journal
 
        ; If the OSD journal is a file, you need to specify the size. This is specified in MB.

        keyring = /vol0/data/osd.$id/keyring

        osd journal size = 1024
 
;       filestore_xattr_use_omap = 0
        filestore journal writeahead = 1
        osd heartbeat grace = 8
 
        debug ms = 0         ; message traffic
        debug osd = 1
        debug filestore = 1 ; local object storage
        debug journal = 0   ; local journaling
        debug monc = 0
        debug rados = 0
 

[osd.0]
        host = obs-10-177-66-4

[osd.1]
        host = obs-10-177-66-4

[osd.10]
        host = obs-10-177-66-4

[osd.11]
        host = obs-10-177-66-4

[osd.12]
        host = obs-10-177-66-4

[osd.13]
        host = obs-10-177-66-4

[osd.14]
        host = obs-10-177-66-4

[osd.15]
        host = obs-10-177-66-4

[osd.16]
        host = obs-10-177-66-4

[osd.17]
        host = obs-10-177-66-4

[osd.18]
        host = obs-10-177-66-4

[osd.19]
        host = obs-10-177-66-4

[osd.2]
        host = obs-10-177-66-4

[osd.20]
        host = obs-10-177-66-4

[osd.21]
        host = obs-10-177-66-4

[osd.22]
        host = obs-10-177-66-4

[osd.23]
        host = obs-10-177-66-4

[osd.24]
        host = obs-10-177-66-6

[osd.25]
        host = obs-10-177-66-6

[osd.26]
        host = obs-10-177-66-6

[osd.27]
        host = obs-10-177-66-6

[osd.28]
        host = obs-10-177-66-6

[osd.29]
        host = obs-10-177-66-6

[osd.3]
        host = obs-10-177-66-4

[osd.30]
        host = obs-10-177-66-6

[osd.31]
        host = obs-10-177-66-6

[osd.32]
        host = obs-10-177-66-6

[osd.33]
        host = obs-10-177-66-6

[osd.34]
        host = obs-10-177-66-6

[osd.35]
        host = obs-10-177-66-6

[osd.36]
        host = obs-10-177-66-6

[osd.37]
        host = obs-10-177-66-6

[osd.38]
        host = obs-10-177-66-6

[osd.39]
        host = obs-10-177-66-6

[osd.4]
        host = obs-10-177-66-4

[osd.40]
        host = obs-10-177-66-6

[osd.41]
        host = obs-10-177-66-6

[osd.42]
        host = obs-10-177-66-6

[osd.43]
        host = obs-10-177-66-6

[osd.44]
        host = obs-10-177-66-6

[osd.45]
        host = obs-10-177-66-6

[osd.46]
        host = obs-10-177-66-6

[osd.47]
        host = obs-10-177-66-6

[osd.48]
        host = obs-10-177-66-8

[osd.49]
        host = obs-10-177-66-8

[osd.5]
        host = obs-10-177-66-4

[osd.50]
        host = obs-10-177-66-8

[osd.51]
        host = obs-10-177-66-8

[osd.52]
        host = obs-10-177-66-8

[osd.53]
        host = obs-10-177-66-8

[osd.54]
        host = obs-10-177-66-8

[osd.55]
        host = obs-10-177-66-8

[osd.56]
        host = obs-10-177-66-8

[osd.57]
        host = obs-10-177-66-8

[osd.58]
        host = obs-10-177-66-8

[osd.59]
        host = obs-10-177-66-8

[osd.6]
        host = obs-10-177-66-4

[osd.60]
        host = obs-10-177-66-8

[osd.61]
        host = obs-10-177-66-8

[osd.62]
        host = obs-10-177-66-8

[osd.63]
        host = obs-10-177-66-8

[osd.64]
        host = obs-10-177-66-8

[osd.65]
        host = obs-10-177-66-8

[osd.66]
        host = obs-10-177-66-8

[osd.67]
        host = obs-10-177-66-8

[osd.68]
        host = obs-10-177-66-8

[osd.69]
        host = obs-10-177-66-8

[osd.7]
        host = obs-10-177-66-4

[osd.70]
        host = obs-10-177-66-8

[osd.71]
        host = obs-10-177-66-8

[osd.8]
        host = obs-10-177-66-4

[osd.9]
        host = obs-10-177-66-4

[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux