Ok, i got it.
Having a stable network will save the system from a node crash? What happens if a osd goes down? Will the clients suffer from freeze mounts and things like that?
Just asking dummy questions to see if i'm the right path, since AFAIK ceph is meant to be a high available/fault tolerant storage system, right?
---
Diego Castro / The CloudFather
GetupCloud.com - Eliminamos a Gravidade
GetupCloud.com - Eliminamos a Gravidade
2016-04-01 13:23 GMT-03:00 Oliver Dzombic <info@xxxxxxxxxxxxxxxxx>:
Hi Diego,
you can see the network connection as your HDD cables.
So if you get interruptions there, its like you are pulling out the HDD
cables of your server/computer and putting it back.
You can just check easily how much your server/computer will like that
with your local HDD's ;-)
----
And no, ceph will not protect you from this.
If the requested data is on a PG / OSD which will receive a network
interrupt, you will get IO Errors.
The question is what the OS of the VM will do with that. Maybe it will
mount the whole HDD read-only.
Maybe it will just throw some errors until its good again.
Maybe you will have a stale/freeze until its good again.
Maybe .....
In any case, a stable network connection is the absolute basic
requirement for network storage. If your cloud environment cant provide
that, you cant provide stable services.
--
Mit freundlichen Gruessen / Best regards
Oliver Dzombic
IP-Interactive
mailto:info@xxxxxxxxxxxxxxxxx
Anschrift:
IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen
HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic
Steuer Nr.: 35 236 3622 1
UST ID: DE274086107
Am 01.04.2016 um 17:31 schrieb Diego Castro:
> Hello Oliver, sorry if i wasn't clear at my first post.
> I agree with you that a network issue isn't desirable but should it
> crash mount clients? I mean, doesn't the client be smart enough to retry
> connection or so?
> My point is cloud environments (public) doesn't have the same
> availability as a local setup, so shouldn't we at least don't freeze the
> clients?
>
>
> ---
> Diego Castro / The CloudFather
> GetupCloud.com - Eliminamos a Gravidade
>
> 2016-04-01 12:27 GMT-03:00 Oliver Dzombic <info@xxxxxxxxxxxxxxxxx
> <mailto:info@xxxxxxxxxxxxxxxxx>>:
>
> Hi Diego,
>
> ok so this is a new case scenario.
>
> Before you said its "until i put some load on it".
>
> Now you say, you can't reproduce it and mention that it happends during
> a (known) network maintenance.
>
> So i agree with you, we can assume that your problems were based on
> network issues.
>
> Thats also was your logs implies:
>
> "failed lossy con, dropping message"
>
> --
> Mit freundlichen Gruessen / Best regards
>
> Oliver Dzombic
> IP-Interactive
>
> mailto:info@xxxxxxxxxxxxxxxxx <mailto:info@xxxxxxxxxxxxxxxxx>
> > <mailto:info@xxxxxxxxxxxxxxxxx <mailto:info@xxxxxxxxxxxxxxxxx>>>:>
> Anschrift:
>
> IP Interactive UG ( haftungsbeschraenkt )
> Zum Sonnenberg 1-3
> 63571 Gelnhausen
>
> HRB 93402 beim Amtsgericht Hanau
> Geschäftsführung: Oliver Dzombic
>
> Steuer Nr.: 35 236 3622 1
> UST ID: DE274086107
>
>
> Am 01.04.2016 um 14:07 schrieb Diego Castro:
> > Hello Oliver, this issue showed very hard to reproduce, i couldn't make
> > it again.
> > My best guess is something with the Azure's network since last week
> > (when happened a lot) there were a ongoing maintenance.
> >
> > Here's the outputs:
> >
> > $ ceph -s
> > cluster 25736883-dbf1-4d7a-8796-50e36f9de7a6
> > health HEALTH_OK
> > monmap e1: 4 mons at
> > {osmbr0=10.0.3.4:6789/0,osmbr1=10.0.3.6:6789/0,osmbr2=10.0.3.14:6789/0,osmbr3=10.0.3.7:6789/0
> <http://10.0.3.4:6789/0,osmbr1=10.0.3.6:6789/0,osmbr2=10.0.3.14:6789/0,osmbr3=10.0.3.7:6789/0>
> >
> <http://10.0.3.4:6789/0,osmbr1=10.0.3.6:6789/0,osmbr2=10.0.3.14:6789/0,osmbr3=10.0.3.7:6789/0>}
> > election epoch 602, quorum 0,1,2,3
> osmbr0,osmbr1,osmbr3,osmbr2
> > osdmap e1816: 10 osds: 10 up, 10 in
> > pgmap v3158931: 128 pgs, 1 pools, 11512 MB data, 3522 objects
> > 34959 MB used, 10195 GB / 10229 GB avail
> > 128 active+clean
> > client io 87723 B/s wr, 8 op/s
> >
> > $ ceph osd df
> > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR
> > 6 1.00000 1.00000 1022G 3224M 1019G 0.31 0.92
> > 1 1.00000 1.00000 1022G 3489M 1019G 0.33 1.00
> > 2 1.00000 1.00000 1022G 3945M 1019G 0.38 1.13
> > 4 1.00000 1.00000 1022G 3304M 1019G 0.32 0.95
> > 7 1.00000 1.00000 1022G 3427M 1019G 0.33 0.98
> > 3 1.00000 1.00000 1022G 4361M 1018G 0.42 1.25
> > 9 1.00000 1.00000 1022G 3650M 1019G 0.35 1.04
> > 0 1.00000 1.00000 1022G 3210M 1019G 0.31 0.92
> > 5 1.00000 1.00000 1022G 3577M 1019G 0.34 1.02
> > 8 1.00000 1.00000 1022G 2765M 1020G 0.26 0.79
> > TOTAL 10229G 34957M 10195G 0.33
> > MIN/MAX VAR: 0.79/1.25 STDDEV: 0.04
> >
> >
> >
> > $ ceph osd perf
> > osd fs_commit_latency(ms) fs_apply_latency(ms)
> > 0 1 2
> > 1 1 2
> > 2 2 3
> > 3 2 3
> > 4 1 2
> > 5 2 3
> > 6 1 2
> > 7 2 3
> > 8 1 2
> > 9 1 1
> >
> >
> >
> >
> >
> >
> >
> > ---
> > Diego Castro / The CloudFather
> > GetupCloud.com - Eliminamos a Gravidade
> >
> > 2016-03-31 18:00 GMT-03:00 Oliver Dzombic <info@xxxxxxxxxxxxxxxxx
> <mailto:info@xxxxxxxxxxxxxxxxx>
> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>> >
> > Hi Diego,
> >
> > lets start with the basics and please give us the output of
> >
> > ceph -s
> > ceph osd df
> > ceph osd perf
> >
> > at best before and after you provike the iowait.
> >
> > Thank you !
> >
> > --
> > Mit freundlichen Gruessen / Best regards
> >
> > Oliver Dzombic
> > IP-Interactive
> >
> > mailto:info@xxxxxxxxxxxxxxxxx <mailto:info@xxxxxxxxxxxxxxxxx>
> <mailto:info@xxxxxxxxxxxxxxxxx <mailto:info@xxxxxxxxxxxxxxxxx>>
> >
> > Anschrift:
> >
> > IP Interactive UG ( haftungsbeschraenkt )
> > Zum Sonnenberg 1-3
> > 63571 Gelnhausen
> >
> > HRB 93402 beim Amtsgericht Hanau
> > Geschäftsführung: Oliver Dzombic
> >
> > Steuer Nr.: 35 236 3622 1
> > UST ID: DE274086107
> >
> >
> > Am 31.03.2016 um 21:38 schrieb Diego Castro:
> > > Hello, everyone.
> > > I have a pretty basic ceph setup running on top of Azure
> Cloud, (4 mons
> > > and 10 osd's) for rbd images.
> > > Everything seems to be working as expected until i put some
> load on it,
> > > sometimes it doesn't complete the process (mysql restore for
> ex.) and
> > > sometimes it does without any issues.
> > >
> > >
> > > Client Kernel: 3.10.0-327.10.1.el7.x86_64
> > > OSD Kernel: 3.10.0-229.7.2.el7.x86_64
> > >
> > > Ceph: ceph-0.94.5-0.el7.x86_64
> > >
> > > On the client side, i have 100%iowait, a lot of "INFO: task
> blocked for
> > > more than 120 seconds"
> > > On the osd side, i have no evidences of faulty disk or
> read/write
> > > latency, but i found the following messages:
> > >
> > >
> > > 2016-03-28 17:04:03.425249 7f7329fc5700 0 bad crc in data
> 641367213 !=
> > > exp 3107019767
> > > 2016-03-28 17:04:03.440599 7f7329fc5700 0 --
> 10.0.3.9:6800/2272 <http://10.0.3.9:6800/2272>
> <http://10.0.3.9:6800/2272>
> > > <http://10.0.3.9:6800/2272> >> 10.0.2.5:0/1998047321
> <http://10.0.2.5:0/1998047321>
> > <http://10.0.2.5:0/1998047321>
> > > <http://10.0.2.5:0/1998047321> pipe(0x13cc4800 sd=54 :6800
> s=0 pgs=0
> > > cs=0 l=0 c=0x13883f40).accept peer addr is really
> 10.0.2.5:0/1998047321 <http://10.0.2.5:0/1998047321>
> <http://10.0.2.5:0/1998047321>
> > > <http://10.0.2.5:0/1998047321> (socket is 10.0.2.5:34702/0
> <http://10.0.2.5:34702/0>
> > <http://10.0.2.5:34702/0>
> > > <http://10.0.2.5:34702/0>)
> > > 2016-03-28 17:04:03.487497 7f7333e6a700 0 --
> 10.0.3.9:6800/2272 <http://10.0.3.9:6800/2272>
> <http://10.0.3.9:6800/2272>
> > > <http://10.0.3.9:6800/2272> submit_message osd_op_reply(20046
> > > rb.0.6040.238e1f29.000000000074 [set-alloc-hint object_size
> 4194304
> > > write_size 4194304,write 0~524288] v1753'32512 uv32512
> _ondisk_ = 0) v6
> > > remote, 10.0.2.5:0/1998047321 <http://10.0.2.5:0/1998047321>
> <http://10.0.2.5:0/1998047321>
> > <http://10.0.2.5:0/1998047321>, failed
> > > lossy con, dropping message 0x12b539c0
> > > 2016-03-28 17:04:03.532302 7f733666f700 0 --
> 10.0.3.9:6800/2272 <http://10.0.3.9:6800/2272>
> <http://10.0.3.9:6800/2272>
> > > <http://10.0.3.9:6800/2272> submit_message osd_op_reply(20047
> > > rb.0.6040.238e1f29.000000000074 [set-alloc-hint object_size
> 4194304
> > > write_size 4194304,write 524288~524288] v1753'32513 uv32513
> _ondisk_ = 0)
> > > v6 remote, 10.0.2.5:0/1998047321
> <http://10.0.2.5:0/1998047321> <http://10.0.2.5:0/1998047321>
> > <http://10.0.2.5:0/1998047321>, failed
> > > lossy con, dropping message 0x1667bc80
> > > 2016-03-28 17:04:03.535143 7f7333e6a700 0 --
> 10.0.3.9:6800/2272 <http://10.0.3.9:6800/2272>
> <http://10.0.3.9:6800/2272>
> > > <http://10.0.3.9:6800/2272> submit_message osd_op_reply(20048
> > > rb.0.6040.238e1f29.000000000074 [set-alloc-hint object_size
> 4194304
> > > write_size 4194304,write 1048576~524288] v1753'32514 uv32514
> _ondisk_ = 0)
> > > v6 remote, 10.0.2.5:0/1998047321
> <http://10.0.2.5:0/1998047321> <http://10.0.2.5:0/1998047321>
> > <http://10.0.2.5:0/1998047321>, failed
> > > lossy con, dropping message 0x12b56e00
> > >
> > > ---
> > > Diego Castro / The CloudFather
> > > GetupCloud.com - Eliminamos a Gravidade
> > >
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> <mailto:ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>>
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
