Re: [EXTERNAL] Re: Instance filesystem corrupt

["Will.Boege" <Will.Boege@xxxxxxxxxx>]

Strangely enough, I’m also seeing similar user issues – a strangely high volume of corrupt instance boot disks.

 

At this point I’m attributing it to the fact that our Ceph cluster is patched 9 months ahead of our RedHat OSP Kilo environment.  However that’s a total guess at this point…..

 

From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of "Keynes_Lee@xxxxxxxxxxx" <Keynes_Lee@xxxxxxxxxxx>
Date: Wednesday, October 26, 2016 at 8:28 PM
To: "ahmedmostafadev@xxxxxxxxx" <ahmedmostafadev@xxxxxxxxx>, "dillaman@xxxxxxxxxx" <dillaman@xxxxxxxxxx>
Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
Subject: [EXTERNAL] Re: [ceph-users] Instance filesystem corrupt

 

 

Hum ~~~seems we have in common

We use

rbd snap create to make snapshot for instances volumes

rbd export and rbd export-diff command to make daily backup.

 

Now we got 29 instances and 33 volumes

 

 

	Keynes  Lee    李 俊 賢
	Direct:	+886-2-6612-1025
	Mobile:	+886-9-1882-3787
	Fax:	+886-2-6612-1991
	E-Mail:	keynes_lee@xxxxxxxxxxx

 

 

From: Ahmed Mostafa [mailto:ahmedmostafadev@xxxxxxxxx]
Sent: Wednesday, October 26, 2016 9:57 PM
To: dillaman@xxxxxxxxxx
Cc: Keynes Lee/WHQ/Wistron <Keynes_Lee@xxxxxxxxxxx>; ceph-users <ceph-users@xxxxxxxxxxxxxx>
Subject: Re: [ceph-users] Instance filesystem corrupt

 

Actually i have the same problem when starting an instance backed up by librbd

But this only happens when trying to start 60+ instance

 

But I decided that this is due to the fact that we are using old hardware that is not able to respond to high demand.

 

Could that be the same issue that you are facing ?

 

On Wednesday, 26 October 2016, Jason Dillaman <jdillama@xxxxxxxxxx> wrote:

I am not aware of any similar reports against librbd on Firefly. Do you use any configuration overrides? Does the filesystem corruption appears while the instances are running or only after a shutdown / restart of the instance?

 

On Wed, Oct 26, 2016 at 12:46 AM, <Keynes_Lee@xxxxxxxxxxx> wrote:

No , we are using Firefly (0.80.7).

As we are using HPE Helion OpenStack 2.1.5, and what the version is was embedded is Firefly.

 

An upgrade was planning, but should will not happen  soon.

 

 

 

 

 

From: Will.Boege [mailto:Will.Boege@xxxxxxxxxx]
Sent: Wednesday, October 26, 2016 12:03 PM
To: Keynes Lee/WHQ/Wistron <Keynes_Lee@xxxxxxxxxxx>; ceph-users@xxxxxxxxxxxxxx
Subject: Re: [EXTERNAL] [ceph-users] Instance filesystem corrupt

 

Just out of curiosity, did you recently upgrade to Jewel?

 

From: ceph-users <ceph-users-bounces@xxxxxxxxxxxxxx> on behalf of "Keynes_Lee@xxxxxxxxxxx" <Keynes_Lee@xxxxxxxxxxx>
Date: Tuesday, October 25, 2016 at 10:52 PM
To: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
Subject: [EXTERNAL] [ceph-users] Instance filesystem corrupt

 

We are using OpenStack + Ceph.

Recently we found a lot of filesystem corrupt incident on instances.

Some of them are correctable, fixed by fsck, but the others have no luck, just corrupt and can never start up again.

 

We found this issue on vary operation systems of instances. They are

Redhat4 / CentOS 7 / Windows 2012

 

Could someone please advise us some troubleshooting direction ?

 

 

	Keynes  Lee    李 俊 賢
	Direct:	+886-2-6612-1025
	Mobile:	+886-9-1882-3787
	Fax:	+886-2-6612-1991
	E-Mail:	keynes_lee@xxxxxxxxxxx

 

 

---------------------------------------------------------------------------------------------------------------------------------------------------------------

This email contains confidential or legally privileged information and is for the sole use of its intended recipient.

Any unauthorized review, use, copying or distribution of this email or the content of this email is strictly prohibited.

If you are not the intended recipient, you may reply to the sender and should delete this e-mail immediately.

---------------------------------------------------------------------------------------------------------------------------------------------------------------

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

--

Jason

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com