"you use XFS on your OSDs?"
This OSD was formatted in BTRFS as a whole block device /dev/sdc (with no partition table). Then I moved from BTRFS to XFS /dev/sdc1 (with partition table), because BTRFS was v-v-very slow. Maybe partprober sees some old signatures from first sectors of that disk...
By "move" I mean removing OSD from cluster as described in official docs http://ceph.com/docs/master/rados/operations/add-or-rm-osds/, and then adding it as new OSD.
пт, 7 авг. 2015 г. в 14:46, Межов Игорь Александрович <megov@xxxxxxxxxx>:
Hi!
>If there is cable or SATA-controller issues, will it be shown in /var/log/dmesg?
If it will lead to read errors, it will be logged in dmesg. If it cause only
SATA command retransmission, it maybe won't logged in dmesg,
but have to be shown in SMART attributes.
And anyway, we face only some (<10) inconsistences during a year
in production usage, not so large amount, as you have. So maybe
it is not a disk error issue.
Also, there are two strange things I can see:
- do you need SELinux enabled? AFAIK, SELinux security attributes are
stored in extended attrs (xatts), and Ceph also use xattrs in intensive
manner. Maybe some problems between them? Try to disable SELinux
at boot - I saw this suggestion quite often, when do trobleshooting.
- As I cen see, you use XFS on your OSDs? But in dmesg we see
a records about the same /dev/sdc:
>BTRFS: device fsid a3176b6c-2e2b-4121-9dc3-f191d469eccc devid 1
>transid 287115 /dev/sdc
and
>XFS (sdc1): Mounting V4 Filesystem>XFS (sdc1): Ending clean mount
Is it btrfs or XFS? I, personally, dont have much success with btrfs.
Some of my test usages ends with completely hang machines.
Megov Igor
CIO, Yuterr
block0 dmesg:[ 5.296302] XFS (sdb1): Mounting V4 Filesystem[ 5.316487] XFS (sda1): Mounting V4 Filesystem[ 5.464439] XFS (sdb1): Ending clean mount[ 5.464466] SELinux: initialized (dev sdb1, type xfs), uses xattr[ 5.506185] XFS (sda1): Ending clean mount[ 5.506214] SELinux: initialized (dev sda1, type xfs), uses xattr[ 5.649101] XFS (sdc1): Ending clean mount[ 5.649129] SELinux: initialized (dev sdc1, type xfs), uses xattr[ 5.668062] systemd-journald[500]: Received request to flush runtime journal from PID 1[ 5.786559] type=1305 audit(1438754525.356:4): audit_pid=629 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1[ 5.870809] snd_hda_intel 0000:00:1b.0: Codec #2 probe error; disabling it...[ 5.976594] sound hdaudioC0D0: autoconfig: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line[ 5.976604] sound hdaudioC0D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)[ 5.976609] sound hdaudioC0D0: hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)[ 5.976613] sound hdaudioC0D0: mono: mono_out=0x0[ 5.976617] sound hdaudioC0D0: dig-out=0x11/0x0[ 5.976621] sound hdaudioC0D0: inputs:[ 5.976627] sound hdaudioC0D0: Rear Mic=0x18[ 5.976632] sound hdaudioC0D0: Front Mic=0x19[ 5.976637] sound hdaudioC0D0: Line=0x1a[ 5.976641] sound hdaudioC0D0: CD=0x1c
block2 dmesg:[ 4.987126] XFS (sda1): Mounting V4 Filesystem[ 4.990737] raid6: sse2x1 601 MB/s[ 5.007765] raid6: sse2x2 632 MB/s[ 5.024711] raid6: sse2x4 1316 MB/s[ 5.024717] raid6: using algorithm sse2x4 (1316 MB/s)[ 5.024720] raid6: using ssse3x2 recovery algorithm[ 5.046099] XFS (sda1): Ending clean mount[ 5.046126] SELinux: initialized (dev sda1, type xfs), uses xattr[ 5.074257] Btrfs loaded[ 5.075283] BTRFS: device fsid a3176b6c-2e2b-4121-9dc3-f191d469eccc devid 1 transid 287115 /dev/sdc[ 5.098721] Adding 4882428k swap on /dev/mapper/root-swap00. Priority:-1 extents:1 across:4882428k SSFS[ 5.157349] XFS (sdc1): Mounting V4 Filesystem[ 5.222992] XFS (sdb1): Mounting V4 Filesystem[ 5.362687] XFS (sdc1): Ending clean mount[ 5.362738] SELinux: initialized (dev sdc1, type xfs), uses xattr[ 5.417178] XFS (sdb1): Ending clean mount[ 5.417208] SELinux: initialized (dev sdb1, type xfs), uses xattr[ 5.429280] systemd-journald[497]: Received request to flush runtime journal from PID 1[ 5.533922] type=1305 audit(1438754526.111:4): audit_pid=626 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1
пт, 7 авг. 2015 г. в 14:08, Межов Игорь Александрович <megov@xxxxxxxxxx>:
Hi!
Do you have any disk errors in dmesg output? In our practice, every time the deep
scrub found inconsistent PG, we also found a disk error, that was the reason.
Sometimes it was media errors (bad sectors), one time - bad sata cable and
we also had some raid/hba firmware issues. But in all cases, when we see inconsistent
PG - we also see disk errors. So, please, check them at your cluster firstly.
Megov Igor
CIO, Yuterra
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
