НА: inconsistent pgs

[Межов Игорь Александрович <megov@xxxxxxxxxx>]

Hi!

>If there is cable or SATA-controller issues, will it be shown in /var/log/dmesg?

If it will lead to read errors, it will be logged in dmesg. If it cause only
SATA command retransmission, it maybe won't logged in dmesg,
but have to be shown in SMART attributes.

And anyway, we face only some (<10) inconsistences during a year
in production usage, not so large amount, as you have. So maybe
it is not a disk error issue.

Also, there are two strange things I can see:
- do you need SELinux enabled? AFAIK, SELinux security attributes are
stored in extended attrs (xatts), and Ceph also use xattrs in intensive
manner. Maybe some problems between them? Try to disable SELinux
at boot - I saw this suggestion quite often, when do trobleshooting.
- As I cen see, you use XFS on your OSDs? But in dmesg we see
a records about the same /dev/sdc:

>BTRFS: device fsid a3176b6c-2e2b-4121-9dc3-f191d469eccc devid 1
>transid 287115 /dev/sdc

and

>XFS (sdc1): Mounting V4 Filesystem

>XFS (sdc1): Ending clean mount

Is it btrfs or XFS? I, personally, dont have much success with btrfs.
Some of my test usages ends with completely hang machines.

Megov Igor
CIO, Yuterr

block0 dmesg:

[    5.296302] XFS (sdb1): Mounting V4 Filesystem

[    5.316487] XFS (sda1): Mounting V4 Filesystem

[    5.464439] XFS (sdb1): Ending clean mount

[    5.464466] SELinux: initialized (dev sdb1, type xfs), uses xattr

[    5.506185] XFS (sda1): Ending clean mount

[    5.506214] SELinux: initialized (dev sda1, type xfs), uses xattr

[    5.649101] XFS (sdc1): Ending clean mount

[    5.649129] SELinux: initialized (dev sdc1, type xfs), uses xattr

[    5.668062] systemd-journald[500]: Received request to flush runtime journal from PID 1

[    5.786559] type=1305 audit(1438754525.356:4): audit_pid=629 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1

[    5.870809] snd_hda_intel 0000:00:1b.0: Codec #2 probe error; disabling it...

[    5.976594] sound hdaudioC0D0: autoconfig: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line

[    5.976604] sound hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)

[    5.976609] sound hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)

[    5.976613] sound hdaudioC0D0:    mono: mono_out=0x0

[    5.976617] sound hdaudioC0D0:    dig-out=0x11/0x0

[    5.976621] sound hdaudioC0D0:    inputs:

[    5.976627] sound hdaudioC0D0:      Rear Mic=0x18

[    5.976632] sound hdaudioC0D0:      Front Mic=0x19

[    5.976637] sound hdaudioC0D0:      Line=0x1a

[    5.976641] sound hdaudioC0D0:      CD=0x1c

block2 dmesg:

[    4.987126] XFS (sda1): Mounting V4 Filesystem

[    4.990737] raid6: sse2x1     601 MB/s

[    5.007765] raid6: sse2x2     632 MB/s

[    5.024711] raid6: sse2x4    1316 MB/s

[    5.024717] raid6: using algorithm sse2x4 (1316 MB/s)

[    5.024720] raid6: using ssse3x2 recovery algorithm

[    5.046099] XFS (sda1): Ending clean mount

[    5.046126] SELinux: initialized (dev sda1, type xfs), uses xattr

[    5.074257] Btrfs loaded

[    5.075283] BTRFS: device fsid a3176b6c-2e2b-4121-9dc3-f191d469eccc devid 1 transid 287115 /dev/sdc

[    5.098721] Adding 4882428k swap on /dev/mapper/root-swap00.  Priority:-1 extents:1 across:4882428k SSFS

[    5.157349] XFS (sdc1): Mounting V4 Filesystem

[    5.222992] XFS (sdb1): Mounting V4 Filesystem

[    5.362687] XFS (sdc1): Ending clean mount

[    5.362738] SELinux: initialized (dev sdc1, type xfs), uses xattr

[    5.417178] XFS (sdb1): Ending clean mount

[    5.417208] SELinux: initialized (dev sdb1, type xfs), uses xattr

[    5.429280] systemd-journald[497]: Received request to flush runtime journal from PID 1

[    5.533922] type=1305 audit(1438754526.111:4): audit_pid=626 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1

пт, 7 авг. 2015 г. в 14:08, Межов Игорь Александрович <megov@xxxxxxxxxx>:

Hi!

Do you have any disk errors in dmesg output? In our practice, every time the deep
scrub found inconsistent PG, we also found a disk error, that was the reason.
Sometimes it was media errors (bad sectors), one time - bad sata cable and
we also had some raid/hba firmware issues. But in all cases, when we see inconsistent
PG - we also see disk errors. So, please, check them at your cluster firstly.

Megov Igor
CIO, Yuterra

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com