> Von: Markus Stockhausen > Gesendet: Freitag, 5. September 2014 20:06 > An: Ethan Wilson; linux-raid@xxxxxxxxxxxxxxx > Betreff: AW: AW: RAID456 direct I/O write performance > > > Von: Ethan Wilson [ethan.wilson@xxxxxxxxxxxxx] > > Gesendet: Donnerstag, 4. September 2014 23:12 > > An: linux-raid@xxxxxxxxxxxxxxx > > Betreff: Re: AW: RAID456 direct I/O write performance > > > > On 04/09/2014 18:30, Markus Stockhausen wrote: > > > A perf record of the 1 writer test gives: > > > > > > 38.40% swapper [kernel.kallsyms] [k] default_idle > > > 13.14% md0_raid5 [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore > > > 13.05% swapper [kernel.kallsyms] [k] tick_nohz_idle_enter > > > 10.01% iot [raid456] [k] raid5_unplug > > > 9.06% swapper [kernel.kallsyms] [k] tick_nohz_idle_exit > > > 3.39% md0_raid5 [kernel.kallsyms] [k] __kernel_fpu_begin > > > 1.67% md0_raid5 [xor] [k] xor_sse_2_pf64 > > > 0.87% iot [kernel.kallsyms] [k] finish_task_switch > > > > > > I'm confused and clueless. Especially I cannot see where the > > > 10% overhead in the source of raid5_unplug might come > > > from? Any idea from someone with better insight? > > > > I am no kernel developer but I have read that the CPU time for serving > > interrupts is often accounted to the random process which has the bad > > luck to be running at the time the interrupt comes and steals the CPU. I > > read this for top, htop etc, which have probably a different accounting > > mechanism than perf, but maybe something similar happens here, because > > _raw_spin_unlock_irqrestore at 13% looks too absurd to me. > > In fact, probably as soon as the interrupts are re-enabled by > > _raw_spin_unlock_irqrestore, the CPU often goes serving one interrupt > > that was queued, and this is before the function > > _raw_spin_unlock_irqrestore exits, so the time is really accounted there > > and that's why it's so high. > > Thanks Ethan for the explanation. > > In between I found the reason for the slowness. RAID5 handles direct IO > write requests sequentially with each one restarting the raid5d() worker > thread (more lke sync IO). The raid1d() worker thread can batch those > requests. Maybe Neil can tell if that behaviour is desired. Hopefully right this time - correcting myself to not spread wrong information: In raid1 case kernel writes via calling raid1_unplug(). Block handling is done within the function inside the calling thread. In raid5 case kernel calls raid5_unplug() for writeout handling. Inside we just have waiting logic for the raid5d() thread completion. So the numbers seem to be reasonable. A funny side effect is: A slow sequential directio or sync writer to a md raid456 on fast devices can be sped up by starting a (independent) reader - even to non md devices. Maybe because of thread scheduling optimization. Markus
**************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. �ber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 ****************************************************************************
