On Tue, May 19, 2020 at 03:12:42PM -0700, Dan Williams wrote: > The original copy_mc_fragile() implementation had negative performance > implications since it did not use the fast-string instruction sequence > to perform copies. For this reason copy_mc_to_kernel() fell back to > plain memcpy() to preserve performance on platform that did not indicate > the capability to recover from machine check exceptions. However, that > capability detection was not architectural and now that some platforms > can recover from fast-string consumption of memory errors the memcpy() > fallback now causes these more capable platforms to fail. > > Introduce copy_mc_generic() as the fast default implementation of > copy_mc_to_kernel() and finalize the transition of copy_mc_fragile() to > be a platform quirk to indicate 'fragility'. With this in place > copy_mc_to_kernel() is fast and recovery-ready by default regardless of > hardware capability. > > Thanks to Vivek for identifying that copy_user_generic() is not suitable > as the copy_mc_to_user() backend since the #MC handler explicitly checks > ex_has_fault_handler(). /me is curious to know why #MC handler mandates use of _ASM_EXTABLE_FAULT(). [..] > +/* > + * copy_mc_generic - memory copy with exception handling > + * > + * Fast string copy + fault / exception handling. If the CPU does > + * support machine check exception recovery, but does not support > + * recovering from fast-string exceptions then this CPU needs to be > + * added to the copy_mc_fragile_key set of quirks. Otherwise, absent any > + * machine check recovery support this version should be no slower than > + * standard memcpy. > + */ > +SYM_FUNC_START(copy_mc_generic) > + ALTERNATIVE "jmp copy_mc_fragile", "", X86_FEATURE_ERMS > + movq %rdi, %rax > + movq %rdx, %rcx > +.L_copy: > + rep movsb > + /* Copy successful. Return zero */ > + xorl %eax, %eax > + ret > +SYM_FUNC_END(copy_mc_generic) > +EXPORT_SYMBOL_GPL(copy_mc_generic) > + > + .section .fixup, "ax" > +.E_copy: > + /* > + * On fault %rcx is updated such that the copy instruction could > + * optionally be restarted at the fault position, i.e. it > + * contains 'bytes remaining'. A non-zero return indicates error > + * to copy_safe() users, or indicate short transfers to copy_safe() is vestige of terminology of previous patches? > + * user-copy routines. > + */ > + movq %rcx, %rax > + ret > + > + .previous > + > + _ASM_EXTABLE_FAULT(.L_copy, .E_copy) A question for my education purposes. So copy_mc_generic() can handle MCE both on source and destination addresses? (Assuming some device can generate MCE on stores too). On the other hand copy_mc_fragile() handles MCE recovery only on source and non-MCE recovery on destination. Thanks Vivek
