Yes this is a problem, unfortunately I missed checking some of the simple-card examples. I wonder we should be specifically looking for "mclk" clock here.That would definitely help mitigate the problem but I really think it's cleaner and safer to just push this down to set_sysclk().
Understand now. I will push patches based on this. Thanks for the details.