you need to use a virtual desktop... Xquartz X11 does not have RANDR. Wine apps will only see 1 resolution, the one your running when it checks. If you run in a virtual desktop, that problem wont exist, as the window can resize and change all it wants... If you use Wineskin, and run it in fullscreen mode, you should have several options it detects. If the resolution you want to run isn't detected, you can set a resolution in WineskinSettings.app and it will be available in the program, and run fullscreen and change the resolution for you if its different than normal. Wineskin has its own version of Xquartz built in, so your X11.app doesn't really matter. Speaking of RANDR in Xquartz, I'm playing with some code I'm tying to get to work to add that, written by someone else other than me, and submitted to Xquartz, so it may be available in the near future... I can't get it running perfectly (yet), but I'm going to work on maybe cleaning it up to work good in Wineskin, even if its not great for all X11 apps.
