So at some point it was apparently decided to arbitrarily pick an encoding. The encoding used was SC_SIZE + wParam - 2. However, Visual FoxPro handles the generation of WM_SYSCOMMAND messages itself, and uses a different value. Testing with SPY++ under WinNT shows that when WinNT sends the message, it encodes wParam as SC_SIZE+wParam-(HTLEFT-WMSZ_LEFT). And this matches what Visual FoxPro generates, so that looks like the correct value to use.
Changelog:
Correctly encode wParam for WM_SYSCOMMAND messages.
Index: dlls/x11drv/winpos.c =================================================================== RCS file: /home/wine/wine/dlls/x11drv/winpos.c,v retrieving revision 1.63 diff -u -r1.63 winpos.c --- dlls/x11drv/winpos.c 8 Jan 2003 21:09:26 -0000 1.63 +++ dlls/x11drv/winpos.c 8 Feb 2003 22:44:11 -0000 @@ -1901,7 +1903,8 @@ else /* SC_SIZE */ { if (!thickframe) return; - if ( hittest && hittest != HTSYSMENU ) hittest += 2; + if ( hittest && ((wParam & 0xfff0) != SC_MOUSEMENU) ) + hittest += (HTLEFT - WMSZ_LEFT); else { set_movesize_capture( hwnd ); Index: windows/nonclient.c =================================================================== RCS file: /home/wine/wine/windows/nonclient.c,v retrieving revision 1.109 diff -u -r1.109 nonclient.c --- windows/nonclient.c 14 Jan 2003 23:41:01 -0000 1.109 +++ windows/nonclient.c 8 Feb 2003 22:44:12 -0000 @@ -2071,8 +2071,15 @@ case HTBOTTOM: case HTBOTTOMLEFT: case HTBOTTOMRIGHT: - /* make sure hittest fits into 0xf and doesn't overlap with HTSYSMENU */ - SendMessageW( hwnd, WM_SYSCOMMAND, SC_SIZE + wParam - 2, lParam); + /* Old comment: + * "make sure hittest fits into 0xf and doesn't overlap with HTSYSMENU" + * This was previously done by setting wParam=SC_SIZE + wParam - 2 + */ + /* But that is not what WinNT does. Instead it sends this. This + * is easy to differentiate from HTSYSMENU, because HTSYSMENU adds + * SC_MOUSEMENU into wParam. + */ + SendMessageW( hwnd, WM_SYSCOMMAND, SC_SIZE + wParam - (HTLEFT-WMSZ_LEFT), lParam); break; case HTBORDER: