This issue was unhidden by the last fix for CORE-17561, so it either blocks or just is related to that ticket.
To reproduce the bug:
- Rebuild ROS with 3638, 4146 and 4158 PRs applied. They contain some win32k improvements required by MS DDraw/D3D to work properly. Also remove ASSERT(FALSE); in IntGdiCreateDisplayDC from win32ss/gdi/ntgdi/dclife.c, since that assert also appears when run any DirectDraw/Direct3D apps.
- Install it as usually, with default or custom settings.
- Install Guest Additions with Direct3D support (for VirtualBox <= 5.2.44). For VMWare, extract VMWare SVGA II video driver from setup.exe/VmVideo.cab manually (since it doesn't work correctly when install it from setup.exe), go to the extracted path, and look at the list of files. There will be a files for 2000 and for XP/2003. Remove the files for 2000 (they have _win2k suffix in the name). Then fix the names for remaining files for XP/2003. Remove _ at the start, and everything after the file extension (starts with dot). Do this for each file. Finally install the video driver via Device Manager from that directory and reboot.
- Replace ddraw.dll, d3d8.dll, d3d9.dll and dxg.sys by versions from XP/2003 and reboot again.
- Install any DirectDraw/Direct3D app/game, which runs in fullscreen. Try to run it. Alternatively, you can use the easier way to test the acceleration: MS DxDiag 8 from DirectX 8.1. Since it does not depend on dxdiagn.dll, it does not necessarily to replace dxdiag.exe and dxdiagn.dll in system32. Just extract dxdiag.exe from dxnt.cab where you like (e. g., on Desktop) and run it. Go to the Display tab and click the DirectDraw test button. Perform all 3 tests by clicking OK in each appearing messagebox. The system will bugcheck after starting the 3rd (fullscreen) test. Same as when you run any fullscreen app or game.
After preforming these steps, the BSOD 0x50 PAGE_FAULT_IN_NONPAGED_AREA will appear. It means some memory address is invalid.
The cause of this is still unknown for me, and it's really hard to find it. In VirtualBox it crashes due to VBoxDisp.dll when calling dxg!DxDdCanCreateSurface, but in VMWare, the guilty file is dxg.sys (which we're using from XP/2003 here). In the 2nd case, BSOD happens when calling dxg!DxDdQueryDirectDrawObject. Here's a log which I got from VMWare Workstation 16.2.1: vmware.log. As I know only VMWare uses the true graphics driver, unlike VirtualBox. So, as we can see, the problem comes from MS dxg.sys, not from the video miniport driver, as I thought earlier. But the behaviour of this bug looks very random with different video cards/drivers. So I can't understand what causes it exactly. What I only understand, something invalid comes definitely from our gdi32/win32k. I hope to get some help with fixing this. Thanks.