¶How to kill Aero Glass on Vista beta 2 with DirectDraw
I figured out why VirtualDub causes Aero Glass to go nuts on Vista beta 2, and the actual reason turned out to be rather bizarre. Had nothing to do with window hierarchy layout.
Aero Glass uses the Desktop Window Manager (DWM) to composite the display, so certain drawing operations from programs that the DWM can't emulate will cause the DWM to disable composition until the program exits. Typical operations that kill the DWM are those that access the full display or those that appear to do so (that the operation is limited to a window can't be determined). Well, I'm not locking the backbuffer or doing anything naughty, but it turns out this was what was disabling the DWM:
// get surface desc
if (FAILED(mpddsPrimary->GetSurfaceDesc(&ddsdPri))) {
DEBUG_LOG("VideoDriver/DDraw: Couldn't get primary descn");
break;
}
Merely calling GetSurfaceDesc() on the primary surface causes the DWM to disable itself. Changing this to use GetDisplayMode() fixed the problem, and it turns out that even good old DirectDraw 3 will work just fine with Aero Glass as long as you use a clipper when you blit to the primary. If you don't use a clipper, bye bye DWM.
Another oddity of beta 2 is that it seems you can create a DirectDraw overlay surface, but it never shows up and you see nice color key green instead. It's not a problem with the color key, because I can disable it and I still won't see an overlay. Sigh.