§ ¶File dialogs break in capture mode with UScreenCapture driver
If you are using VirtualDub with the UScreenCapture driver to capture video from the desktop screen, you may run into problems with the Set Capture File option in VirtualDub not bringing up a file dialog, at least on Windows 7. This appears to be due a problem within v2.0.14 of UScreenCapture.ax which also reproduces with Microsoft GraphEdit. I am trying to contact the vendor for a resolution, but have not gotten a reply as of yet.
VirtualDub's file dialogs fail to come up because of an internal error within the Windows common file dialog. The specific failure path is GetSaveFileName() -> DialogBoxIndirectParam() -> CreateWindowEx(ComboBoxEx) -> SetWindowSubclass() -> GlobalAddAtom(). That in turn is caused by UScreenCapture's worker thread calling SetProcessWindowStation() with a handle to WinSta0 that does not have the WINSTA_ACCESSGLOBALATOMS access right. This only happens while the capture driver is running, but as it pretty fundamentally borks UI process state, it's not something I can easily work around safely.
Stupid vendor mutating process-global state. Cases like this make me doubt that being able to load plugins in-process is a good thing.
tobi - 19 05 13 - 09:51
Unfortunately more common than you think. I once tracked down a locale bug to D3DX calling setlocale() to force a particular CRT locale around its parser. Unfortunately, that's a process-wide state and D3DX is linked against the dynamic CRT, so you can imagine what that did to some of the other worker threads in the process.
Phaeron - 19 05 13 - 09:55
This has been already been fixed in UScreenCapture. Use version 2.0.15.
Steve Fox - 24 05 13 - 16:08
You mean "just fixed," since 2.0.14 was the latest available as of two weeks ago... but yes, I've confirmed 2.0.15 no longer calls SetProcessWindowStation() and the problem no longer occurs.
Phaeron - 24 05 13 - 17:09
Please keep comments on-topic for this entry.
If you have unrelated comments about VirtualDub, the forum is a better place to post them.