§ ¶Metro style apps and the Win32 API
It's been way too long since I posted, hasn't it? Time to change that.
There's been a bit of discussion about the difference between "desktop apps" and "Metro style apps" on Windows 8, particularly as it pertains to Windows RT (previously Windows on ARM), which currently isn't going to allow third party development of desktop apps. Part of the confusion comes from the fact that you can use native C++ code in a Metro style app. Native C++ doesn't mean you can just recompile an existing C++ app written on top of Win32, though, and there are lots of misconceptions about what this means for converting existing apps to Metro.
I haven't actually written a Metro app, but I have looked a bit at what's involved out of curiosity. Looking at the documentation and the requirements, I'm of the opinion that most C++ apps will require a major rework to conform to Metro app requirements. This is based on the restrictions to Win32 API usage that will require invasive changes to either the C++ code itself or the frameworks that the C++ code is based on. Nevertheless, it's useful to actually look at the API lists and get an idea of where the trouble spots are, which is what I'm going to do here. Think of this as a what-if exercise, kind of what would happen if someone at work just asked you for a wild guess at what would need to change to retarget Metro.
Of course, in order to do this we'd need some guidelines, so we're going to assume:
- That you actually want to use Windows and a Metro style app. Yeah, I know, definitely not always true... just pretend for the sake of this exercise, then.
- That the application is appropriate for Metro conversion and doesn't have obviously crazy Win32 usage. Sysinternals Process Explorer is not likely to be a good target app for conversion.
- That the UI is going to be rewritten from scratch, so that it actually suits Metro and a tablet. The entire Win32 UI subsystem is gone, so there's no point in wondering about that.
- That we're mainly talking about a C++ core that's reasonably well architected, i.e. has the Win32 API calls reasonably wrapped and is decently modular, so that if an API call has a direct replacement it isn't a big deal to swap it out. We're looking for stumbling blocks here, like a lack of equivalent functionality.
- That the resultant app will be written to properly conform to requirements and won't try to do things like subvert the API restrictions.
- That you're using Visual C++ and can easily interface to WinRT through C++/CX when needed. Code only compiles under GCC? Uh, sorry, that's a whole other set of problems that I can't cover here. I think it's possible to use WinRT entirely through COM APIs, but it's likely to, uh, build character.
Our main reference is going to be "Win32 and COM for Metro style apps" from MSDN:
...which, as of this writing, is preliminary and subject to change of course. I'm just going to go down this list, as well as lookup APIs to see if they're marked as "desktop only" or "desktop and Metro apps", and see what that means for existing code.
So, with all of this in mind, what stumbling blocks that we might encounter?
Well, time to update that file I/O code, starting with using CreateFile2() instead of CreateFile(). Not CreateFileEx()? Someone on the Windows team got tired of *Ex() jokes?
FlushFileBuffers() is allowed, which surprises me as it's one of the best ways to absolutely destroy system performance. I suppose it's still needed for reliable storage.
ReadFileEx() and WriteFileEx() are missing... which I think means no I/O completion ports. It seems like in some ways, Windows 3.1/9x style code actually survives the Metro transition better than NT-style code.
Memory mapped files are supported, interestingly. I guess it'd be bad to ban that as memory mapping helps save memory.
DirectInput: Gone, only XInput allowed. Okay, I can't cry about this one, because DirectInput was a maze of GUIDs and enumeration callbacks and wasn't really direct anything anymore, and XInput is much easier to use. The latter does suffer from the "all gamepads are Xbox 360 gamepads" problem, though.
Debugging and Error Handling
No SetUnhandledExceptionFilter(). This function was long abused in DLLs, so this is also understandable, but it's a bit disappointing as there were lots of good uses for this function, like dumping last ditch recovery information before the program dies. I guess the argument is that Metro style apps shouldn't crash....
I don't see SetVectoredExceptionHandler() either. Hmm.
Direct2D and Direct3D 11. Missing from this list: GDI, GDI+, DirectDraw, Direct3D 9, Direct3D 10, and OpenGL. I don't think there are many programs which wouldn't have to have their graphics layer rewritten for this one.
D3DCompile() is at least supported, which is handy if you need to do really crazy shader transformations. Interestingly, D3DDisassemble() is also supported.
Missing: WaveOut, DirectSound, Multimedia APIs (including Video for Windows), DirectShow. Okay, I'm definitely not shedding a tear for the last two. The replacements: Media Foundation, XAudio2, and Core Audio.
Not having used Media Foundation, I have to wonder how codecs work in this environment. Pluggable codecs help with format independence -- what basically made VirtualDub possible to begin with -- but they're also an magnificent mechanism for app cross-breakage.
XAudio2 doesn't bother me much... particularly if the program I'm working on has its own mixer and just wants a low latency output stream anyway.
Collateral damage: the removal of the multimedia APIs also removes timeGetTime(). I've seen systems where this is the only reliable timing source with millisecond precision. Hopefully QueryPerformanceCounter() is reliable on all Windows 8 systems.
No Winsock or WinInet. Perhaps understandable, given a desire to prevent random connections and the accumulated layers of cruft in Winsock, but there's a lot of tried and tested socket code out there, which will become less tried and tested when rewritten to use equivalent WinRT APIs.
The allowed DHCP calls are curious.
I don't know a whole lot about printing, but given that historically printing has been tied to GDI, it's not surprising that a lot has changed here.
This is where things get interesting.
A lot of COM stuff is still here. COM is underneath WinRT, so this doesn't surprise me. Looks like you'll have to finally change CoInitialize() to CoInitializeEx().
OutputDebugStringA() is absent, but OutputDebugStringW() is allowed. Don't know about you, but I don't generally have my trace messages in Debug builds in wide chars -- wrapper time. For that matter, do they really allow OutputDebugStringW() in a shipped app? That's one of my pet peeves. Traces in dev builds only, please.
GetModuleHandle() and GetModuleHandleEx() are missing. The latter is unfortunate as it's how a library can get the HINSTANCE of the DLL or EXE it's in, but maybe that's less needed without the USER APIs. I guess it's back
to the VirtualQuery() trick otherwise.
VirtualAlloc(), etc. are missing... Ow. That's painful, as that's the main way you allocate memory with alignment requirements, like for a custom allocator -- your own malloc() replacement, or an even more specialized heap that has major alignment requirements like a small block allocator or garbage collector. (HeapAlloc STILL doesn't support 16 byte alignment for SSE.) It's also how you generally allocate memory suited for FILE_FLAG_NO_BUFFERING... although it looks like CreateFileMapping() is still available, if you're willing to juggle the file handles too.
No CreateThread(). There's a Windows.System.Threading namespace suggested as a replacement, but it has a task API instead of a thread API. It looks like it might be abusable to simulate threads, although I'm not sure of the ramifications.
No HeapCreate(). That was an easy way to segregate allocations into another heap without dragging your own allocator.
Sleep() is gone. Okay, they want you to use another mechanism that doesn't lock up a thread... but occasionally, all you need is a plain sleep. I saw someone suggest WaitForSingleObject(GetCurrentThread()) with a timeout... I think I would be mortified to have to use that.
WaitForSingleObject() and WaitForMultipleObjects() have been deprecated instead for their Ex variants... err, OK. I'm not even sure you're allowed to use the APIs that would require an alertable wait.
The Ex/ExW versions of CreateEvent(), CreateMutex(), InitializeCriticalSection(), etc. are required. Okay, directly mappable, but pretty much guarantees revision of any threaded code. Looks like the basic semantics are the same at least so if everything's wrapped you can swap the isolation layer.
Hey look, PulseEvent() is finally dead... after being unusable for all 18 years that it existed....
No GetProcessAffinityMask(). This was one of the ways to tell how many hardware threads you should use. GetSystemInfo() is gone too... that makes it a bit hard to tell how many hardware threads are available, with 2+ being the most interesting test. Maybe there's a WinRT API for it.
FlushInstructionCache() is missing, VirtualAlloc/VirtualProtect() is missing, and CreateFileMapping() cannot set the EXECUTE flag. From what I can tell, this means you aren't allowed to do any sort of dynamic code generation in a Metro style app. This kinda sucks. Any high-performance emulator or VM is going to want to do this, and it's also useful for trampolines from callback pointers. I wonder if writable code segments are allowed.
Again, taking a look at the list above, there's a pretty good chance that the C++ library you might have been planning to reuse in your Metro style app will require some porting work and may possibly have blocking API usage issues, even if you're targeting only x86/x64 and your library is only a GUI-less utility library. Check the API list carefully and make sure the app you have in mind is even possible before estimating porting time.
Is D3DCompile really supported? MS says it's supported for development, but won't pass the AppStore tests: http://social.msdn.microsoft.com/Forums/..
Which is a shame since apparently there's no good way to generate shader assembly either; and there are very valid use cases where runtime shader generation/compilation is a must.
Aras Pranckevičius (link) - 16 04 12 - 18:56
It appears you're correct, it's not actually allowed and they need to update MSDN.
The way around this would be to pregenerate the shaders beforehand and link them in... it can be a crazy number of shaders, but if you do it just right, they compress really well....
Phaeron - 16 04 12 - 19:09
No dynamic code generation? What about games that want to use luajit?
Z.T. - 16 04 12 - 19:25
"HeapAlloc STILL doesn't support 16 byte alignment for SSE"
Is something wrong with _aligned_malloc? http://msdn.microsoft.com/en-us/library/..
Klimax - 17 04 12 - 05:20
I think that in WinRT you are suppose to use std::thread and std::this_thread::sleep() instead of CreateThread()/Sleep().
spax - 17 04 12 - 05:44
I would imagine a game that wants luajit would be a Windows desktop game, and not Metro. After all, you're not running luajit on an iPad.
Ankur - 17 04 12 - 07:39
DirectDraw does seem to be available.
kjk (link) - 17 04 12 - 09:36
> Is something wrong with _aligned_malloc?
_aligned_malloc() is a wrapper around malloc()/HeapAlloc() and thus needs an additional header to point to the enclosing unaligned block. With small objects it will turn the heap into swiss cheese as its packing efficiency is low. It is also very wasteful for large alignments.
I once wrote a specialized small block allocator with very low overhead that also supported 16 byte alignment. The limitation was that it needed 64K blocks aligned on 64K boundaries, which it used VirtualAlloc() for, and which would be much less efficient with HeapAlloc(). In general, any allocator that uses address masking tricks will be less efficient when retargeted on top of another heap that doesn't support the required alignment.
> DirectDraw does seem to be available.
Really? DirectDrawCreate() is marked desktop app only. I suppose you could use CoCreateInstance(), but even if that passed I'm not sure it would work in practice. Mixing 3D APIs I could see, but I don't see DirectDraw getting along well with the Direct3D rendering from the Metro UI system anyway.
Phaeron - 17 04 12 - 15:50
"HLSL compliation is time, processor, and power-consuming"
You heard it, the professional said it, no need for D3DCompile, better not run anything that would tickle the cpu above 0%, just don't do anything at all. Instead, pre-generate all the 2^N permutations of your ifdef sprinkled shaders.
Gabest - 17 04 12 - 16:28
Been there, done that. To be fair, the HLSL compiler *is* pretty heavy -- a long shader can take hundreds of milliseconds to compile. On top of that, at least with older versions there was a giant lock around most of it such that you couldn't parallelize it with threads.
D3DX used to have a facility where you could compile portions of HLSL to fragments and then link them together using the fragment linker. The idea was to bake out the parsing and high-level compilation overhead, leaving only a quick stitch and optimization pass at runtime. Unfortunately, this plan was hampered by the fragment linker never quite working properly.
Binary shaders would be a good alternative since it's fairly easy with current shader models to splice together asm, and the driver has to do a translation + optimization pass anyway. The current attitude of the DX team toward shader assembly has put a damper on this, however.
Phaeron - 17 04 12 - 16:47
Sure, some times it's possible to precompile shaders offline.
Other times, not so much. iPad has an app that allows you to interactively edit shaders and see results - not possible on Metro. A tablet app with a graphical shader editor (nodes & connections etc.) - not possible. Doing a WebGL implementation - not possible (would need something like ANGLE, which translates GLSL into HLSL and then compiles that).
Aras Pranckevičius (link) - 17 04 12 - 18:11
Most important vendors can't afford loosing this market. No matter how restricted the new set of APIs is - they _will_ build their apps. I guess that is the thinking behind this relentless cut.
tobi - 18 04 12 - 03:02
100ms? Like I/O, do it on a thread, hide latency. Is that a big deal?
Gabest - 20 04 12 - 09:11
I tried windows 8, the latest release. After realizing there is no start menu, no minimize, no taskbar, most apps are incopatible, could not find most common interfaces, I realized that the beautiful machine I just built from the ground up running at an amazing 4.7GHz was being a good candidate to throw out the window due to the frustration of using this so called OS. They should rename it to Windows 8 POS. After windows 7 becomes obsolete, MAC here I come. Enough is enough with the Microsoft BS.
release bad carma;
evropej - 25 04 12 - 06:53
>>Someone on the Windows team got tired of *Ex() jokes?
You mean InitCommonControlSEX()? :)
>>...memory mapping helps save memory.
It is also good for processing files larger than available RAM.
So, an image viewer that uses OpenGL for HW accelerated image display/pan/zoom won't work? Great.
>>...No Winsock or WinInet
What to use instead of sockets and WinInet?
That one is not too hard assuming that _vsnprintf() still exists:
void __cdecl Trace(char * format, ...);
void __cdecl Trace(char * format, ...)
_vsnprintf(sbuffer, sizeof(buffer), format, arglist);
// If you make all your strings UTF8 you won't have a
// headache if you decide to port VirtualDub to Linux :)
// (Linux wchar_t is not compatible with Windows wchar_t)
MultiByteToWideChar(CP_UTF8, 0, sbuffer, -1, wbuffer, 1024, NULL, NULL);
>>VirtualAlloc(), etc. are missing... Ow.
Ouch... brain hurts... I can't make any serious stuff without that API.
Ow... seriously WTF... I am never going to install that crap of the OS which only encourages .Net programming!
Given that major gaming studios (Valve, Blizzard, EA) have all announced Linux support and that gaming was the last thing keeping me on Windows platform I think I know what my next OS install will be. If only Avery would port VirtualDub and make it native...
Igor Levicki (link) - 08 05 12 - 00:44
This may be usefull to know for people wanting to develop for WinRT:
Igor Levicki (link) - 18 05 12 - 23:52
Oh yeah baby... no more developing of free desktop applications in Windows 8:
In short, Visual Studio 11 Express will not allow creation of desktop applications.
That means it will be impossible to compile VirtualDub with it.
Igor Levicki (link) - 24 05 12 - 21:00
> That means it will be impossible to compile VirtualDub with it.
VS 2010 will *not* suddenly stop working ...
Frank - 26 05 12 - 13:42
> VS 2010 will *not* suddenly stop working ...
My copy of Visual C++ 6.0 still works too, but that doesn't mean I use it for development anymore, or that libraries and SDKs support it much.
This change in Express is going to cause problems for open source projects that were relying on it for Windows builds, and it's a bit of a betrayal after it was used as justification for eliminating the relatively inexpensive Standard tier of Visual Studio. GCC/MinGW is a comparatively poor development experience on Windows and isn't a good alternative, especially if you are trying to target newer OS APIs or have an existing binary plugin API to deal with.
Phaeron - 26 05 12 - 14:01
> ...No CreateThread()
BTW: I've been a C++ programmer (Win & Linux) for 20 years now and have seen a thing or two. For a first version, WinRT is remarkably well thought out and implemented.
> the OS which only encourages .Net programming!
Dude, that's just nonsense.
> most apps are incopatible
Win 8 on the desktop is about as necessary as a hole in the head ... But I got pretty much everything that works under XP/Win7 working under Win 8 as well. And it's just a preview.
Frank - 26 05 12 - 14:20
Don't get me wrong, I think this is a terrible decision.
Frank - 26 05 12 - 14:23
>VS 2010 will *not* suddenly stop working ...
No, VS2010 will not stop working, but you won't have support for issues you might have with it, and you won't have security updates for SDK and runtime libraries.
>Dude, that's just nonsense.
How else do you call deliberate removal of C/C++ compiler from Windows 8 SDK and Visual Studio 11 Express which can't create desktop applications? Gentle encouragement?
How do you think they will limit Visual Studio 11 Express from creating desktop applications?
I can bet that they will do it by not providing C/C++ compiler, and restricting C# and VB .Net libraries to include only Metro API.
Good thing Intel Parallel Composer is way better than gcc/mingw, cheaper than Visual Studio, and can be used from command line. You will only have to find a linker and librarian (you could use the ones from the old SDK).
Igor Levicki (link) - 27 05 12 - 04:13
Oh and take a look at this:
"C++ developers can also use the multi-targeting capability included in Visual Studio 11 to continue using the compilers and libraries included in Visual Studio 2010 to target Windows XP and Windows Server 2003. Multi-targeting for C++ applications currently requires a side-by-side installation of Visual Studio 2010. Separately, we are evaluating options for C++ that would enable developers to directly target XP without requiring a side-by-side installation of Visual Studio 2010 and intend to deliver this update post-RTM."
What that means for new C++ developers is that they will have to purchase VisualStudio 11 _AND_ Visual Studio 2010 in order to compile applications compatible with Windows XP and Windows Server 2003. Guess what, starting the development in C++ has just had its price doubled compared to C# and VB .Net.
Do you still think that what I am saying is nonsense?
Igor Levicki (link) - 27 05 12 - 04:19
I have always considered using GetModuleHandle() and GetModuleHandleEx() to obtain the HMODULE of a DLL to be a pretty dirty hack since you need to specify the name of the DLL (otherwise the HINSTANCE of the process is returned).
The HMODULE of the DLL is passed to DllMain(), and that still works in WinRT.
Markus Ewald (link) - 22 06 12 - 06:27
Microsoft changed their mind:
There will be desktop edition after all.
Igor Levicki (link) - 23 06 12 - 21:59
ReadFileEx is deprecated in favor of ReadBufferAsync (which is a CLR function), apparently. In Metro, everything seems to be IBuffers, although after a lot of whining they did include IBufferByteAccess to better interface with older code. ifdef'd code to the rescue, yay.
foxyshadis.com (link) - 29 10 12 - 05:03
"I think it's possible to use WinRT entirely through COM APIs, but it's likely to, uh, build character."
I laughed. :)
vonj - 31 10 12 - 01:58
It's also "nice" that DX11.1 only runs F9_1 on Surface. Microsoft also stripped out compile and reflect from the shader system, so I have to build my own. Now FX is only for development purposes (and dynamic interfaces). What a shambles.
Alecazam - 02 12 12 - 07:23
The more sinister problem with the threading architecture, is that you can no longer WaitEx on a thread handle. You have to wait on an event tied to the thread handle. The threads are all part of a thread pool, get recycled, and so the wait would be indefinite. Shawn Heargraves has a Win32 thrad impl for WinRT, but have the thread handle recycle after the async task completes but way before CloseHandle is a big problem for our legacy code.
Alecazam - 02 12 12 - 07:34