none
D3DCompile and Metro style games

    Question

  • I need to compile shaders on runtime, but I have discovered that static linking to D3DCompile make the executable to not start (it does not even reach the entry point)

    I have managed to solve this by dynamic loading the d3dcompile_44.dll, is this the only/proper way? Can I publish an application by including the dll in the package?

     

    Thanks

    B

    Monday, January 02, 2012 9:23 PM

Answers

  • Runtime compliation is not supported for deployment. D3DCompile can be used for local development by including D3DCompile_44.dll in your package, but to deploy your Metro style application you use precompiled shaders (you can use FXC.EXE in VS 11 to do this or arrange some other system to get the compiled shader data into data files). D3DCompile_44.dll will not pass the AppStore tests.

    Note that Dynamic loading is also not supported for Metro style applications as LoadLibrary/LoadLibraryEx will also fail AppStore tests.

    Wednesday, January 04, 2012 8:07 PM
  • What the documentation doesn't currently do a good job of explaining is that there are TWO different cases...

     

    * You can use D3DCompile succesfully in a Metro style application BUT ONLY DURING DEVELOPMENT.

    * You cannot DEPLOY a Metro style application through the AppStore because D3DCompiler_44.DLL will not pass the integestion tests for "safe for Metro" API validation.

     

    The assumption is that you may well use runtime compliation for fast iteration during development, but you should precompile all your shaders through some mechanism before you DEPLOY it since HLSL compliation is time, processor, and power-consuming.

    Friday, January 13, 2012 1:17 AM

All replies

  • Runtime compliation is not supported for deployment. D3DCompile can be used for local development by including D3DCompile_44.dll in your package, but to deploy your Metro style application you use precompiled shaders (you can use FXC.EXE in VS 11 to do this or arrange some other system to get the compiled shader data into data files). D3DCompile_44.dll will not pass the AppStore tests.

    Note that Dynamic loading is also not supported for Metro style applications as LoadLibrary/LoadLibraryEx will also fail AppStore tests.

    Wednesday, January 04, 2012 8:07 PM
  • Isn't that completly contradictory to the actual documentation? On the supported win32+com api I can found d3dcompile and LoadPackagedLibrary ARE supported.

    Our rendering system relies a lot on runtime compilation as we use a flexible lighting and meterial system, trying to pre-compile all combinations would be imposible and limiting our number of shaders would also mean to drop features like a unified lighting system....

     

    Thursday, January 05, 2012 12:42 PM
  • I stated that LoadLibrary/LoadLibraryEx is not available for Metro style apps. To use LoadPackagedLibrary, the DLL in question must be packaged and therefore will be subject to the API scanning and thus will be already validated as not containing any non-Metro entry-points.

    D3Compile_44.DLL will function in the Metro style context, but will not pass API scanning for deployment.

    Friday, January 06, 2012 11:23 PM
  • So what is the solution, remove the d3dcompile methods? Why then it is listed as allowed api? Is this an error on the documentation?
    Tuesday, January 10, 2012 3:23 PM
  • Sorry for the confusion. LoadPackagedLibrary is a legacy Metro style app API for doing explict linking. The 'target' DLL must still pass AppStore entry-point validation since it is packaged in an AppX, which D3DCompile_44.DLL will not.
    Tuesday, January 10, 2012 9:54 PM
  • Thanks, althought I was referring to D3DCompile... it is also on the legacy allow api.... Any idea how can be used on a metro style game, or it was an error on documentation?
    Thursday, January 12, 2012 10:19 PM
  • What the documentation doesn't currently do a good job of explaining is that there are TWO different cases...

     

    * You can use D3DCompile succesfully in a Metro style application BUT ONLY DURING DEVELOPMENT.

    * You cannot DEPLOY a Metro style application through the AppStore because D3DCompiler_44.DLL will not pass the integestion tests for "safe for Metro" API validation.

     

    The assumption is that you may well use runtime compliation for fast iteration during development, but you should precompile all your shaders through some mechanism before you DEPLOY it since HLSL compliation is time, processor, and power-consuming.

    Friday, January 13, 2012 1:17 AM
  • Oh.  This is another hurdle we'll have to cross.  We're using Microsoft-sourced libraries that dynamically create shaders on the fly.  I have no idea how many permutations of the shaders it creates at runtime.

    Glad I stumbled across this thread now and not later down the track.

    Friday, January 13, 2012 5:39 AM
  • One solution can be to have the 'runtime' compliation locally generate a cache of shader objects as you use and test your application. You then save off the cache of compiled shader blobs as a file and then ship it in the 'final' version of your application. This is the approach used by the Unreal Engine for example since all the shader generation is very 'artist driven'.

    The new complication for Metro style application is that you also need to "cache" any shader metadata you require to use the shader blobs as well since the D3Compiler_*.DLL also includes the shader reflection APIs.

    Friday, January 13, 2012 8:01 PM
  • you use precompiled shaders (you can use FXC.EXE in VS 11 to do this or arrange some other system to get the compiled shader data into data files).


    I've been looking all over for exactly how I'm supposed to do this. I've used stuff like D3DCompile before, but never this sort of thing and to be honest searchs and MSDN haven't really been much of a help for me here. Is there some place with a tutorial on how to load a precompiled shader for use you could point me to?
    (I have successfully precompiled the shader and added the precompiled file to my project to load, just no idea how I take said precompiled file and actually do something useful with it)
    • Edited by Eric Fisher Sunday, February 26, 2012 12:29 AM Updating note to mention I've already successfully precompiled the shader.
    Sunday, February 26, 2012 12:27 AM
  • Take a look at the Windows 8 Metro app sample "ResourceLoading" which loads prebuilt shaders and other assets from files.

    All a 'pre-compiled' shader is a binary blob of data generated by the HLSL compiler. The HLSL compiler can either save this data directly to a file or it can generate a C-compatible header file that just defines the binary blob as an array.

    Monday, February 27, 2012 9:20 PM
  • Is this still the case with the Consumer Preview?

    The documentation for the D3DCompiler functions says nothing about only being availble for development, it says they can be used in Metro Style apps.

    http://msdn.microsoft.com/en-us/library/windows/apps/dd607324.aspx

    Of course you have to bundle the dll for this to actually work and then it fails cert. If we still aren't allowed to ship with the d3dcompiler dll bundled and it's not going to be included as part of the system you guys really ought to mention it in the documentation.

    Friday, March 09, 2012 5:07 AM
  • So I can understand why compiling shaders at runtime is a bad idea and usage of D3DCompile_xx.dll is disallowed for Metro apps.

    We I need access to is the shader reflection API which can read internals of the binary shader blob.  Since reflection is part of D3DCompile_xx.dll it also cannot be used at runtime.  I don't see reflection as being "time, processor, and power-consuming" as it seems to be just reading data already in the blob.

    Is ID3D11ShaderReflection just not allowed in a Metro app (contrary to what the docs say)?

    Saturday, April 07, 2012 11:43 PM
  • MS guys may hopes us to use the most efficiency way. Reflection data also can be preprocessed and saved with the compiled shader. Or you can implement the shader manager yourself.


    C++ DX11

    Sunday, April 08, 2012 4:49 AM
  • Reflection data also can be preprocessed and saved with the compiled shader.


    Correct me if i'm wrong... but as I understand it the compiled shader data already contains the metadata used for reflection.  So me duplicating that information and saving it along with the shader bytecode seems like a waste.
    Sunday, April 08, 2012 5:31 AM
  • If you can, just using the metadata. ( I've no idea about it)

    Or you can save the reflection results in debug version, and use it in release without ID3D11ShaderReflection.  Not a good solution, but at least it does work.


    C++ DX11

    Sunday, April 08, 2012 9:54 AM
  • And if ID3D11ShaderReflection does not use those "Bad APIs" , just remove the APIs from the D3DComile_xx.DLL PE file's INT. Then will it pass the App Cert ?
    I've not tried yet.

    C++ DX11

    • Edited by Raptor K Sunday, April 08, 2012 10:31 AM
    Sunday, April 08, 2012 10:21 AM
  • That's really an unfortunate choice to restrict the use of D3DCompiler only to dev time. It's not like D3DCompiler is a third party dll...

    I was expecting to develop an application to perform live/interactive coding of shaders and this won't be possible. And as mentioned by Tom Spilman, not only the compilation is forbidden but because D3DCompiler was also providing shader reflection, we must also package on the side of our application, shader metadatas that are already stored in the shader blobs (!). The simple helloworld case where you map a static "c struct" to a constant buffer is far from enough when you need flexibility and pluggability in a shader system.

    So to compile a shader, I would have to provide a remote WebService for this? bhaaa...




    Alexandre Mutel - SharpDX - NShader - Code4k

    Tuesday, April 10, 2012 7:35 AM
  • In theory one could create an alternative reflection API using bits of Mojoshader which knows how to parse compiled FX/HLSL and extract the metadata.

    The compiler issue I suspect will eventually be solved by Microsoft.... it seems stupid to not be able to make shader tools in Metro. 

    In the mean time maybe if someone is really hard up for an HLSL compiler in Metro the work of the Wine team might be a start:

      http://wiki.winehq.org/HLSLCompiler

      http://repo.or.cz/w/wine.git/tree/HEAD:/dlls/d3dcompiler_43

    Tuesday, April 10, 2012 7:59 AM
  • In theory one could create an alternative reflection API using bits of Mojoshader which knows how to parse compiled FX/HLSL and extract the metadata.

    The compiler issue I suspect will eventually be solved by Microsoft.... it seems stupid to not be able to make shader tools in Metro. 

    In the mean time maybe if someone is really hard up for an HLSL compiler in Metro the work of the Wine team might be a start:

      http://wiki.winehq.org/HLSLCompiler

      http://repo.or.cz/w/wine.git/tree/HEAD:/dlls/d3dcompiler_43


    About the reflection of shader bytecode blob format, there is already some documentation from the Windows WDK itself from the headers files "d3d10TokenizedProgramFormat.hpp", "d3d11TokenizedProgramFormat.hpp" but I would find this as a huge waste of time to recode this and not really "future proof"...

    Alexandre Mutel - SharpDX - NShader - Code4k




    Tuesday, April 10, 2012 8:07 AM
  • You can use D3DGetBlobPart to retrieve the reflection data. Then parse and save it and call D3DStripShader to remove the reflection data from the shader. 
    • Edited by Beliaz Wednesday, July 18, 2012 9:53 AM
    Wednesday, July 18, 2012 9:52 AM
  • You can use D3DGetBlobPart to retrieve the reflection data. Then parse and save it and call D3DStripShader to remove the reflection data from the shader. 

    The problem here is not how to access reflection data nor to strip them, but the fact that It forces us to replicate reflection APIs and data on the side of our shaders, which is a pure waste of time to develop (that's of course not a huge deal to develop, just a plainly stupid task). At least, they could have forbid only D3DCompile.* API from a WinRT app and allow reflection APIs.

    Alexandre Mutel - SharpDX - NShader - Code4k

    Wednesday, July 18, 2012 12:37 PM
  • I agree, it seems like a huge duplication of effort to have every developer re-implement the shader reflection APIs. It's pretty unfortunate that runtime compilation is not allowed as there are a number of interesting use cases for dynamically generated shaders in Metro style apps but I can understand why they might feel that restriction is needed. Not allowing the reflection APIs or providing any alternative (at least documenting the format of the reflection data embedded in the existing shaders for example) is a real oversight and is going to create a lot of unnecessary work for developers.
    Tuesday, August 21, 2012 1:04 AM
  • Note that for Windows Store apps on Windows 8.1 Preview, the D3DCompile APIs are now available for production use. The D3DCompile_47.DLL is included with Windows 8.1 so you don't included it in your AppX package.

    See MSDN
    Tuesday, July 16, 2013 8:29 PM
  • OK , it comes back. But why it was removed. I feel ...
    Friday, July 19, 2013 2:52 AM
  • There is a story as to why it shipped the way it did for Windows 8, but that's probably best told over drinks at an conference. What matters now is that it is supported for Windows 8.1.

    The original guidance still applies: if you can precompile your shaders, then do so. Runtime HLSL compilation is never 'fast', and this is particularly true on tablet form-factor platforms. The user experience, performance, and battery life is far superior if your shaders are shipped precompiled.

    You can use any number of techniques for shipping precompiled shaders:

    • Shader blobs can be written out as C header files and included directly into your application (DirectXTK's BasicEffect uses this approach)
    • The Visual Studio 2012and 2013 IDE can be used to compile your shaders into .cmo files at build time which you can ship in your AppX
    • You can make use of D3DCompile on Windows 8 or Windows 8.1 to generate a 'shader cache' during development and ship the fully populated cache to end-users in your AppX.

    For the 'shader combinatorial' problem that drives some people to think they have to do runtime compilation, you may want to look at the new HLSL shader linker technology fit for your application.

    Not having shader reflection in Windows 8 was a major limitation, which is now addressed in Windows 8.1.



    Friday, July 19, 2013 4:47 AM