D3DCompile and Metro style games

Question

I need to compile shaders on runtime, but I have discovered that static linking to D3DCompile make the executable to not start (it does not even reach the entry point)

I have managed to solve this by dynamic loading the d3dcompile_44.dll, is this the only/proper way? Can I publish an application by including the dll in the package?

 

Thanks

B

Answer 1

Runtime compliation is not supported for deployment. D3DCompile can be used for local development by including D3DCompile_44.dll in your package, but to deploy your Metro style application you use precompiled shaders (you can use FXC.EXE in VS 11 to do this or arrange some other system to get the compiled shader data into data files). D3DCompile_44.dll will not pass the AppStore tests.

Note that Dynamic loading is also not supported for Metro style applications as LoadLibrary/LoadLibraryEx will also fail AppStore tests.

Answer 2

Isn't that completly contradictory to the actual documentation? On the supported win32+com api I can found d3dcompile and LoadPackagedLibrary ARE supported.

Our rendering system relies a lot on runtime compilation as we use a flexible lighting and meterial system, trying to pre-compile all combinations would be imposible and limiting our number of shaders would also mean to drop features like a unified lighting system....

 

Answer 3

I stated that LoadLibrary/LoadLibraryEx is not available for Metro style apps. To use LoadPackagedLibrary, the DLL in question must be packaged and therefore will be subject to the API scanning and thus will be already validated as not containing any non-Metro entry-points.

D3Compile_44.DLL will function in the Metro style context, but will not pass API scanning for deployment.

Answer 4

So what is the solution, remove the d3dcompile methods? Why then it is listed as allowed api? Is this an error on the documentation?

Answer 5

Sorry for the confusion. LoadPackagedLibrary is a legacy Metro style app API for doing explict linking. The 'target' DLL must still pass AppStore entry-point validation since it is packaged in an AppX, which D3DCompile_44.DLL will not.

Answer 6

Thanks, althought I was referring to D3DCompile... it is also on the legacy allow api.... Any idea how can be used on a metro style game, or it was an error on documentation?

Answer 7

What the documentation doesn't currently do a good job of explaining is that there are TWO different cases...

 

* You can use D3DCompile succesfully in a Metro style application BUT ONLY DURING DEVELOPMENT.

* You cannot DEPLOY a Metro style application through the AppStore because D3DCompiler_44.DLL will not pass the integestion tests for "safe for Metro" API validation.

 

The assumption is that you may well use runtime compliation for fast iteration during development, but you should precompile all your shaders through some mechanism before you DEPLOY it since HLSL compliation is time, processor, and power-consuming.

Answer 8

Oh.  This is another hurdle we'll have to cross.  We're using Microsoft-sourced libraries that dynamically create shaders on the fly.  I have no idea how many permutations of the shaders it creates at runtime.

Glad I stumbled across this thread now and not later down the track.

Answer 9

One solution can be to have the 'runtime' compliation locally generate a cache of shader objects as you use and test your application. You then save off the cache of compiled shader blobs as a file and then ship it in the 'final' version of your application. This is the approach used by the Unreal Engine for example since all the shader generation is very 'artist driven'.

The new complication for Metro style application is that you also need to "cache" any shader metadata you require to use the shader blobs as well since the D3Compiler_*.DLL also includes the shader reflection APIs.

Answer 10

you use precompiled shaders (you can use FXC.EXE in VS 11 to do this or arrange some other system to get the compiled shader data into data files).

I've been looking all over for exactly how I'm supposed to do this. I've used stuff like D3DCompile before, but never this sort of thing and to be honest searchs and MSDN haven't really been much of a help for me here. Is there some place with a tutorial on how to load a precompiled shader for use you could point me to?
(I have successfully precompiled the shader and added the precompiled file to my project to load, just no idea how I take said precompiled file and actually do something useful with it)

Answer 11

Take a look at the Windows 8 Metro app sample "ResourceLoading" which loads prebuilt shaders and other assets from files.

All a 'pre-compiled' shader is a binary blob of data generated by the HLSL compiler. The HLSL compiler can either save this data directly to a file or it can generate a C-compatible header file that just defines the binary blob as an array.

Answer 12

Is this still the case with the Consumer Preview?

The documentation for the D3DCompiler functions says nothing about only being availble for development, it says they can be used in Metro Style apps.

http://msdn.microsoft.com/en-us/library/windows/apps/dd607324.aspx

Of course you have to bundle the dll for this to actually work and then it fails cert. If we still aren't allowed to ship with the d3dcompiler dll bundled and it's not going to be included as part of the system you guys really ought to mention it in the documentation.

Answer 13

So I can understand why compiling shaders at runtime is a bad idea and usage of D3DCompile_xx.dll is disallowed for Metro apps.

We I need access to is the shader reflection API which can read internals of the binary shader blob.  Since reflection is part of D3DCompile_xx.dll it also cannot be used at runtime.  I don't see reflection as being "time, processor, and power-consuming" as it seems to be just reading data already in the blob.

Is ID3D11ShaderReflection just not allowed in a Metro app (contrary to what the docs say)?

Answer 14

MS guys may hopes us to use the most efficiency way. Reflection data also can be preprocessed and saved with the compiled shader. Or you can implement the shader manager yourself.

---

C++ DX11

Answer 15

Reflection data also can be preprocessed and saved with the compiled shader.

Correct me if i'm wrong... but as I understand it the compiled shader data already contains the metadata used for reflection.  So me duplicating that information and saving it along with the shader bytecode seems like a waste.

Answer 16

If you can, just using the metadata. ( I've no idea about it)

Or you can save the reflection results in debug version, and use it in release without ID3D11ShaderReflection.  Not a good solution, but at least it does work.

---

C++ DX11

Answer 17

And if ID3D11ShaderReflection does not use those "Bad APIs" , just remove the APIs from the D3DComile_xx.DLL PE file's INT. Then will it pass the App Cert ?
I've not tried yet.

---

C++ DX11

Answer 18

That's really an unfortunate choice to restrict the use of D3DCompiler only to dev time. It's not like D3DCompiler is a third party dll...

I was expecting to develop an application to perform live/interactive coding of shaders and this won't be possible. And as mentioned by Tom Spilman, not only the compilation is forbidden but because D3DCompiler was also providing shader reflection, we must also package on the side of our application, shader metadatas that are already stored in the shader blobs (!). The simple helloworld case where you map a static "c struct" to a constant buffer is far from enough when you need flexibility and pluggability in a shader system.

So to compile a shader, I would have to provide a remote WebService for this? bhaaa...

---

Alexandre Mutel - SharpDX - NShader - Code4k

Answer 19

In theory one could create an alternative reflection API using bits of Mojoshader which knows how to parse compiled FX/HLSL and extract the metadata.

The compiler issue I suspect will eventually be solved by Microsoft.... it seems stupid to not be able to make shader tools in Metro. 

In the mean time maybe if someone is really hard up for an HLSL compiler in Metro the work of the Wine team might be a start:

  http://wiki.winehq.org/HLSLCompiler

  http://repo.or.cz/w/wine.git/tree/HEAD:/dlls/d3dcompiler_43

Answer 20

In theory one could create an alternative reflection API using bits of Mojoshader which knows how to parse compiled FX/HLSL and extract the metadata.

The compiler issue I suspect will eventually be solved by Microsoft.... it seems stupid to not be able to make shader tools in Metro. 

In the mean time maybe if someone is really hard up for an HLSL compiler in Metro the work of the Wine team might be a start:

  http://wiki.winehq.org/HLSLCompiler

  http://repo.or.cz/w/wine.git/tree/HEAD:/dlls/d3dcompiler_43

About the reflection of shader bytecode blob format, there is already some documentation from the Windows WDK itself from the headers files "d3d10TokenizedProgramFormat.hpp", "d3d11TokenizedProgramFormat.hpp" but I would find this as a huge waste of time to recode this and not really "future proof"...

---

Alexandre Mutel - SharpDX - NShader - Code4k

Answer 21

You can use D3DGetBlobPart to retrieve the reflection data. Then parse and save it and call D3DStripShader to remove the reflection data from the shader. 

Answer 22

You can use D3DGetBlobPart to retrieve the reflection data. Then parse and save it and call D3DStripShader to remove the reflection data from the shader. 

The problem here is not how to access reflection data nor to strip them, but the fact that It forces us to replicate reflection APIs and data on the side of our shaders, which is a pure waste of time to develop (that's of course not a huge deal to develop, just a plainly stupid task). At least, they could have forbid only D3DCompile.* API from a WinRT app and allow reflection APIs.

---

Alexandre Mutel - SharpDX - NShader - Code4k

Answer 23

I agree, it seems like a huge duplication of effort to have every developer re-implement the shader reflection APIs. It's pretty unfortunate that runtime compilation is not allowed as there are a number of interesting use cases for dynamically generated shaders in Metro style apps but I can understand why they might feel that restriction is needed. Not allowing the reflection APIs or providing any alternative (at least documenting the format of the reflection data embedded in the existing shaders for example) is a real oversight and is going to create a lot of unnecessary work for developers.

Answer 24

Note that for Windows Store apps on Windows 8.1 Preview, the D3DCompile APIs are now available for production use. The D3DCompile_47.DLL is included with Windows 8.1 so you don't included it in your AppX package.

See MSDN

Answer 25

OK , it comes back. But why it was removed. I feel ...

Answer 26

There is a story as to why it shipped the way it did for Windows 8, but that's probably best told over drinks at an conference. What matters now is that it is supported for Windows 8.1.

The original guidance still applies: if you can precompile your shaders, then do so. Runtime HLSL compilation is never 'fast', and this is particularly true on tablet form-factor platforms. The user experience, performance, and battery life is far superior if your shaders are shipped precompiled.

You can use any number of techniques for shipping precompiled shaders:

• Shader blobs can be written out as C header files and included directly into your application (DirectXTK's BasicEffect uses this approach)
• The Visual Studio 2012and 2013 IDE can be used to compile your shaders into .cmo files at build time which you can ship in your AppX
• You can make use of D3DCompile on Windows 8 or Windows 8.1 to generate a 'shader cache' during development and ship the fully populated cache to end-users in your AppX.

For the 'shader combinatorial' problem that drives some people to think they have to do runtime compilation, you may want to look at the new HLSL shader linker technology fit for your application.

Not having shader reflection in Windows 8 was a major limitation, which is now addressed in Windows 8.1.