The normal code is like:
createDeviceFlags |= D3D11_CREATE_DEVICE_DEBUG;
Since we want to debug some issues in our release version app, we still enable it in release build. Seems we get some crash now(but no crash in debug build and run), is there any limitation to use debug layer runtime in a release built app?
You can use this flag in a "Release" build (I.e. an optimized build) on a developer or test machine. Likely your crash is related to some aspect of your app that changes when optimized, so look for a bug in your app such as an uninitialized variable.
Keep in mind the SDK Debug Layer is not present on end-user machines, so D3D11_CREATE_DEVICE_DEBUG will always fail. This is only an option for developer systems and test systems when the Windows 8.0 SDK is installed on the machine (it's included with VS 2012).