locked
Low level crypto API decrypt function fail after verifying clear sign message in Outlook 2010; 2013 RRS feed

  • Question

  • In our add-on we use "low level" crypto api function "CryptMsgControl" to decrypt PKCS7 messages. Recently we notice the issue when the decryption fail and the "CryptMsgControl" function return "0x0000000D" error.

    To reproduce the issue user needs to ...

    -select clear signed message (Microsoft native handling),

    -click to verify it (little red cert icon on the corner),

    -close verification dialog,

    -go to enveloped message (Our add-on handling),

    -try to decrypt the message, it will fail with the error above.

    Work around ...

    - close, and open Outlook decrypt of enveloped messaged work again

    - delete signed message, decryption will work

    - select the same signed message, DO NOT verify the signature, select any other message, decryption will work.

    This list of work around made us certain there is the problem with Outlook or underlying crypto API which was probably introduced some time back as we use this code for more than 10 years. As of my research with I'll post later I think during verification of clear sign message, temporary password generated to verify keychain and it looks like it cached or context is not released properly. This is why when we use low level API after that this temporary password used and password prompt doesn't come up to accrue access to private key. Of cause this temp password is wrong and the function return error.

    The following is our research ...

    1. As of debugging through the decryption code the Microsoft crypto function which fail is "CryptMsgControl" (http://msdn.microsoft.com/en-us/library/windows/desktop/aa380220%28v=vs.85%29.aspx ); this function is the "low level" crypto api function to decrypt a PKCS7 message. There is "high level" function of crypto api "CryptDecryptMessage" (http://msdn.microsoft.com/en-us/library/windows/desktop/aa379915(v=vs.85).aspx ) which we don't use in purpose. This high level function provide UI box for entering password we cannot hook and show our password dialog instead.
    2. The decryption function produce error "wrong password entered"; there is not much to say, the password used to open up private key is wrong. The function doesn't populate any UI for password; this means it uses password from somewhere ... cache from previous crypto operation (user just validated signed e-mail and temporary password was generated by crypto API to build keychain to check validity)?
    3. This temporary password should not be kept and available to next crypto operation as the crypto context must be cleared after use by calling function "CertFreeCertificateContext" ( http://msdn.microsoft.com/en-us/library/windows/desktop/aa376075(v=vs.85).aspx ); Something keeps reference to it and probably Outlook doing it as ongoing bug (same issue on OL2010; 2013)
    4. It has to be the way to set for crypto API setting do not cache the passwords at all. This lead me to registry settings used by MS crypto. There are private key caching constants (http://msdn.microsoft.com/en-us/library/windows/desktop/aa387403(v=vs.85).aspx ). One of them clearly should indicate we don't want any private key password cache "AllowCachePW".

    And the following is try/fail to fix the issue ...

    1. Verified and debugged existing code for decryption. Inside our function there are lot of manipulations which can lead to the error in the "CryptMsgControl" function. As such ... opening message stream; decoding the content; enumerating certificates of recipients; acquiring private keys for this recipients and finally using first acquired private key context for MS function to decrypt the message. All of those function before looks like work and produce stable result in case when decryption function work and when it's failing.
    2. Tried to acquire crypto context, which I know will not work as the private key will be opened by some "cached" password; release this context and acquire it again to use with decrypt function. This failed. 
    3. Tried to open totally new pointer to System Store to use it for all of the function for look up recipients credentials. This failed as well. 
    4. Tried different flags to use with "CryptAcquireCertificatePrivateKey" (http://msdn.microsoft.com/en-us/library/windows/desktop/aa379885%28v=vs.85%29.aspx ); this happen when we enumerate certificates inside the message and trying to get corresponding private key to open it up. This didn't help.
    5. Tried to use different registry keys for crypto API as of "research #4" with restart of Outlook / System; this didn't make any difference. 

    Is there any advise, other than open MSDN ticket for support?

    Thank you,

    Slava

    Friday, December 19, 2014 5:49 PM

All replies

  • Hi Slava,

    Thanks for your question.

    This is the forum to discuss questions and feedback for Microsoft Office client. Since your query is directly related to the function of CryptMsgControl, I'm moving it to the forum of Windows Desktop Development, where you can get more experienced responses:

    https://social.msdn.microsoft.com/Forums/en-US/home?forum=windowssecurity

    The reason why we recommend posting appropriately is you will get the most qualified pool of respondents, and other partners who read the forums regularly can either share their knowledge or learn from your interaction with us. Thank you for your understanding.

    Regards,

    Ethan Hua

    Forum Support


    Come back and mark the replies as answers if they help and unmark them if they provide no help.

    If you have any feedback on our support, please click here
    Monday, December 22, 2014 6:33 AM