none
What is the true scope of CP_UTF8 in MapiMessage::ulReserved?

    Question

  • MapiMessage documentation page [1] mentions a special value possible for MapiMessage::ulReserved: when it's CP_UTF8, then the struct's "lpszSubject, lpszNoteText, lpszMessageType, lpszDateReceived, lpszConversationID" are "UTF-8 instead of ANSI strings". I couldn't find other mentions of this special value in the documentation.

    Yet, while debugging Thunderbird MAPI implementation, I see that MS Word 2016 uses this special value when sends the document using File->Share->Email->Send as Attachment (when it uses MAPISendMail, even when MAPISendMailW is available); and when doing so, the special value has at least one additional effect: in the MapiFileDesc struct [2] pointed to by lpFiles, both its strings lpszPathName and lpszFileName and also become UTF-8 (I see that when file name is not representable in current ACP, and ACP isn't the beta UTF-8 in Windows 10).

    In my testing, I couldn't come with a method how to test other strings in the data passed by Word; but it is possible that also the strings inside MapiRecipDesc [3] (lpszName, lpszAddress), could be affected (e.g., what is pointed by lpOriginator, lpRecips) - or not, e.g. if they are expected to be in ASCII (who knows - there's no mention about that in the documentation!).

    So the question is: *what is the real scope of the CP_UTF8 value of MapiMessage::ulReserved*? Which exhaustive set of data should I treat as UTF-8 with that flag when implementing MAPI DLL? Thank you!

    [1] https://docs.microsoft.com/en-us/windows/desktop/api/mapi/ns-mapi-mapimessage

    [2] https://docs.microsoft.com/en-us/windows/desktop/api/mapi/ns-mapi-mapifiledesc

    [3] https://docs.microsoft.com/en-us/windows/desktop/api/mapi/ns-mapi-mapirecipdesc



    • Edited by MikeKaganski Saturday, January 19, 2019 1:08 PM MapiFileDesc::lpszPathName is also affected
    Friday, January 18, 2019 3:02 PM

All replies

  • So the question is: *what is the real scope of the CP_UTF8 value of MapiMessage::ulReserved*?
    Wouldn't that be an implementation detail of whatever email client is providing Simple MAPI functionality?
    Friday, January 18, 2019 9:13 PM
  • Wouldn't that be an implementation detail of whatever email client is providing Simple MAPI functionality?

    Of course not. It's user application (e.g., Word) that sends the data with the flag set, and strings encoded accordingly; and the email client that implements Simple MAPI must interpret that properly. See e.g. TB bug 1521007 [1].

    [1] https://bugzilla.mozilla.org/show_bug.cgi?id=1521007

    Friday, January 18, 2019 10:20 PM
  • What I meant was that the MS documentation was silent regarding the use of UTF-8 for the mapifiledesc and mapirecipdesc structs.
    Friday, January 18, 2019 11:00 PM
  • The MapiRecipDesc and MapiFileDesc structs are part of the data pointed to by different MapiMessage members; they are not opaque data blobs, but have their well-defined members, including strings. Those strings are prepared by calling application, and are meant for mail client processing; and failing that (due to assuming encoding different than calling application used to fill them) results in wrong addresses, names, file names, and/or attachments missing completely. In general, they are no less part of the public API than the members mentioned in the MapiMessage documentation as affected by the CP_UTF8 setting; and since at least one string inside the MapiFileDesc is also affected, there is clearly documentation defect about that.
    Saturday, January 19, 2019 6:26 AM
  • Some more data I was able to get.

    There is a MAPISendMailHelper helper function [1] that is designed to allow applications to use MAPISendMailW semantics, and not rely on its presence in the MAPI32.DLL, regardless of OS (the MAPISendMailW is only available in the Windows-provided DLL since Windows 8 [2]). It is implemented in Windows SDK (e.g., in Windows 10 SDK v.17763): see %Program Files (x86)%\Windows Kits\10\Include\10.0.17763.0\um\MapiUnicodeHelp.h for full source code. It tries to load MAPI32.DLL in DLL load path (normally from Windows' System32), and if it contains MAPISendMailW, it's used; otherwise, more steps are preformed, including conversion from passed UTF-16 to 8-bit encodings if required. And it has an implementation of the UTF-8 encoding which is discussed in my question.

    What I found out is that it checks a dedicated registry DWORD value "SupportUTF8", which should be under mail client reg key [3] [4], and be non-0 to indicate that the library supports CP_UTF8 indication (again, the registry value is undocumented as far as I can tell).

    MS Word, apparently, just ignores the setting, and sends UTF-8-encoded data regardless if the registry setting is present or not.

    Another thing to note about the MAPISendMailHelper is that at least of SDK 10.0.17763.0, it is incompliant with the documentation [5]. The documentation says that lpszSubject, lpszNoteText, lpszMessageType, lpszDateReceived, lpszConversationID are affected by the CP_UTF8 flag; but MAPISendMailHelper only handles lpszSubject and lpszNoteText accordingly, and all other strings (including those inside MapiFileDesc and MapiRecipDesc, and notably MapiFileDesc::lpszPathName) being encoded using ACP (this is not only incompliant with documentation, but also with what Word does wrt MapiFileDesc::lpszPathName!).

    The documentation and implementations inconsistencies need attention!

    [1] https://docs.microsoft.com/en-us/windows/desktop/api/mapiunicodehelp/nf-mapiunicodehelp-mapisendmailhelper

    [2] https://docs.microsoft.com/en-us/windows/desktop/api/mapi/nc-mapi-mapisendmailw

    [3] https://docs.microsoft.com/en-us/previous-versions/windows/desktop/windowsmapi/mapi32-dll-stub-registry-settings

    [4] https://docs.microsoft.com/en-us/office/client-developer/outlook/mapi/how-to-choose-a-specific-version-of-mapi-to-load

    [5] https://docs.microsoft.com/en-us/windows/desktop/api/mapi/ns-mapi-mapimessage


    Saturday, January 19, 2019 10:05 AM
  • Or perhaps MS believes that the documentation is adequate since the Windows LPSTR data type would also be used for UTF-8.  I guess we'll wait and see if this topic receives an authoritative response.

    Saturday, January 19, 2019 11:40 AM