none
How to read eml with large size(i.e contents lots of msg attachment) and allocate memory for it without crashing the application. RRS feed

  • Question

  • Hi,

    Since IConverterSession is not retaining the format(HTML) when attachment is MSG type itself, so I am using Redemption which does  fix this issue. But it has one limitation as it reads all the contents of mail at one shot one put it into a buffer. so when i tried to dynamically allocate the memory as big as the size of file(mail)  application crashes because  my clients does not have enough memory to allocate that much memory. I tried to handle the exception using try and catch and also using nothrow. At the worst case we i can't even catch the  exception. the application aborts with Debug error Invalid Allocation size:4294967295.

    can someone help me out with this issue? i would be grateful.

    thanks,

    Thopulou


    thopulou Associate Software Engineer

    Wednesday, March 28, 2018 11:37 AM

Answers

  • Buffer.Allocate((ulFileSize.LowPart + 1) * sizeof(wchar_t));



    void Allocate(const SIZE_T size)
    {
    len = size + 1;
    ptr = new wchar_t[len];
    wmemset(ptr, L'\0', len);
    }

    The call to Buffer.Allocate passes in a value that is a little more than twice the file size (in bytes) returned by GetFileSizeEx since it multiplies the returned value by sizeof(wchar_t) which is 2.

    Then, the allocate function uses the 2x value to set the number of elements to be allocated in a wchar_t array.  So the number of bytes that the new operator is trying to allocate is now slightly more than 4x the size of the file.


    • Marked as answer by D thopulou Thursday, April 5, 2018 6:17 AM
    Thursday, March 29, 2018 12:07 PM
  • Yes, but you also allocate a memory buffer. Have you tried to avoid using HrRFC822MessageToIMessage completely and simply use RDOMail.Import(..., olRfc822)? It will not read the whole file into memory.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    • Marked as answer by D thopulou Thursday, April 5, 2018 6:17 AM
    Thursday, March 29, 2018 3:56 PM
  • If you are using a memory mapped file, why not use RDOMail.SaveAs? It will do the same thing.

    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    • Marked as answer by D thopulou Monday, April 9, 2018 9:51 AM
    Wednesday, April 4, 2018 3:59 AM
  • STG_E_SHAREVIOLATION is returned because e you are specifying STGM_SHARE_EXCLUSIVE flag and the MSG fiel is still open (probably by the RDOMail object).

    Are you sure you need that flag? Did you release the corresponding RDOMail object?


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    • Marked as answer by D thopulou Thursday, April 5, 2018 6:17 AM
    Wednesday, April 4, 2018 3:40 PM

All replies

  • What is the relevant snippet of your code? Why not save the message to a file?


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    Wednesday, March 28, 2018 3:42 PM
  • Hi Dmitry,

    I actually save the message to a file

    here is my code snippet:

    hSourceEMLFile = CreateFile(sMIMEFilePath, GENERIC_READ, 0, 0, OPEN_EXISTING, 0, 0);
    if (INVALID_HANDLE_VALUE == hSourceEMLFile)
    {
    break;
    }

    LARGE_INTEGER ulFileSize;
    if (!GetFileSizeEx(hSourceEMLFile, &ulFileSize))
    {
    break;
    }

    if (0 == ulFileSize.LowPart)
    {
    break;
    }

    SmartWCHARArray Buffer;

    It fails at allocate function code snipet is given below:

    Buffer.Allocate((ulFileSize.LowPart + 1) * sizeof(wchar_t));
    if (NULL == Buffer.ptr)
    {
    break;
    }

    DWORD dwRead = 0;
    if (!ReadFile(hSourceEMLFile, Buffer.ptr, ulFileSize.LowPart, &dwRead, NULL))
    {
    break;
    }

    Buffer.ptr[dwRead] = 0;
    Buffer.ptr[ulFileSize.LowPart] = 0;

    hr = m_pfnRFCToMsg(spTargetMessage.GetNonAddRefedInterface(),
    m_spMAPISession.GetNonAddRefedInterface(), dwRead, Buffer.ptr);

    Allocate Function:

    struct SmartWCHARArray
    {
    SmartWCHARArray() : ptr(NULL), len(0)
    { }

    void Allocate(const SIZE_T size)
    {
    len = size + 1;
    ptr = new wchar_t[len];
    wmemset(ptr, L'\0', len);
    }

    ~SmartWCHARArray() { SAFE_DELETE_ARRAY(ptr); }
    void Detach(){ SAFE_DELETE_ARRAY(ptr); }

    wchar_t* ptr;
    SIZE_T len;

    };

    Please help me out this is a high priority escalation.

    thank you so much.

    thanks,

    thopulou


    thopulou Associate Software Engineer

    Thursday, March 29, 2018 5:56 AM
  • Buffer.Allocate((ulFileSize.LowPart + 1) * sizeof(wchar_t));



    void Allocate(const SIZE_T size)
    {
    len = size + 1;
    ptr = new wchar_t[len];
    wmemset(ptr, L'\0', len);
    }

    The call to Buffer.Allocate passes in a value that is a little more than twice the file size (in bytes) returned by GetFileSizeEx since it multiplies the returned value by sizeof(wchar_t) which is 2.

    Then, the allocate function uses the 2x value to set the number of elements to be allocated in a wchar_t array.  So the number of bytes that the new operator is trying to allocate is now slightly more than 4x the size of the file.


    • Marked as answer by D thopulou Thursday, April 5, 2018 6:17 AM
    Thursday, March 29, 2018 12:07 PM
  • Yes, but you also allocate a memory buffer. Have you tried to avoid using HrRFC822MessageToIMessage completely and simply use RDOMail.Import(..., olRfc822)? It will not read the whole file into memory.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    • Marked as answer by D thopulou Thursday, April 5, 2018 6:17 AM
    Thursday, March 29, 2018 3:56 PM
  • Hi,

    Thank you so much for your answer. I will do the necessary changes.

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Monday, April 2, 2018 9:09 AM
  • Hi Dimitry,

    Thank you so much for your suggestions. I have tried using RDOMail.Import(..., olRfc822). I will share with my manager about using  RDOMail.Import(..., olRfc822).

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Monday, April 2, 2018 9:14 AM
  • Hi Dimitry,

    For testing i am  using Memory - Mapped Files APIs 

    like this 

    DWORD dwRead = 0;
    HANDLE hMapFile;
    LPCTSTR pBuf;
    hMapFile = CreateFileMapping(hSourceEMLFile, NULL, PAGE_READONLY, 0, ulFileSize.LowPart, NULL);
    if (hMapFile == NULL)
    {
    return 1;
    }
    pBuf = (LPTSTR)MapViewOfFile(hMapFile,   // handle to map object
    FILE_MAP_READ, 
    0,
    0,
    0);
    //CloseHandle(hMapFile);

                           Failed Here Pbuf Contains Junk(chinese Characters) values

    hr = m_pfnRFCToMsg(spTargetMessage.GetNonAddRefedInterface(),
    m_spMAPISession.GetNonAddRefedInterface(), dwRead, (PVOID)pBuf);

    So my question is can i use Memory-Mapped Files APIs with Redemption's HrRFC822MessageToIMessage ?

    I am very new to Memory-Mapped Files and Redemption.

    I would be very grateful again if you could point where i am making mistake

    Thanks,

    Thopulou



    thopulou Associate Software Engineer

    Tuesday, April 3, 2018 10:58 AM
  • pBuf = (LPTSTR)MapViewOfFile(hMapFile,   // handle to map object

    FILE_MAP_READ, 
    0,
    0,
    0);
    //CloseHandle(hMapFile);

                           Failed Here Pbuf Contains Junk(chinese Characters) values


    If MapViewOfFile had failed then pBuf would be null. 

    I suspect that you are building for UNICODE so the cast to LPTSTR is resolving to LPWSTR (i.e., wchar_t*).  If you know that your data is ASCII and not Unicode then cast to LPSTR.

    Tuesday, April 3, 2018 11:18 AM
  • If you are using a memory mapped file, why not use RDOMail.SaveAs? It will do the same thing.

    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    • Marked as answer by D thopulou Monday, April 9, 2018 9:51 AM
    Wednesday, April 4, 2018 3:59 AM
  • Hi Dimitry,

    I am now using RDOMail.Import() and RDOMail.SaveAs() and eml to msg converted successfully.

    But i have more error in my code.

    i am using StgOpenStorage() like this

    DWORD dwFlags = (bOpenReadWrite ? STGM_READWRITE : STGM_READ) | STGM_SHARE_EXCLUSIVE | STGM_TRANSACTED;
    //const DWORD dwFlags = STGM_READWRITE | STGM_SHARE_EXCLUSIVE | STGM_TRANSACTED;

    hr = StgOpenStorage(sFilePath, NULL, dwFlags, NULL, 0, spIStorage.GetReleasedInterfaceReference());

    It gives me an hr error as 0x80030020 : A share violation has occurred

    what am i missing?

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Wednesday, April 4, 2018 12:26 PM
  • Hi,

    I am dealing with data that contains Unicode as well.

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Wednesday, April 4, 2018 12:29 PM
  • Hi,

    This answer really helps. Now service is not crashing. This will go as a patch. Currently i am trying other solutions using memory-maped files and Redemption.

    Thank you for your answer

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Wednesday, April 4, 2018 12:33 PM
  • STG_E_SHAREVIOLATION is returned because e you are specifying STGM_SHARE_EXCLUSIVE flag and the MSG fiel is still open (probably by the RDOMail object).

    Are you sure you need that flag? Did you release the corresponding RDOMail object?


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    • Marked as answer by D thopulou Thursday, April 5, 2018 6:17 AM
    Wednesday, April 4, 2018 3:40 PM
  • Hi Dimitry,

    This is how i tried to get MSG in unicode format

    hr = spRDOMail->Import(sMIMEFilePath, _variant_t(1024));
    if (S_OK == hr)
    {
    hr = spRDOMail->SaveAs(sDestinationFilePath, _variant_t((int)rdoSaveAsType::olMSGUnicode));
    }

    but msg file still has some junk values after conversion

    eg: if an email has chinese  characters in subject field(eg.漢字)

    when converted to msg using above code it will contain junk values like $B4A;z(B

    what am i making mistake here?

    Thanks,

    Thopulou


    thopulou Associate Software Engineer


    • Edited by D thopulou Thursday, April 5, 2018 4:44 PM
    Thursday, April 5, 2018 4:44 PM
  • Where does spRDOMail variable come from?

    You can create a standalone MSG fie using RDOSession.CreateMessageFromMsgFile / RDOMail.Import / RDOMail.Save

    Note the last Save, not SaveAs.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    Thursday, April 5, 2018 6:12 PM
  • Hi Dimitry,

    I created MSG file using RDOSession.CreateMessageFromMsgFile with FileFormat as mffUnicode =2 - Unicode MSG file

    And it is working perfectly.

    this is how i created spRDOMail

    IRDOMailPtr spRDOMail = NULL;
    hr = pSession->CreateMessageFromMsgFile(sDestinationFilePath, varType, varFormat, &spRDOMail);

    Thank you so much for helping me out.

    with Redemption work is made much easier.

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Monday, April 9, 2018 9:50 AM
  • Hi Dmitry,

    I had one test case left out.

    when i convert MIME content which has MSG as attachment with some Chinese characters(漢字) in Subject field

    it is not rendered properly. Instead of 漢字 it has ??

    i tested using both "HrRFC822MessageToIMessage" and import() and SaveAs().

    i can see this in MFCMAPI as well.

    MIME Content has this:

    Subject: =?iso-2022-jp?B?dGVzdCBmb3IgY2hpbmVzIGFuZCBtc2cgGyRCNEE7ehsoQg==?= which is in Japanese character Encoded in BASE64.

    So if MIME content has Japanese or Chinese characters do i have to pass Some Char set to HrRFC822MessageToIMessage and (import() and SaveAs()) so that it renders properly?

    image for converted MSG using HrRFC822MessageToIMessage and (import() and SaveAs())

    Image for MFCMAPI:


    thopulou Associate Software Engineer

    Friday, April 13, 2018 7:25 AM
  • Hi,

    I create MSG file using CreateMessageFromMsgFile with  FileFormat as mffUnicode =2 - Unicode MSG file

    IRDOMailPtr spRDOMail = NULL;
    hr = pSession->CreateMessageFromMsgFile(sDestinationFilePath, varType, varFormat, &spRDOMail);

    And i saved MSG as Unicode 

    hr = spRDOMail->Import(sMIMEFilePath, _variant_t(1024));
    if (S_OK == hr)
    {
    hr = spRDOMail->SaveAs(sDestinationFilePath, _variant_t((int)rdoSaveAsType::olMSGUnicode));
    }

    All Japanese Characters will be covered under _variant_t((int)rdoSaveAsType::olMSGUnicode?

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Friday, April 13, 2018 11:46 AM
  • Hi,

    My confusion is first Subject renders Properly but the Subject in the attachment is not rendered Properly

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Friday, April 13, 2018 12:14 PM
  • Do you have the problematic EML file? Can you send it (zipped) to dmitry (at) dimastr (dot) com?

    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    Friday, April 13, 2018 4:05 PM
  • Hi Dmitry,

    When I convert EML using IConverterSession() Subject Field is rendered Properly.

    I have mailed you the Zipped EML file to your ID

    Thank you so much.


    thopulou Associate Software Engineer


    • Edited by D thopulou Monday, April 16, 2018 6:52 AM
    Monday, April 16, 2018 6:51 AM
  • Hi Dmitry,

    The issue is resolved.

    I did a silly mistake by passing olRFC822 (1024) instead of olMsgUnicode (9) to import() function.

    i got to find out only after visiting your website 

    http://www.dimastr.com/redemption/RDOMail.htm#methods

    Thank you so much for your concerns.

    I am happy that it's working now 


    thopulou Associate Software Engineer

    Monday, April 16, 2018 11:03 AM
  • Hi Dmitry,

    Sorry i had to come back to this thread again for one concern.

    this is how i call import()

    VARIANT varFormat; varFormat.vt = VT_I4; varFormat.intVal = 2;

     IRDOMailPtr spRDOMail = NULL;
     hr = pSession->CreateMessageFromMsgFile(sDestinationFilePath, varType, varFormat, &spRDOMail);
     if (NULL != spRDOMail)
     {
     hr = spRDOMail->Import(sMIMEFilePath, _variant_t((int)rdoSaveAsType::olMSGUnicode));
      if (S_OK == hr)
     {
     hr = spRDOMail->SaveAs(sDestinationFilePath, _variant_t((int)rdoSaveAsType::olMSGUnicode));
      }
     }

    when i passed olMSGUnicode to the import() function it returns STG_E_FILEALREADYEXISTS

    the strange thing is that it works perfectly for two tests but after that it returns STG_E_FILEALREADYEXISTS and it never worked again.

    If i pass olRFC822_Redemption and olRFC822_Outlook to import() it does renders the "Subject Field" with Chinese charaters properly but the side effect is that it does not retain the HTML format of the of attached MSG file it changed to plain text.

    If i pass olRFC822 (1024) to import() HTML format is retained in the attached MSG  but "Subject Field" in MSG attachment with chinese characters is not rendered properly.

    eg: Instead of 漢字 it has ??

    Also tried in other way like this but it does not work either

    IRDOSessionPtr spSession = NULL;
    hr = CoCreateInstance(clsid, NULL, CLSCTX_INPROC_SERVER, _uuidof(IRDOSession), (LPVOID*)&spSession);
     if (hr != S_OK)
     {
        Log.Add(_T("Failed to create RDOSession"));
        break;
    }
    hr = spSession->Logon(_variant_t(L""), _variant_t(L""), _variant_t(VARIANT_FALSE), _variant_t(VARIANT_FALSE), _variant_t(NULL), _variant_t(VARIANT_TRUE));
    IRDOFolderPtr spDeletedItemsFolder = NULL;
    hr = spSession->GetDefaultFolder(rdoDefaultFolders::olFolderInbox, &spDeletedItemsFolder);
      if (NULL == spDeletedItemsFolder)
      {
      break;
       }
    IRDOItemsPtr spDeletedItems = NULL;
    hr = spDeletedItemsFolder->get_Items(&spDeletedItems);
      if (NULL == spDeletedItems)
      {
       break;
      }
    IRDOMailPtr spRDOMail = NULL;
    spDeletedItems->Add(_variant_t(olNoteItem), &spRDOMail);
      if (NULL == spRDOMail)
      {
        break;
      }

    if (NULL != spRDOMail)
     {
        hr = spRDOMail->Import(sMIMEFilePath, _variant_t((int)rdoSaveAsType::olMSGUnicode));
      if (S_OK == hr)
       {
      hr = spRDOMail->SaveAs(sDestinationFilePath, _variant_t((int)rdoSaveAsType::olMSGUnicode));
      }

    }

    So my Question is,is there other way to call import() function so that when i pass olMSGUnicode it will not return error STG_E_FILEALREADYEXISTS?

    i think if i pass olMSGUnicode to import properly it will work.

    is this the correct way of using Import()?

    hr = spRDOMail->Import(sMIMEFilePath, _variant_t((int)rdoSaveAsType::olMSGUnicode));

    if wrong how should i use?

    Why RDOMail.Import(..., olRfc822) is not rendering the Chinese Characters in the attached MSG Subject Field?

    I am helpless and i feel Redemption is best way to solve this issue.

    Please help me out in this. I tried googling but i couldn't get much information.

    Since this is high escalated issue i need to make sure it covers all the test cases.

    Sorry for long post i have to sure my issue is addressed properly and understandable.

    Thank you so much in Advance.




    thopulou Associate Software Engineer


    • Edited by D thopulou Wednesday, April 18, 2018 5:32 PM
    Wednesday, April 18, 2018 5:31 PM
  • Replace the call to SaveAs() with Save() - it will save the original MSG fiel specified in the call to CreateMessageFromMsgFile().

    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!


    Wednesday, April 18, 2018 5:55 PM
  • Hi Dmitry,

    I did as you said i called Save() instead of SaveAs() but its not working

                   varFormat is mffUnicode =2

                    IRDOMailPtr spRDOMail = NULL;
    hr = pSession->CreateMessageFromMsgFile(sDestinationFilePath, varType, varFormat, &spRDOMail);

    if (NULL != spRDOMail)
    {
    hr = spRDOMail->Import(sMIMEFilePath, _variant_t(1024));
    if (S_OK == hr)
    {
    hr = spRDOMail->Save();
    }

                      

    I am calling Import(..., olRfc822) with Rfc822

    But the Chinese characters in Subject Field of MSG attachment is not rendered properly and it has ??

    If i call Import(..., olMSGUnicode) with MSGUnicode it throws error STG_E_FILEALREADYEXISTS

    what am i missing?

    The Redemption version i am using is 5.14

    Thanks


    Thopulou Associate Software Engineer(iManage)



    • Edited by D thopulou Thursday, April 19, 2018 11:36 AM
    Thursday, April 19, 2018 11:34 AM
  • The current (and older) versions of Redemption have a bug - the embedded message attachment display name loses Unicode characters when an EML file is imported. Contact me at dmitry (at) dimastr (dot) com for an updated version.

    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.5 is now available!

    Friday, April 20, 2018 3:22 PM
  • Hi Dmitry,

    I have sent you updates to the above provided Email id.

    Thanks,

    Thopulou


    thopulou Associate Software Engineer

    Monday, April 23, 2018 11:53 AM