none
POP protocol email 64 bit encoding issue RRS feed

  • Question

  • I am reading emails using POP protocol.
    The email encoding is base64.
    After I read the email, I forwarded the email using SMTP.
    The receiving client is using Outlook 2007 to view the email.
    The weird thing is in my Outlook 2007 (10.0.6562.5003) SP2 MSO, all the characters show up normal, but in another machine that uses the same Outlook 2007 like my machine, some characters do not show up correctly.
    For example:
    Country’s Reliance
    should read Country’s Reliance. In my Outlook 2007, this shows up correctly, but not in another machine's Outlook 2007.
    Is it correct that it has something to do with the setting of the Outlook ? Or, maybe the setting of the mail server the receives the email ?
    What setting shall I check ?
    Thank you

    Wednesday, February 15, 2012 5:43 PM

All replies

  • That looks like UTF8 encoding.

    Where exactly and how are the messages displayed?

    Does the message specify the code page?

    Are both machines running under the same user locales?


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!


    Wednesday, February 15, 2012 6:05 PM
  • Both machines view the message in Outlook 2007.
    What do you mean by code page ?
    How do I check the user locales ?

    Wednesday, February 15, 2012 9:15 PM
  • But how do you get the message to Outlook?

    Do you programmatically access a POP3 server and copy the messages to Outlook or let Outlook handle that through its POP3/SMTP provider?

    As for the locale, check Control Panel | Region and Language.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Wednesday, February 15, 2012 9:38 PM
  • Thru a VB6 program I access a POP3 mail server (using a POP3.cls), then read the raw body of the email.
    If the raw body of the email contains the word base64 (which it does in this case), then the program decode the base64 using POP3.cls (Coerce 4 base 64 encoded bytes into 3 decoded bytes by converting 4, 6 bit values (0 to 63) into 3, 8 bit values. Transform the 8 bit value into its scii character equivalent).
    Then, the program sends that decoded strings (poSendMail.Message = sRaw where sRaw is the decoded strings) to another email (like my email and another person's email) using SMTP.
    I and the other person open the email using Outlook 2007.

    In terms of the locale, which tab shall I compare, the Regional Options, Languages or Advanced ?
    I am pretty sure the other machine using the same Locale setting, since we are both in the US and in Central Time.

    Thank you for your help.

    Wednesday, February 15, 2012 9:52 PM
  • The raw data will be 8 bit, do you perform any codepage manipulation? Do you preserve the message header so that Outlook khows how to convert the incoming 8 bit data into Unicode using the appropriate code page?


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Wednesday, February 15, 2012 10:48 PM
  • BTW, if you need to convert/extract data from a MIME message, you have no choice but to use a real MIME parser that knows how to handle the codepages.

    I have used Lumisoft MIME parser in the past.

    If you want to parse MIME messages and import them to Outlook, you can use Redemption - it allows to import MIME messages using RDOMail.Import(..., olRfc822) - http://www.dimastr.com/redemption/RDOMail.htm. Even if you do not want to import a message into one of Outlook folders, you can create a temporary standalone MSG file using RDOSession.CreateMessageFromMSGFile and import the MIME message into that MSG file.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Wednesday, February 15, 2012 11:18 PM
  • Thanks for the reply.
    >The raw data will be 8 bit, do you perform any codepage manipulation?
    I use the VB6 POP3.cls that I download from the internet to get the rawtext of the email (sRaw = POP3.RawEmailText(intLoop)), then since it is base 64, I call POP3.DecodeBase64String to get the decoded string of the base64. Is that considered codepage manipulation ?

    >Do you preserve the message header so that Outlook khows how to convert the incoming 8 bit data into Unicode using the >appropriate code page ?
    The original email in base64 has this at the top
    X-Vipre-Scanned: 011671GA012E110116111-FEA
    Received: (qmail 4794 invoked by uid 78); 10 Feb 2012 08:55:54 -0000
    Received: from unknown (HELO abc) (0.0.0.0)
      by 0 with SMTP; 10 Feb 2012 08:55:54 -0000
    Return-Path: <a@abc.com>
    Received: from [0.0.0.0] ([0.0.0.0:1234] helo=SJC-ABCD-01-abc.com)
     by cm-mr22 (envelope-from <a@abc.com>)
     (ecelerity 0.0.0.0 r(37554)) with ESMTP
     id C9/0C-01779-59BD43F4; Fri, 10 Feb 2012 03:55:53 -0500
    Received: from abc.com (0.0.0.0) by
     SJC-ABCD-01-abc.com (0.0.0.0) with Microsoft SMTP Server
     (TLS) id 0.0.0.0; Fri, 10 Feb 2012 00:55:44 -0800
    Received: from ABC.com ([0.0.0.0]) by
     abc.com ([0.0.0.0]) with mapi; Fri, 10 Feb 2012
     14:25:32 +0530
    From: aa <aa@abc.com>
    To: bbb <bbb@abc.com>
    CC: ccc <ccc@abc.com>
    Date: Fri, 10 Feb 2012 14:25:30 +0530
    Subject: My subject
     Round-Up: February 10, 2012
    Thread-Topic: My Topic
     Round-Up: February 10, 2012
    Thread-Index: Aczn0KHGrlCQdBE2Tf2fOUoCeapf/QAAK7ng
    Message-ID: <ABCDEDF@abc.com>
    Accept-Language: en-US
    Content-Language: en-US
    X-MS-Has-Attach: yes
    X-MS-TNEF-Correlator:
    acceptlanguage: en-US
    Content-Type: multipart/related;
     boundary="_004_132Q24_";
     type="multipart/alternative"
    MIME-Version: 1.0
    X-Auto-Response-Suppress: DR, OOF, AutoReply

    --_004_132Q24_
    Content-Type: multipart/alternative;
     boundary="_000_132Q24_"

    Is that the header ?
    The VB6 program then search for "Content-Type: text/html;" and then "Content-Transfer-Encoding:", and if it finds the word base64, it then decodes the part of the email starting from there.
    This is what's being sent out in the email using SMTP:
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
    <HTML xmlns:fo = "<HEAD">http://www.w3.org/1999/XSL/Format"><HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170"></HEAD>
    <BODY>
    :
    </BODY></HTML>

    Thursday, February 16, 2012 2:58 PM
  • VB6 strings are always ANSI, so you should be Ok.

    The problem is that there is no charset information in the Content-Type header, you need something like:

    Content-Type: text/plain; charset="utf-8"


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Thursday, February 16, 2012 3:06 PM
  • At the top of the email that's being sent out, there is this header:
    <HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170">
    </HEAD>
    <BODY>

    Is that what you mean ?

    Thursday, February 16, 2012 3:50 PM
  • That is an HTML attribute. MIME itself needs to know how to process the characters in the upper half of the ASCII table.

    In your case Outlook MIME converter just uses the current code page to convert to Unicode, that is why you get 3 weird looking characters instead of treating that 3 byte sequence as a UTF8 control word plus character.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Thursday, February 16, 2012 3:54 PM
  • So, I will need to include the following in the email that I send out: Content-Type: text/html; charset="utf-8" ?
    How about Content-Transfer-Encoding: ?

    Where shall I put that in the email that I am sending out ?
    Currently, this is how the email looks like (this is sRaw in poSendMail.Message = sRaw)
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
    <HTML xmlns:fo = "<HEAD">http://www.w3.org/1999/XSL/Format"><HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170"></HEAD>
    <BODY>
    :
    </BODY>
    </HTML>

    Thursday, February 16, 2012 4:06 PM
  • Yes for content-type.

    Content-Transfer-Encoding is also needed, but it tells the MIME parser how to decode the data (7bit/8bit/base64/etc), *not* how to interpret it.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Thursday, February 16, 2012 4:40 PM
  • Where shall I put the content-type and Content-Transfer-Encoding   in the email that I am sending out ?
    Currently, this is how the email looks like (this is sRaw in poSendMail.Message = sRaw)

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
    <HTML xmlns:fo = "<HEAD">http://www.w3.org/1999/XSL/Format"><HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170"></HEAD>
    <BODY>
    :
    </BODY>
    </HTML>

    Thank you again for your help

    Thursday, February 16, 2012 4:56 PM
  • It needs to be in each MIME part header (or the main MIME header if there is only one part).

    I still dont' understand why you recreate the message in such a crude way... Why do you do that?


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Thursday, February 16, 2012 5:01 PM
  • The MIME header exist in the email that I read, but since I only start decoding from after Content-Type: text/html; charset="utf-8"
    Content-Transfer-Encoding: base64, there is no MIME header in the email that I am sending.
    Is that the problem ?
    But, if that's the problem, then why the email that I received in my machine looks OK, but the exact same email doesn't look OK on another email ?
    I recreate the message because I have to replace some cid: with another .png file and I need to add some words at the bottom of the email.

    Thursday, February 16, 2012 6:06 PM
  • Again, you cannot just pick a MIME part without the content (charset in your case). If there is no header, it is up to the particular parser and gazillion environment variables (current code page, system locale, Outlook/Windows version, etc.) how the data will be interpreted.

    One way or another, you need a real MIME parser.

    Why do you recreate the header? Can't you just replace the data without removing the headers?


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Thursday, February 16, 2012 6:15 PM
  • > Can't you just replace the data without removing the headers?
    I did that, but now the word " MIME-Version: 1.0 Content-Type: text/html;charset="UTF-8" Content -Transfer - Encoding: quoted -printable" appears at the top of my email in Outlook, and actually now my machine has the same problem with the other machine where words such as "Country's Reliance" shows up as "Country’s Reliance".

    This is how I start my email with the MIME header, is it wrong ?

    MIME-Version: 1.0
    Content-Type: text/html;charset="UTF-8"
    Content -Transfer - Encoding: quoted -printable
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
    <HTML xmlns:fo = "<HEAD">http://www.w3.org/1999/XSL/Format"><HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170"></HEAD>
    <BODY>
    :
    </BODY>

    Thursday, February 16, 2012 7:49 PM
  • There must be an empty line between the MAPI header and the actual payload:

    MIME-Version: 1.0
    Content-Type: text/html;charset="UTF-8"
    Content -Transfer - Encoding: quoted -printable

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
    <HTML xmlns:fo = "<HEAD">http://www.w3.org/1999/XSL/Format"><HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170"></HEAD>
    <BODY>
    :
    </BODY>


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!


    Thursday, February 16, 2012 8:24 PM
  • Thanks.
    I added the extra empty line, but still have the same problem, where the word " MIME-Version: 1.0 Content-Type: text/html;charset="UTF-8" Content -Transfer - Encoding: quoted -printable" appears at the top of my email in Outlook, and words such as "Country's Reliance" shows up as "Country’s Reliance". Do I need to change the charset to something other than UTF-8, and do I need to change the encoding to something othe rthan quoted-printable ?

    MIME-Version: 1.0
    Content-Type: text/html;charset="UTF-8"
    Content -Transfer - Encoding: quoted -printable

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
    <HTML xmlns:fo = "<HEAD">http://www.w3.org/1999/XSL/Format"><HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170"></HEAD>
    <BODY>

    Thursday, February 16, 2012 9:37 PM
  • Are you sure there is no empty line at the top?

    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Thursday, February 16, 2012 9:40 PM
  • You mean before "MIME-Version: 1.0" is there an empty line ?
    There is no empty line before "MIME-Version: 1.0".
    "MIME-Version: 1.0" is the first line in the email that I sent.
    Is it the wrong charset or encoding maybe ?
    Thursday, February 16, 2012 10:16 PM
  • No, the fact that Outlook sees the header as part of the body means that the basic MIME structure is wrong.


    Dmitry Streblechenko (MVP)
    http://www.dimastr.com/redemption
    Redemption - what the Outlook
    Object Model should have been
    Version 5.2 is now available!

    Thursday, February 16, 2012 10:22 PM
  • OK. I am not familiar with MIME, can you see anything wrong in this ?

    MIME-Version: 1.0
    Content-Type: text/html;charset="UTF-8"
    Content -Transfer - Encoding: quoted -printable

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
    <HTML xmlns:fo = "<HEAD">http://www.w3.org/1999/XSL/Format"><HEAD>
    <META content="text/html; charset=UTF-8" http-equiv=Content-Type>
    <META name=GENERATOR content="MSHTML 8.00.6001.19170"></HEAD>
    <BODY>
    :
    </BODY>

    Thursday, February 16, 2012 10:35 PM