none
EWS 1.2, Exchange 2010 SP2/SP3 - Appointment message body gets wrecked on update RRS feed

  • Question

  • We're seeing a very irritating issue with EWS and Exchange when updating the message body on an Appointment. To head it off, it does not appear to be the bug detailed here:

    http://blogs.msdn.com/b/dhruvkh/archive/2012/05/18/sending-appointments-from-ews-apps-where-did-my-html-go.aspx

    It happens on both 2010 SP2 and SP3. We're creating Appointments with some HTML formatting the message body. Nothing fancy, just a few tables and a little bit of inline styling (since a css style header disappears into a black hole when you try to use it as part of the Appointment message body). We're able to make a nice-looking message body that goes out in the meeting requests. Well, with the exception of downlevel text at the top, but there doesn't seem to be much we can do about that short of a double-tap save.

    The problem comes into play if we need to update the message body on the Appointment. All of this is automated, so the user cannot directly edit it. If we make any changes to the message body, it completely wrecks the HTML. It's still an HTML-formatted message, but much of the styling is lost and it looks awful. This happens without fail when we alter the message body of an Appointment with any considerable level of HTML formatting. BodyType is HTML, of course. I've done some digging on the best body algorithm and some of the MAPI properties related to it. Some interesting bits of note:

    • When the Appointment is first saved, PidTagRtfInSync is true. PidTagNativeBody is 2, which indicates an RTF message. PidTagBody is present. PidTagRtfCompressed is present. However, PidTagBodyHtml is not present. After updating the Appointment with a message body edit, PidTagBodyHtml is present, but now PidTagRtfInSync is false and PidTagNativeBody is 3.
    • If PidTagBodyHtml is set directly as an extended property instead of setting the message body via the EWS Appointment object, then the formatting is wrecked right out of the gate. At least it's consistent that way...
    • Attempting to set PidTagNativeBody manually had no effect.
    • When viewing the PR_BODY_HTML contents between a good and bad message, the good one will contain the transformed HTML. It doesn't look much like what we had initially handed to it, but gets changed in the conversion process. The bad one is much closer to what we wrote in actual HTML, but has still had much of the styling removed. It's as if creating the Appointment does the RTF conversion, but any subsequent changes use a completely different code path serverside and there does not appear to be any way around it short of even more guess and check with EWS and Exchange chewing up HTML. This is complicated by being unable to find any documentation that details how and what gets changed, discarded, etc in the process.
    • This will always affect requests, but only seems to affect Appointments in the calendar of attendees if the body includes the downlevel text. Appointments without the downlevel text appear to survive updates with their formatting intact. Appointments with downlevel text get their formatting wrecked with updates.

    Has anyone run into this before? It was painful enough when we found that our previously-used email HTML formatter wouldn't work with EWS and everything had to be inlined, but now it seems that there are even more inexplicable problems to get around. Is there a way to update the message body without this happening?


    Wednesday, December 11, 2013 12:24 AM

All replies

  • Just when you think you have it fixed...

    I thought I had a way around this. Instead of setting the MessageBody in the EWS Appointment object, I directly set the PidTagBodyHtml extended property (tag 0x1013). This allowed me to use the full CSS header without losing it! I fired off a few tests. Sure enough, OWA showed all formatting with no issue! I was feeling pretty excited, thinking I finally had a solution to this problem. Then I checked it in the desktop client, which is Outlook 2013 in this case. Sorry, but no dice. Very basic format, just about everything lost. I exported the item with MFCMAPI to get the following:

    • PR_BODY shows the usual basic formatting, which appears to be what I'm seeing in Outlook on the desktop. Bold characters are preserved. The tables are nothing but bold headers and badly-spaced tabs.
    • PR_BODY_HTML shows exactly what was put into it. This seems accurate. This is what I'm seeing in OWA.
    • PR_RTF_COMPRESSED shows the very same HTML, albeit wrapped up in all of the extra RTF formatting/tags.
    • PR_RTF_IN_SYNC is false.
    • The downlevel text is being inserted in the correct place. It is placed in the body and does not appear to be the cause of any of this. Still annoying that it cannot be disabled.
    • MFCMAPI exports the "WrapCompressedRTFEx best body" function as well. This comes through with ulStreamFlags = 2 and szStreamFlags of MAPI_NATIVE_BODY_TYPE_HTML. It is identical to the PR_BODY_HTML.

    Okay, so that's odd. All of that points to displaying proper HTML. However, if I look at the item directly in MFCMAPI, the table shows the following:

    • PR_BODY - Not enough memory error. Opening the property shows the full body anyway.
    • PR_RTF_COMPRESSED - Not enough memory error. Opening the property shows the full body anyway.
    • PR_BODY_HTML - Comes through correctly as a binary property.
    • PR_RTF_IN_SYNC is false

    Looking at this, it seems that the initial request does NOT retrieve the PR_NATIVE_BODY_INFO, even though I can retrieve it with zero problems via EWS. Then it throws out of memory errors on both the RTF and plain text body properties, even though they still come through properly and even display in MFCMAPI. The size of the HTML body is just under 2k, the RTF is just over 2k, and the plain text body is just under 500 bytes. I would think an ROP response would have more than enough headroom for all of those. What the heck gives here? Is there a configuration or setting serverside that should be adjusted to deal with this?


    Thursday, December 12, 2013 7:46 PM
  • I don't know the answer to your problem but have you had a read of the Best Body Retrieval Algorithm document http://msdn.microsoft.com/en-us/library/cc463905(v=exchg.80).aspx That should outline the rules that the server and client should follow when returning the body to a user. How each client then implements this is down to the client/developer eg OWA in 2013 use EWS etc but every clients different eg Outlook,OWA,ActiveSync,BES   

    Cheers
    Glen

    Friday, December 13, 2013 2:41 AM
  • I've pored through that document a number of times while going over this, which is why I figure that Outlook 2010/2013 desktop clients are somehow not retrieving the PR_NATIVE_BODY_INFO tag and are moving ahead to the conditionals demonstrated by the table in the document, where the PR_BODY and PR_RTF_COMPRESSED properties throw out of memory errors, the PR_BODY_HTML property comes through. On further inspection, this should display the HTML body according to the conditional chain specified in the document! How is it that Outlook 2010/2013, which, according to the document, should follow this algorithm, are not following it? Even MFCMAPI indicates that the chosen body type should be HTML in this case.

    This leaves me with a threefold problem: Why is the PR_NATIVE_BODY_INFO property not being retrieved, and why do the other two body types throw an out of memory error? The best body algorithm tries to retrieve all five within a single ROP, which should have plenty of space for a few kilobytes worth of string/binary data. Then Outlook apparently isn't even following the specification that it's supposed to follow in the first place.

    The thing is, when using the MessageBody property in EWS, you should be able to see the same behavior. Make an Appointment with a nice little HTML body. Use a few inline style tags that don't get eaten. Make sure you have at least one other person getting an invitation. Save. Take a look in the desktop client. It should look okay as you're getting the RTF body. Then update the Appointment with a new/altered MessageBody. Unless something is very screwy on my end, you'll see the same behavior: The Appointment in Outlook is now using the plain text body and it looks quite ugly.

    It's also a bit odd that the plain text body shows some minor formatting that is preserved from the original. Sometimes you'll still see bold text, sometimes you may see a bit of background shading in the table, but it's definitely not HTML (cannot view source) and it's definitely not RTF (as sync is false and it does find the plain text body).

    In addition, I dug around in the ROP specs, but couldn't find any concrete information on why the body properties throw out of memory errors. I would assume that this specific ROP request is intended to be very small and that there isn't sufficient space in the response buffer, but I can find no way to confirm that.
    Friday, December 13, 2013 5:04 PM
  • I've confirmed identical behavior with EWS 2.0, although that was really more of a hail mary than anything.

    Is it possible that anyone else can confirm this behavior with 2010 SP3 or any other version?

    Monday, December 16, 2013 5:44 PM
  • I got frustrated enough that I implemented Microsoft's RTF compression algorithm (I'm using EWS Java 1.2, so it doesn't come with that capability, and yes, as stated in the previous post, I've confirmed that the issue is identical with EWS 2.0 in C#). Well, the decompression portion of it, anyway. I've been able to decompress the binary RTF body property to have a look inside. I've created an Appointment by setting the Body (as HTML) with "good" HTML that looks decent when the Appointment is first saved. I then copied out the RTF data, updated it with the SAME BODY (only one word was changed in a table heading, displayed text, not HTML), saved again, and ran a diff against what I got the first time.

    They are completely different. They contain the same basic text, but the formatting is massively changed. The "good" RTF file will have actual RTF formatting with little to no HTML references at all. It also has "generator Microsoft Exchange Server formatConverter converted from html" in the header of the RTF file. The "bad" one is an unholy mashup of RTF wrapping up HTML tags. There is only indication of HTML conversion is a \fromhtml1 right after the ansi declaration. They're nothing alike!

    I cannot come to any conclusion other than this being a glaring bug in Exchange. Behavior should not differ in the RTF compression algorithm between save and update. The RTF compression algorithm serverside should not differ when the same content is placed in the same object (or type of object) in the same way, period. However, it's shockingly obvious that this is not the case.

    EDIT: Fixed an error in my RTF decompressor that didn't take the circular dictionary into account. I can now verify that they are being properly decompressed and that Word can handle both RTF files properly. The breakage issue still remains, though.

    Thursday, December 26, 2013 11:49 PM
  • I can 100% confirm this behavior with C#, EWSMA 2.0 and Exchange 2010 Service Pack 3 (with Update Rollup 4).

    Create (Appointment) Item works fine. That means it keeps all formatting the way it should do it in the messageBody and any Update of the messageBody destroys the formatting.

    I can also confirm most of the additional observations. 

    This is a real issue for us, and we really need this fixed.

    thanks in advance

    Gregor 


    Thursday, February 20, 2014 10:53 AM
  • I 've opened a support case on this (and referenced this very good detailed analysis of the issue in the hope its okay for you). Dunno if its okay to post the Case ID right here? I'll share it when its usefull to solve the issue. :)

    hth

    Gregor Stefka

    Thursday, February 20, 2014 11:32 AM
  • Thanks for the info, Gregor. It's good to know that I'm not crazy, heh. I posted a more detailed thread with some support responses here:

    http://social.msdn.microsoft.com/Forums/en-US/52cbb92b-91c8-42e4-946c-f318d824069b/msoxrtfex-and-msoxbbody-exchange-behavior-does-not-match-the-documentation?forum=os_exchangeprotocols#55af4bde-c7aa-4f8c-87e2-247c1cb0cb43

    I'd be very interested to know what you are able to learn on this issue should anything come from the support ticket. For the time being, I've fully implemented the RTF compression/decompression algorithm. It seems that setting the RTF body directly should work with valid RTF data and compression. I say should because I cannot be certain that there would not be side effects, but I've been able to take a "good" compressed RTF body from one Appointment, decompress it, recompress it (which differs a bit from the original compressed content, but is still valid), and insert it into an Appointment with a "bad" body. It displays correctly in Outlook 2010/2013 desktop and OWA. Still, it's impossible to say if this would have unintended side effects, so I still can't be certain that it's a viable long-term solution.

    Friday, February 21, 2014 4:57 PM
  • Access to the RTF Itembodies through EWSMA would solve the issue for us and seems easy to implement for Microsoft?  Accessing the bodies via MAPI is more than cumbersome for us to implement. 

    Did you ever verfied this issue against a Exchange Server 2013? Its wouldnt be  a solution but a perspective in the long run. We are still a few weeks away from our Exchange Server 2013 Test enviroment to test it ourself....

    Gregor Stefka

    PS: No reaction on the Case so far.

    Monday, February 24, 2014 2:40 PM
  • I don't currently have access to a 2013 server, only 2010 SP2/SP3. You can retrieve and set the compressed RTF Message Body as an Extended Property (tag 0x1009, binary type), but like I said, I cannot say if this will have unintended effects, and you'd need to have an RTF body ready to go.


    Monday, February 24, 2014 5:19 PM
  • Microsoft Support - after a short email dialog - finally answered: 

    Hi Gregor,

     

    I’ve looked into this and I can reproduce the issue as well. Now given that the new HTML body does not get converted to a compressed RTF stream and that the PR_RTF_COMPRESSED property does not get populated, Outlook will always default to the plain text body as the appointment logic implemented in the Outlook client does not support the PR_HTML property.

     

    So yes, I acknowledge that the problem is there but I’m afraid there is nothing we can do about it at the moment and given your service level. Solving this issue would require some advanced debugging and that kind service is reserved to our premier customers.

     

    Thank you or your understanding.

    Thursday, February 27, 2014 2:35 PM
  • So, what now? Any premier customers around?
    Thursday, February 27, 2014 2:36 PM
  • That response doesn't match what I've seen. The PR_RTF_COMPRESSED property DOES get populated. You can see it plainly if you grab it as an extended property and decompress it. The new data from the HTML body you passed to it is there, but the formatting gets completely wrecked. Look at a "good" RTF body (with Notepad or something similar so you can see everything) from a freshly-created Appointment (with attendees and invitations sent), then update the body and check the RTF body again. The RTF formatting is completely different. This might not be a problem in some cases, but the Outlook desktop client will use the RTF body whenever available despite what the Best Body algorithm might say. This is explained a bit more in the thread I linked earlier.

    Thursday, February 27, 2014 5:05 PM
  •  

    It seems like this will not be fixed or changed by Microsoft in any way. So, we take the performance penalty and access the other itembodies trough extended properties (tag) and deal on the RTF body on our own. It works for us now as intended in the first place (with firstclass properties only.

    thank you  for sharing all the details  that helped us to work around this issue.

    Gregor Stefka

    Monday, March 3, 2014 9:43 AM
  • Ugh... hopefully I can ask you gentleman how to solve my issue.

    I recently implemented code to allow our users to turn-on HTML-encoding for when our Sync'ing app goes to add/update items for them into Exchange.

    If thev'ye turned-on HTML formatting, when I create a CalendarItemType's (in C# based off the EWSProxies.EWS.Itemtype) the Body.Value is being set with a string which contains very minimal HTML tags... just <br>'s instead of "\r\n" and for URL link values we are embedding into the text, I'm encapsulating those within an <a href> 

    Example: <a href="http://TheWebSite.com/EventDisplay.aspx?uri=1234">Event Display Link</a>

    My problem is... when I retrieve an item from Exchange... there is a huge amount of extra HTML encoding it wraps around everything! 

    By inspecting in the debugger... I've discerned that :

    For Appts... the main bunch of text that I inserted in an item when originally creating it is contained just within the <div> tag.

    For Tasks... the main bunch of text that I inserted in an item when originally creating it is contained just within the <body> tag (and on these there is no <div> tag created).

    I was able to use the HTML Agility Pack and load this Body.Value string into the enhanced HtmlDocument object.  I first attempt to parse first for the <div> nodes contents. If the <div> is not-null that's an Appt.  If the <div> is null, I instead parse out the <body> nodes contents, and those are Tasks.

    I noticed for Appts :

    For every <br> tag I had in my original text.. it appends two consecutive "\r\n" 's after it!  No problem I can just do a Replace("\r\n", "")  and zap those. But then the <a href> I have in there... it encapsulates it with a <font> node where it sets a color for the link and also an underline <u> node.  I can handle the <u> nodes easy enough with a Replace("<u>","") and a Replace("</u>", "");  But what about the damn <font> node?  It's in the format of <font color=\"blue\"></font> and I'm sure the \"blue"\ part could be different for every user.

    I noticed for Tasks :

    It prepends a "\r\n" just inside the begininng<body> tag. Each of the original <br> tags gets an "\r\n" appended behind it.  and for the TaskType for the <a href> node IT DOES NOT encapsulate it with the <font> and <u> nodes as it does for the Appt.

    How can I stop it from changing the Body content when retrieving it?  Will it not change anything if I go the extra mile and make sure to wrap everything with an <html> and <body> node when initially creating the items?

    Any and all help on this issue is sorely appreciated.

    Thank You,

    -G


    • Edited by TheGDizzo Monday, June 16, 2014 7:44 PM
    Monday, June 16, 2014 7:31 PM