none
Exchange Web Services not working with ISO-8859-1 (Office 365) RRS feed

  • Question

  • Disclaimer:

    As far as I can tell this is ONLY an issue with Office 365.  I have not had any issues with on-premise Exchange servers.  The issue started 2-3 weeks ago.  I was hoping a recent incident fix that Microsoft deployed (EX30045) would fix it and was advised by MS reps to wait and see if it fixed it; unfortunately, the issue still exists after the incident fix.  I am reporting it now in the hopes that it can be fixed (or perhaps another solution recommended).

    Issue:

    When making a request to EWS I am using ISO-8859-1 in the XML declaration, like this:

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <soap:Envelope xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/' xmlns:t='http://schemas.microsoft.com/exchange/services/2006/types'>
        <soap:Header>
            <t:RequestServerVersion Version="Exchange2010" />
        </soap:Header>
        <soap:Body>
            <ResolveNames xmlns='http://schemas.microsoft.com/exchange/services/2006/messages' xmlns:t='http://schemas.microsoft.com/exchange/services/2006/types' ReturnFullContactData='true'>
                <UnresolvedEntry>sean</UnresolvedEntry>
            </ResolveNames>
        </soap:Body>
    </soap:Envelope>

    The response from EWS is: 400 Bad Request

    I noticed that when I change the encoding to UTF-8 or remove the XML declaration all-together I do not get a 400 error.

    <?xml version="1.0" encoding="UTF-8"?>
    <soap:Envelope xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema' xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/' xmlns:t='http://schemas.microsoft.com/exchange/services/2006/types'>
        <soap:Header>
            <t:RequestServerVersion Version="Exchange2010" />
        </soap:Header>
        <soap:Body>
            <ResolveNames xmlns='http://schemas.microsoft.com/exchange/services/2006/messages' xmlns:t='http://schemas.microsoft.com/exchange/services/2006/types' ReturnFullContactData='true'>
                <UnresolvedEntry>sean</UnresolvedEntry>
            </ResolveNames>
        </soap:Body>
    </soap:Envelope>

    My app has been using ISO-8859-1 in the XML declaration for about 2 years now without issues.  It continues to work fine with on-premise Exchange servers.  It just started getting these 400 errors from Office 365 EWS in the past 2-3 weeks.  I had 80+ users report the issue to me... so I know it's not just me!

    The reason I added ISO-8859-1 was to support special characters such as letters with accents or double dots , for example: ü. 

    In the past, I found that special characters will fail UTF-8 schema validation. 

    For example, this request where I am creating a new event that has a ü in the subject and body will fail schema validation: 

    <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
        <soap:Header>
            <t:RequestServerVersion Version="Exchange2010" />
        </soap:Header>
        <soap:Body>
            <CreateItem xmlns="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" SendMeetingInvitations="SendToNone">
                <SavedItemFolderId>
                    <t:FolderId Id="AAMkAGVhNjkwMjc3LTIzNDMtNGIxNi04ZGFmLWRmZGZkOTkwYzkyMAAuAAAAAADaExiwUPUESr2GeD8IP+grAQAzFC3cUB82TpFnYg9jtCVeAACcuPTzAAA=" />
                </SavedItemFolderId>
                <Items>
                    <t:CalendarItem xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
                        <Subject><![CDATA[Test character ü]]></Subject>
                        <Sensitivity>Normal</Sensitivity>
                        <Body BodyType="HTML"><![CDATA[I took a look at the line and position and found this character:<br/>ü]]></Body>
                        <ReminderIsSet>true</ReminderIsSet>
                        <ReminderMinutesBeforeStart>10</ReminderMinutesBeforeStart>
                        <Start>2015-09-29T12:00:00Z</Start>
                        <End>2015-09-29T13:00:00Z</End>
                        <IsAllDayEvent>false</IsAllDayEvent>
                        <LegacyFreeBusyStatus>Busy</LegacyFreeBusyStatus>
                        <Location><![CDATA[]]></Location>
                        <StartTimeZone Id="Eastern Standard Time"></StartTimeZone>
                        <EndTimeZone Id="Eastern Standard Time"></EndTimeZone>
                    </t:CalendarItem>
                </Items>
            </CreateItem>
        </soap:Body>
    </soap:Envelope>

    The response from EWS is:

    <?xml version="1.0" encoding="utf-8"?>
    <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
        <s:Body>
            <s:Fault>
                <faultcode xmlns:a="http://schemas.microsoft.com/exchange/services/2006/types">a:ErrorSchemaValidation</faultcode>
                <faultstring xml:lang="en-US">The request failed schema validation: '�' contains invalid UTF8 bytes.</faultstring>
                <detail>
                    <e:ResponseCode xmlns:e="http://schemas.microsoft.com/exchange/services/2006/errors">ErrorSchemaValidation</e:ResponseCode>
                    <e:Message xmlns:e="http://schemas.microsoft.com/exchange/services/2006/errors">The request failed schema validation.</e:Message>
                    <t:MessageXml xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
                        <t:LineNumber>0</t:LineNumber>
                        <t:LinePosition>0</t:LinePosition>
                        <t:Violation>'�' contains invalid UTF8 bytes.</t:Violation>
                    </t:MessageXml>
                </detail>
            </s:Fault>
        </s:Body>
    </s:Envelope>

    I am requesting that MS either:

    1. Fix the issue so that ISO-8859-1 works again on Office 365 EWS OR ... 

    2. Please recommend how developers should send special characters like ü to EWS.

    Thank you


    • Edited by Sean McKeon Tuesday, September 29, 2015 10:46 AM
    Tuesday, September 29, 2015 10:42 AM

Answers

  • To close the loop:

    After I figured out how to open an incident I was able to get some help from a MS Engineer.
    He was able to recreate the issue with Office 365.  He confirmed that ISO-8859-1 did not work with Office 365 but did work with on-premise servers.  So the issue is confirmed.

    However, he also asked why I needed to use ISO-8859-1.  He was able to use special characters just fine with UTF-8.  I went back and reviewed my code and did A LOT of testing.  Eventually I figured it out.  Posting here in case others happen to read this and have a similar issue.

    My app is using HttpClient to connect to Exchange Web Services (SoapUI is also using HttpClient).
    It looks like I need to specify UTF-8 when creating the post body (which I was not doing).
    I was specifying UTF-8 as the "Content-Type" header, but that was not enough.
    When you set the post entity you need to make sure it's in UTF-8 format, otherwise special characters will show up as question marks "?" or the sync may fail due to invalid schema.

    Before:
    HttpClient client = builder.build();
    URL url = new URL(postUrl);
    HttpPost post = new HttpPost(url.toURI());
    post.setEntity(new StringEntity(postBody));
    ...

    After:
    HttpClient client = builder.build();
    URL url = new URL(postUrl);
    HttpPost post = new HttpPost(url.toURI());
    post.setEntity(new StringEntity(postBody, "UTF-8"));
    ...

    After making the change I am able to use a ü character.  I verified that it appears in OWA just fine.

    So it looks like I have my solution.  I can use UTF-8 instead of ISO-8859-1.  The ISO-8859-1 issue remains for Office 365, but I am no longer impacted.
    • Marked as answer by Sean McKeon Tuesday, October 13, 2015 1:14 PM
    Tuesday, October 13, 2015 1:13 PM