none
HttpListener returns 400 Bad Request when path contains %7F RRS feed

  • Question

  • I have set up a System.Net.HttpListener and it works fine in most cases. Recently I have run into problems with certain combinations of urlencoded characters in the path for example .../localhost/content/Default/%c2%83%ca%8b%cc%8d.JPG will trigger 400 Bad Request. %c2%83%ca%8b%cc%8d is the url encoding of the unicode character "ƒʋ̍". Does anyone have a clue why the HttpListener bother about the URL encoded characters? Shouldn't it be passed through to my callback so I can deal with the URL decoding in my code?

    Friday, December 11, 2015 1:03 PM

All replies

  • Hi Magnus,

    I've tried to reproduce your issue using the following code, but I am failed with error "the parameter is incorrect ". So I am failed to reproduce your issue. Could you provide us a simplified demo, we'll take a further look.

    Code snippet

      static void Main(string[] args)
            {
                SimpleListenerExample(new[] { "http://localhost:8080/index/", "http://localhost:8080/%c2%83%ca%8b%cc%8d.JPG/" });
            }
    
            // This example requires the System and System.Net namespaces.
            public static void SimpleListenerExample(string[] prefixes)
            {
                if (!HttpListener.IsSupported)
                {
                    Console.WriteLine("Windows XP SP2 or Server 2003 is required to use the HttpListener class.");
                    return;
                }
                // URI prefixes are required,
                // for example "http://contoso.com:8080/index/".
                if (prefixes == null || prefixes.Length == 0)
                    throw new ArgumentException("prefixes");
    
                // Create a listener.
                HttpListener listener = new HttpListener();
                // Add the prefixes.
                foreach (string s in prefixes)
                {
                    listener.Prefixes.Add(s);
                }
                listener.Start();
                Console.WriteLine("Listening...");
                // Note: The GetContext method blocks while waiting for a request. 
                HttpListenerContext context = listener.GetContext();
                HttpListenerRequest request = context.Request;
                // Obtain a response object.
                HttpListenerResponse response = context.Response;
                // Construct a response.
                string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
                byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
                // Get a response stream and write the response to it.
                response.ContentLength64 = buffer.Length;
                System.IO.Stream output = response.OutputStream;
                output.Write(buffer, 0, buffer.Length);
                // You must close the output stream.
                output.Close();
                listener.Stop();
            }

    >>%c2%83%ca%8b%cc%8d is the url encoding of the Unicode character "ƒʋ̍". Does anyone have a clue why the HttpListener bother about the URL encoded characters?

    In addition, from above message, I am curious about why you encoding of the unicode character "ƒʋ̍". As I tested with "http://localhost:8080/ƒʋ̍/"  successfully.

    If I misunderstood you, please feel free to let me know.

    Have a nice day!

    Kristin


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, December 14, 2015 8:49 AM
  • Here comes the example I use (actually same method SimpleListenerExample as you are using, but different prefix)

    public static void SimpleListenerExample(string[] prefixes)
    {
    if (!HttpListener.IsSupported)
    {
    Console.WriteLine("Windows XP SP2 or Server 2003 is required to use the HttpListener class.");
    return;
    }
    // URI prefixes are required,
    // for example "http://contoso.com:8080/index/".
    if (prefixes == null || prefixes.Length == 0)
    throw new ArgumentException("prefixes");

    // Create a listener.
    HttpListener listener = new HttpListener();
    // Add the prefixes.
    foreach (string s in prefixes)
    {
    listener.Prefixes.Add(s);
    }
    listener.Start();
    Console.WriteLine("Listening...");
    // Note: The GetContext method blocks while waiting for a request. 
    HttpListenerContext context = listener.GetContext();
    HttpListenerRequest request = context.Request;
    // Obtain a response object.
    HttpListenerResponse response = context.Response;
    // Construct a response.
    string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
    byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
    // Get a response stream and write the response to it.
    response.ContentLength64 = buffer.Length;
    System.IO.Stream output = response.OutputStream;
    output.Write(buffer, 0, buffer.Length);
    // You must close the output stream.
    output.Close();
    listener.Stop();
    }

    static void Main(string[] args)
    {
    SimpleListenerExample(new string[] { "http://+:8111/" });
    }

    Then I send a request "http://localhost:8111/content/Default/%c2%83%ca%8b%cc%8d.JPG" that will get a 400 - Bad Request. You can use the WebRequest sniplet below to try it out.

    static void Main(string[] args)
    {
    // Create a GET request
    // EXAMPLE 1 WORKING WebRequest request = WebRequest.Create("http://localhost:8111/content/Default/%7D.JPG");
    // EXAMPLE 2 NOT WORKING - 400 Bad Request - WebRequest request = WebRequest.Create("http://localhost:8111/content/Default/%7F.JPG");
    // EXAMPLE 3 NOT WORKING - 400 Bad Request
    WebRequest request = WebRequest.Create("http://localhost:8111/content/Default/%c2%83%ca%8b%cc%8d.JPG");

    request.Method = "GET";
    WebResponse response = request.GetResponse();
    Console.WriteLine(((HttpWebResponse)response).StatusDescription);
    using (Stream dataStream = response.GetResponseStream())
    {
    StreamReader reader = new StreamReader(dataStream);
    string responseFromServer = reader.ReadToEnd();
    Console.WriteLine(responseFromServer);
    reader.Close();
    }

    response.Close();
    }

    Monday, December 14, 2015 1:55 PM
  • Hi Magnus,

    Thanks for providing more useful information.

    Now I can reproduce your issue, It looks like these code considers %7f as an invalid character for an URL.

    Here is a page that describes HTML URL-encoding Reference

    You can see %7f corresponds an empty ASCII Value. I also tried %1f,%1e and so on, They are all the invaild characters for an URL. Please try to avoid these characters next time.

    Have a nice day!

    Kristin


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.



    • Edited by Kristin Xie Tuesday, December 15, 2015 3:01 AM
    • Proposed as answer by Kristin Xie Thursday, December 24, 2015 6:02 AM
    Tuesday, December 15, 2015 2:59 AM
  • Hi Kristin,

    I'm sorry but I don't agree that %7F, or as a matter of fact any %xx combination, is an invalid character in a URL. The decoded value (i.e. the control character DEL) is of course an illegal character to use in a URL, but its URL encoded representation is not (please check the IETF RFC1738 on Uniform Resource Locators).

    What I don't really understand is why the HttpListener even bothers about the path? It should be the work of the code using the listener to URL decode it an handle any "illegal" characters (e.g. by sending an appropriate error code). As it is right now the using code will not even get noticed that a request was received so it cannot do any actions to handle it.

    BR,

    Magnus

    Tuesday, December 15, 2015 10:54 AM
  • Maybe this can help

    I am not familiar with the HttpListener and a TesT Project would need me to enable that listening (i am currently way too lazy to enable that) so sorry that i don't try it myself, but you should be able to use the RawUri instead of the QueryString (the latter is parsed).

    The QueryString-Property just calls some method which uses the Raw uri as one of it's inputs.

    ^above link will show you the code of HttpListenerRequest, you just need to scroll to the QueryString property on somewhat of a third of the scroll view)


    Please be so kind to close your Threads when you found an answer, these Threads should help everyone with similar issues.
    You can close a Thread via the"Mark as Answer" link below posts. You can mark your own posts as answers if you were not helped out but found a solution, in such a case, please provide the answer.
    Happy coding
    PS: I assure everyone that I did not ever had the desire to offend anyone.

    Tuesday, December 15, 2015 2:22 PM
  • Hi MDeero, Thanks for your reply. I've already checked that post on the RawUri. Unfortunately that will not work since HttpListener.GetContext will never release because the lower layers of the .NET implementation sends the error response 400 to the client without even involving my code. So from my code's perspective, wating for the HttpListener.GetContext to release, it is as if no request at all has been made from the client. BR, Magnus
    Tuesday, December 15, 2015 8:12 PM
  • I thought it would be like that (but i figured... couldn't hurt to try).

    Have you also tried to set the Threads Culture (on which the HttpListener runs) to the UTF8-Encoding? (Also something maybe worth trying)

    Otherwise, i don#t think there is any Extension point that can be used, if you really need to make it work with the class at hand, you should check in the .NET sources to see whether you can override the problematic function in a class derived from the Listener. ( as far as i could see, you are out of luck, or the URL encoding is not the cause).

    Seems like you would have to use some other means for your HttpCommunication. (I would suggest something, but I have not idea what you need)

    Regards


    Please be so kind to close your Threads when you found an answer, these Threads should help everyone with similar issues.
    You can close a Thread via the"Mark as Answer" link below posts. You can mark your own posts as answers if you were not helped out but found a solution, in such a case, please provide the answer.
    Happy coding
    PS: I assure everyone that I did not ever had the desire to offend anyone.

    Wednesday, December 16, 2015 6:55 AM
  • Hi Kristin,

    I'm sorry but I don't agree that %7F, or as a matter of fact any %xx combination, is an invalid character in a URL. The decoded value (i.e. the control character DEL) is of course an illegal character to use in a URL, but its URL encoded representation is not (please check the IETF RFC1738 on Uniform Resource Locators).

    What I don't really understand is why the HttpListener even bothers about the path? It should be the work of the code using the listener to URL decode it an handle any "illegal" characters (e.g. by sending an appropriate error code). As it is right now the using code will not even get noticed that a request was received so it cannot do any actions to handle it.

    BR,

    Magnus

    @Magnus,

    Sorry to let you misunderstand.

    From my point of view, HttpListener cannot recognize the URL contains %7F. In order to prove that, I would suggest you test TcpListener to see if it supports or not.

    Best regards,

    Kristin


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, December 16, 2015 8:10 AM
  • @Magnus,

    What's the problem now? Per my understanding, HttpListener cannot recognize the URL contains %7F as my second reply. You also cantry to test %1f,%1e and so on, all of them are same.

    Here is a page that describes HTML URL-encoding Reference

    Best regards,

    Kristin


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, December 24, 2015 6:03 AM
  • I wonder which software layer rejects the %7F code. Is it the .NET Framework class library, the user-mode HTTP API, or the kernel-mode http.sys driver? I suspect it is http.sys, because:

    • If I browse to "http://localhost/%7F", then the "400 Bad Request" response has "Server: Microsoft-HTTPAPI/2.0".
    • If I browse to "http://localhost/%7E", then the "404 Not Found" response has "Server: Microsoft-IIS/7.5".

    If http.sys is the only thing blocking this request, then KB820129 documents an "AllowRestrictedChars" Registry value that might help:

    You can create the following DWORD values under the following registry key:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

    Registry key Default value Valid value range Registry key function WARNING code
    AllowRestrictedChars 0 Boolean If nonzero, Http.sys accepts hex-escaped chars in request URLs that decode to U+0000 – U+001F and U+007F – U+009F ranges. 0

    It also documents "EnableNonUTF8" but that one is already enabled by default.

    Tuesday, December 29, 2015 12:55 PM
  • Actually, %c2%83 decodes to U+0083, which is a control character that may mean NO BREAK HERE. "ƒ" is U+0192 and should be encoded as %c6%92.

    In Windows-1252, U+0192 would be encoded as 0x83; but if you were using Windows-1252 in the URI, then it would be just %83 rather than %c2%83.

    The request indeed seems incorrectly encoded, if it was intended to mean the Unicode characters "ƒʋ̍". However, URIs can contain arbitrary percent-encoded bytes that do not have to mean characters in any encoding. I think it is reasonable to expect HttpListener to pass through requests that contain such URIs.

    I set the "AllowRestrictedChars" Registry value and restarted the http driver and dependent services, and a request to http://localhost/%7F now apparently reaches IIS; I get "404 Not Found" rather than "400 Bad Request". Unfortunately, http://localhost/%00 still results in "400 Bad Request" and "Server: Microsoft-HTTPAPI/2.0". http://localhost/?%00 is okay though, even without AllowRestrictedChars.

    So you may have two ways around the problem:

    (a) Set AllowRestrictedChars=1 and hope that the input doesn't contain any %00 bytes.

    (b) Change the higher-level protocol so that the percent-encoded bytes are in the query rather than in the path. I suppose that would prevent you from registering URI prefixes specifically for them, though.

    • Edited by ranta Tuesday, December 29, 2015 4:20 PM tried with AllowRestrictedChars
    Tuesday, December 29, 2015 3:55 PM
  • With the following line of code:

    Console.WriteLine(System.Web.HttpUtility.UrlEncode("ƒʋ̍"));

    You get:

    %c6%92%ca%8b%cc%8d

    So, when in doubt, check with your UrlEncode()/UrlDecode().

    Wednesday, December 30, 2015 2:39 AM
    Answerer