none
Why I got 503 Error with HTTP Post when data available, but when no data, HTTP Post requests went though RRS feed

  • Question

  • Hello:
    I have a rather strange issue.  I have a C# .Net Core program, I can login to a trading web site, and get some csrf tokens, then with those csrf tokens, I should be able to HTTP post some data to a specific web page.
    Ie. HTTPS POST https://myweb.com/markets/page=1
    It is actually an order placing web site, just like buy/sell stocks.
    When I see there are some offers available via a web browser, then I want to place an order by HTTP Post using the token and application/x-www-form-urlencoded form data, something like this:
    csrftoken=XYZ&side=0&price=1.10&market=123&runner=456&type=1&price_formatted=1.10&amount=1.00
    Whenever I see there are some offers available via web browser, I got 503 Service Temporary Unavailable error.
    But whenever there is no offers available via web browser, the HTTP Post request went through, but I got the following Json reply:
    {
      "canceled": true,
      "matching_id": false,
      "amount": 0
    }
    I sent the HTTP Post data by a Rest Client Insomnia 6.6.2, so I can see the server reply and save the reply in the HTTP session.

    The following is Timeline records from Insomnia Version 6.6.2 for my HTTP Post Request:
    * Preparing request to https://myweb.com/market/betslip/
    * Using libcurl/7.57.0-DEV OpenSSL/1.0.2o zlib/1.2.11 libssh2/1.7.0_DEV
    * Current time is 2019-09-08T15:30:13.610Z
    * Disable timeout
    * Enable automatic URL encoding
    * Enable SSL validation
    * Enable cookie sending with jar of 14 cookies
    * Found bundle for host fairlay.com: 0x1fba9df0620 [can pipeline]
    * Re-using existing connection! (#0) with host fairlay.com
    * Connected to fairlay.com (104.28.26.236) port 443 (#0)

    > POST /market/betslip/ HTTP/1.1
    > Host: fairlay.com
    > Accept: text/html,application/xhtml,*/*
    > Body: text
    > Content-Type: application/x-www-form-urlencoded
    > Accept-Encoding: gzip, deflate
    > Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
    > authority: fairlay.com
    > origin: https://myweb.com
    > Referer: https://myweb.com/markets/live/?sort=close_date
    > User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3835.0 Safari/537.36
    > X-Requested-With: XMLHttpRequest
    > Cookie: csrftoken=YSJYVItqYjKsAPaekHOCLIBD5mabAw2v8G4Bw7SJcpLNqhJJvoxDN5GUar6Ob9pO; _gid=GA1.2.1876159809.1567948880; _fbp=fb.1.1567948879900.1716178483; _gat_UA-41965734-1=1; sessionid=n6rp26vlgys2onzycwrhv4yvx1ovfx9t; _ga=GA1.2.539566304.1567948880; __cfduid=da870fff4f0fbff353bb3e6e838082b001567948878; _hjIncludedInSample=1
    > x-csrf-token: nxogvGr5pcPO7mgSUkmV1hr2hSa3HjmaxlJT65QoDiQ9XOPn515W3EwjmX6GiWJt
    > Content-Length: 174
    csrftoken=nxogvGr5pcPO7mgSUkmV1hr2hSa3HjmaxlJT65QoDiQ9XOPn515W3EwjmX6GiWJt&side=0&price=2.01&market=20679428&runner=43888464&type=2&price_formatted=2.01&amount=1.00
    * upload completely sent off: 174 out of 174 bytes
    < HTTP/1.1 200 OK
    < Date: Sun, 08 Sep 2019 15:30:13 GMT
    < Content-Type: application/json
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < X-Frame-Options: SAMEORIGIN
    < Vary: Cookie

    * cookie size: name/val 9 + 32 bytes
    * cookie size: name/val 7 + 29 bytes
    * cookie size: name/val 8 + 0 bytes
    * cookie size: name/val 7 + 7 bytes
    * cookie size: name/val 4 + 1 bytes
    * cookie size: name/val 8 + 3 bytes

    < Set-Cookie: sessionid=n6rp26vlgys2onzycwrhv4yvx1ovfx9t; expires=Sun, 22 Sep 2019 15:30:13 GMT; HttpOnly; Max-Age=1209600; Path=/; SameSite=Lax
    < Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
    < Server: cloudflare
    < CF-RAY: 5131f65f3823cc42-ZRH
    < Content-Encoding: gzip

    The returned Json reply is like this:
    {
      "canceled": true,
      "matching_id": false,
      "amount": 0
    }

    I want to know in this case, is my HTTP Post request is correct in format or not, but why when there are some offers available, I can never send HTTP Post request due to 503 error, but only when there is no offers available, my HTTP Post requests went through, yet got canceled reply.
    Thanks,
    Friday, September 20, 2019 8:09 PM

All replies

  • 503 has nothing to do with the data you send. 5xx errors are server side. 503 means the server is not available and could be a throttling or busy issue. They are not available (or purposefully failing your call). 4xx errors indicate your request is invalid and that by fixing your request the server should process it.

    In some cases a server may generate a 5xx error if you are not passing sufficient security information in order to throw off hackers. That may be the case here. It seems like the server is using ASP.NET and therefore you likely got some form data and cookies when you logged in. The cookie(s) and form data need to be transmitted in subsequent calls otherwise the server will fail the request. Cookies are generally created at first login and are then passed from then on. If you're using WebClient then you can set up a cookie container that will capture the cookies returned from the server and passed to subsequent requests. Google for WebClient and cookie container. Ultimately it just involves you creating a derived WebClient that overrides the function to create a web request and hook up a cookie container that is persisted for the life of the client. The cookies are handled automatically after that provided you use the same client each time. HttpClient indirectly does a similar thing but it requires more work to get it set up. Still you can go that route if you need to.

    As for the form data (e.g. viewstate) it is a little harder and depends upon the site. In general you send a GET request to the server to load a page. That page returns back as part of the data a couple of values needed on POST requests including __VIEWSTATE, the viewstate key and possibly some other related values. That data must be passed back to any POST requests in order for the server to validate. Doing a GET request to the URL you POST to first will give you that. You then need to locate the values and extract them. Here's a blog article on one approach. If you are unclear what values are being sent back to the server on a POST then use the browser's F12 tools to look at the data sent to the server for the POST data. In the request header will be the cookies and form data that is being sent.

    When you are building up the form data to send in the POST you need to add these values back in. Personally I use a dictionary to store the key-value pairs and then I wrote a simple helper function to convert to a NameValueCollection. That is what is needed by WebClient's UploadValues methods require and is how you can post data. That method is responsible for converting the data to form data. You'll need to set the content-type but beyond that it should just work. You can verify this by monitoring the network call. 


    Michael Taylor http://www.michaeltaylorp3.net

    Monday, September 23, 2019 2:33 PM
    Moderator