Azure CDN bandwidth abuse by malicious bandwidth vampire requests RRS feed

  • Question

  • Hello,

    So here's the situation. Let's say that I am building a picture gallery website and I would like to use Azure CDN to deliver my content for me. In the backend, Azure CDN will pull content from an Azure storage account. CDN is fast and powerful, but it seems though it can be a little unsecured in terms of preventing someone from being able to pull content in very large quantities and thus leaving a user with a huge bandwidth bill. Let me demonstrate what I mean.

    So last night I decided to write a simple console app that would download a simple image from my future to be picture gallery website, in a for loop, the code is below:

    namespace RefererSpoofer
        class Program
            static void Main(string[] args)
                HttpWebRequest myHttpWebRequest = null;
                HttpWebResponse myHttpWebResponse = null;
                for (int x = 0; x < 1000; x++)
                    string myUri = "http://myazurecdnendpoint.azureedge.net/mystoragecontainer/DSC00580_1536x1152.jpg";
                    myHttpWebRequest = (HttpWebRequest) WebRequest.Create(myUri);
                    myHttpWebRequest.Referer = "www.mywebsite.com";
                    myHttpWebResponse = (HttpWebResponse) myHttpWebRequest.GetResponse();
                    Stream response = myHttpWebResponse.GetResponseStream();
                    StreamReader streamReader = new StreamReader(response);
                    Image image = Image.FromStream(streamReader.BaseStream);
                    image.Save(string.Format("D:\\Downloads\\image{0}.Jpeg", x), ImageFormat.Jpeg);

    This console application makes 1000 super-fast continuous requests to an image file that is hosted on my Azure CDN endpoint, and saves them to 'D:\Downloads' folder on my PC, with each filename corresponding to the for{} loop iteration, i.e. image1.jpeg, image2.jpeg, etc.

    So what just happened? In about 1 minute of time, I have cost myself 140MB of bandwidth. With this being a Premium CDN, priced at $0.17/GB, let's do the math together:  0.14GB * 60minutes * 24hours * 30days * 0.17cents/GB = $1028.16 of bandwidth costs just if someone (a competitor for example) wanted to make a single request for a single image for a duration of the month to jeopardize my website. I think you guys can see where I am going with this...my website will have thousands of images, in hi-res, btw, the image that I was using in this example was a mere 140KB in size. These types of requests can come from anonymous proxies, etc.

    So the question that I have is: How can Azure protect a customer against such abuse? Obviously the customer can't be stuck paying $5,000, $20,000 for bandwidth resulting from malicious requests.

    Now Azure Premium CDN has an advanced Rules Engine, that can filter out requests based on Refer, and respond with a 403 error in case the Refer doesn't match your website. But, the Refer can be faked, as I did in the above code sample, and CDN still allows the requests to be served, I tested with a Refer spoof. This sucks, a lot of people use Refer to prevent 'hotlinking', but what does it matter if it can be faked with just a line of code?

    A couple of ideas that I've had in regards to prevent such abuse and huge bandwidth cost for customers:

    1. When a request comes for content to the CDN, CDN could make a call to the client server passing in a) IP address of the user b) the CDN Uri requested. And then the client server would check how many times this Uri was requested from this particular IP, and if the client logic sees that it was requested let's say 100 times over the past minute, then obviously this would signal abuse, because browsers cache images, while malicious requests don't. So the client machine would simply reply 'false' to serving the content for this particular request. This would not be a perfect solution since the additional callback to client infrastructure would cause a small delay, yet it's definitely better than being potentially stuck with a bill that looks like the amount of money you have saved up in your savings account. 

    2. A better solution. Built in a limit for number of times a file can be served over CDN within a particular time frame, per ip. For example, in the example of the image file above, if one could configure the CDN to serve no more than let's say 50 image requests / IP / within 10 minute time frame. If the abuse was detected, then CDN could, for a time defined by a customer a) serve a 403 for a particular abused uri. or b) server 403 for all uri's if the request is coming from an abuser IP. All times / options should be left configurable to the customer. This would definitely help. There's no callback here which saves time. The downside is that CDN will have to keep track of Uri/IP address/ Hit count.

    Which solutions would NOT work:

    1. Signed URL's won't work because the signature query string parameter would be different every time and browsers would constantly make requests for data, effectively wiping out browser cache for images.

    2. Having a SAS access signature for blob would not work either because a) Uri is different every time b) There's no limit on how many times you can request a blob once SAS is granted. So abuse scenario is still possible.

    3. Check your logs and simply ban by IP. I was testing this type of abuse via anonymous proxy yesterday and it worked like a charm. Switched IPs in a matter of seconds and continued abuse (of my own content) for testing purposes. So this is out as well, unless you have a nanny to monitor your logs.

    Solutions that can work, but are not feasible:

    1. Filter requests on your web server. Sure, this would be the best way to control the issue and track the number of requests / IP, and simply not serve the content when abuse is detected. But then you loose humongous for not delivering your content over super-fast, proximity-to-client optimized CDN. Besides the fact that your servers will be slowed down a lot by serving out large byte content such as images.

    3. Simply bite the bullet and not worry about it. Well...then you know that the pothole that will take your wheel out is just down the road, so no, it's not a comfortable feeling. By the way, let's say that this does happen to a client, how would Azure handle such an issue? Would Azure give a full refund for such abuse if customer can prove that this was indeed what's happened? would Azure give a partial refund? or any refund amount at all?

    With all of the above said, the Premium CDN offering from Azure with custom Rules Engine might offer a solution somewhere in there, but with a very poor documentation and a lack of examples one only would have to guess how to properly protect yourself, so that's why I am writing this post.

    As far as I know there's also a WAF security rule set coming for Azure CDN, could anyone comment around what time frame it is planned to come out? and would it have any features to detect & deflect the scenario described in this post?

    I would appreciate if this would also be brought to the Azure CDN developers attention so that a) maybe they can comment and b) build the countermeasures once they are aware of the issue

    Any suggestions are appreciated, I am very open minded on the issue.

    Thank you for reading.

    Thursday, February 18, 2016 5:38 PM

All replies

  • Thanks for your very thorough and detailed feedback. Overall the most secure and controlled solution is one where you require authentication for each request. When content is made publicly accessible to everyone, one is exposed to risk of large bandwidth usage as a result of either a malicious attack on ones site or just because a sites content became very popular. The CDN is designed to handle massive and fast usage spikes worldwide. This inherently adds additional risk for customers versus directly using an origin such as Azure Storage which doesn't have the same scale capabilities. There are a few solutions to consider some that are available now and others that we are working on providing later this year. 

    Solutions available now:

    1. You can set a spending limit on your Azure account to limit impact of large bandwidth usage - see https://azure.microsoft.com/en-us/pricing/spending-limits/ for additional details on this.

    2. If you don't expect to have large # of requests for your content (e.g. 100's of concurrent requests) consider not using Azure CDN and just having your customers directly access content from your origin. Overall Azure CDN is optimized for large concurrent requests for content. If you have a lot of long tail static content that isn't accessed frequently in many cases you will have better performance with requests going directly to your origin.

    Solutions that will be available later this year:

    1. Token authentication capabilities via the Premium SKU. Token authentication will allow one to require that specific or all requests must be authenticated. An encrypted token must be provided by clients and defines minimum requirements that must be met - e.g. client IP, allowed/denied countries, expiration time for token, allowed hosts, allowed referrers, etc.

    2. Real time alerts via the Premium SKU. Real time alerts will allow you to receive notification either via email or HTTP POST when specific thresholds have been reached - e.g. bandwidth, # of requests, HTTP errors, etc.

    3. Full CDN WAF offering. This may be the ideal solution for you as it will allow one to define legitimate / illegitimate traffic via threat detection measures, Access controls (e.g. IP address, country, etc.), and global settings. You will have the ability to either generate alerts or automatically block threats - e.g. throttle bandwidth.

    Sunday, February 28, 2016 10:51 AM
  • Hi Anton,

    Thank you for you reply. I was just about to post this on CDN feedback forum, but I've noticed you replied here. So my thoughts after reading your reply are below:

    In regards to the points you have mentioned for the options that are available now:

    1. The spending limit seems more like of the last line of defense for the account. While we surely will be having that, it's not exactly how we would want to prevent someone from just sucking up the bandwidth if they wanted to (and us having to pay that bill). Also once the limit is reached it looks like it would shut down the services, this can't happen in production environment.

    2. That's was one of the solutions I described that would not be feasible, having to serve images from web servers directly will probably kill the web servers. Having to serve images out of dedicated servers would likely double the hosting costs and maintenance costs for the customer.

    In regards to the options that would be available later this year:

    1.  Are these singed URLs? I don't think these would work because they would wipe out the browser's possibility to cache the images. How could a browser cache the image if the uri is always different with a unique token parameter at the end of it?

    2. Certainly receiving alerts could be an option if the bandwidth goes through the roof. But one has to realize that banning simply by IP in that case is ineffective, anonymous proxies could swap IPs in seconds, I have tested this scenario. Everything else about the request can be faked, including headers, referrer, browser version, etc. I spoofed referrer in the code that I have provided above, and azure CDN was no longer blocking these requests with a 403 error. So the alerts are somewhat useless unless you really can do something about what's happening.

    3. Sounds a little more interesting, specifically the bandwidth throttling feature. But how would it be configured? What really needs to happen is the ability for CDN to limit number of requests from a particular IP, for a particular CDN uri, for a particular time frame. So an attacker for example can't request a particular image file / video file, from their IP, more than X times, in Y time frame. Would WAF be able to do that? I looked into Amazon's WAF but didn't see anything.

    Looking at what's been done with WAF now, it only seems to provide a very basic functionality against attacks, because it's just so easy to fake anything about the request programmatically, one would be busy just maintaining the rules all the time. What this all really feels like is having a door made out of cardboard on your house that gives you a false sense of security, it's really a joke, that's what it is. Btw, right now it takes about 3 hours for the rule to kick-in with the Premium SKU..but customers need response in seconds or minutes in the worst case scenario. We need a better product that can handle non-standard scenarios, not just "typical" cases, something that can hold its ground in advanced cases as well. With the scenario that I am describing here, it's so obvious to one that a scenario like this can happen (and probably happens all the time) when your content is exposed publicly. As a customer, let me tell you, we love the speed of CDN but we also want our security. This really shouldn't be that hard to do and can be implemented with a dictionary look ups that are O(1) in efficiency in the CDN back-end.

    I am sharing this because this really does matter, and is a real issue, and no one has stepped up to this yet. Files are larges nowadays, bandwidth is expensive, hackers are merciless, and competition is stiff. We need solutions to address such issues.

    Tuesday, March 1, 2016 6:44 PM
  • I find it really hard to justify the use of the CDN now that I realized the problem with malicious scenarios such as the ones AzureCloudDev pointed out.

    Who can really afford to take the risk of infinite size bill for the very likely scenario of malicious behaviour?

    Having spending limit for sure removes the possibility of infinite bill. However it doesn't sound like very dynamic solution when shit hits the fan. Maybe some of the upcoming features Anton mentioned will help with the problem. However with the new features managing the CDN might become everything but easy, making it no-go option for smaller teams and startups.

    So there might not be any single obvious solution for the problem.

    I am wondering how many developers are taking the risk of using the CDN without really understanding how vulnerable their bill really is. The CDN is expected to be the public user facing interface, contrary to for example Azure Storage which is mainly used to build secure backends.

    Sunday, March 20, 2016 11:34 PM