I had a little problem last week. I built a very small website – really just a one pager with a single API – whacked it up on an Azure website and then promptly had a quarter of a million people visit it in three days. Uh, bugger?
Ok, what’s behind the website is a little more clever than that but the real point is that I didn’t expect it to take off so quickly. This leads to interesting problems; for example I wasn’t aware that the “Shared” Azure website I was using is limited to 20,000 page views a day. I did eight times that in one day. It broke. Well actually, Azure just turned the site off and fortunately someone told me within a few minutes and I just moved the slider over one to make it a “Standard” site. Crisis over.
The real issue though was bandwidth. Now this is a very well optimised site to begin with plus it’s very simple but still, you serve enough requests and your usage starts looking like this:
That’s just over 23GB in 24 hours from a very small site. Despite how well optimised the site was for speed, I realised that it wasn’t well optimised for egress data on the service I was paying for. I realised, for example, that I’d served up 15GB of jQuery alone – that’s minified and HTTP compressed too. Crikey.
There are a whole different set of challenges involved when optimising for large traffic volumes, let me walk you through how I tackled it.
Let me start by recapping on how the site was structured to begin with. If it hadn’t had been so well optimised from the outset I might have had a serious cost problem when things got big! Many sites these days are loading up a couple of MB just to get into the front page; if that had been my case one week into this project, I’d be looking at about a terabyte of data hitting my wallet which even at Azure’s excellent prices is many hundreds of dollars per month.
Moving on, the site uses Bootstrap, jQuery and Font Awesome, all of which were bundled and minified to ensure one request for CSS and one for JS. The CSS was loaded early and the JS was loaded late, the latter even bundling the Google Analytics code into the external script file to ensure it was cached.
Speaking of cache, all static resources were given a far-reaching cache expiration of one year. Of course they were all also served gzipped which is the default position for Azure anyway.
In terms of images, there were initially only 5 and they were very well optimised SVGs (just the logos of various pwned companies) which kept them down to a total of 12.1KB. Unfortunately I simply couldn’t get Azure to compress them though, either over HTTP or by saving them as SVGZ (it just didn’t want to serve the mime type despite all the usual attempts to have it recognised).
All unnecessary response headers were turned off. There’s no ASP.NET version or MVC version or powered by header. The only non-essential stuff is an XFO header and a session affinity cookie and server header added by Azure.
The API returns a string array of pwned sites in JSON. The aforementioned affinity cookie is redundant but other than that it’s a very, very tight HTTP chat.
Frankly, all of this is pretty much the starting point for any website regardless of expected scale because it’s easily done and has a great, positive impact on performance. But it wasn’t enough.
Prioritising scale objectives
Let me drill into what I meant earlier on when talking about optimising for speed versus optimising for egress data as whilst there’s a lot of cross over, there are also some key points of difference:
- Speed: This is all about the user experience – how can I get the site loaded as fast as possible so that it’s usable (this is a key word that I’ll come back to a lot).
- Data volume: Here I needed to keep the amount of outbound data on the service I was paying for at an absolute minimum.
Let me give you an example of where these objectives may be mutually exclusive: In order to reduce the number of HTTP connections the browser needed to make, I used ASP.NET bundling and minification features to roll jQuery into the one bundle along with my custom JS and serve it all up in the one request. That’s great, but it meant I was paying to serve jQuery, If I had used a CDN such as Google’s instead, yes, there’d be an extra HTTP request but the data wouldn’t be hitting my account. Of course the counter-argument is that a CDN offers the advantage of reduced latency if it can get the content closer to the browser to begin with. I’ll come back to that.