CDN Caching Problems - The "Vary: User-Agent" Header

If you are using a CDN for serving your static or dynamic content, you better be aware of the HTTP vary: user-agent header. 

The Vary header is a simple (and sometimes useful) directive that can lead to a lot of trouble with CDN's. The idea of this header is rather straightforward - when you serve the file - you might want to map the same URL (for instance, a JavaScript file), to different contents. In other words, the content may "vary" by parameters such as user-agent (the navigator the user is on, etc.).

This is especially true if your site supports both Mobile and Desktop, on which case, sending the "Vary by: User Agent" HTTP header is actually recommended by big players such as Google. However, big players in the CDN market, like Akamai, by default, will not cache any content with the "vary: user-agent" header.

The reason is probably because there are many user-agents out there, and it does not seem feasible for the CDN to store copies of all of the cached content in thousands of copies (the amount of user agent's is ever getting larger). As a result, the CDN might not cache the response for files with the vary by: user agent header.

If you accidentally send the header and you are using a CDN, you can expect to experience the following nasty problems:

1. Your server load will increase dramatically, since requests will hit the origin instead of getting the content from the CDN.

2. Your users will experience slower response times because your origin will have to recomptue the file's content on every request, and the Geo-based content delivery will not be activated (the request will have to reach your origin server anyway).

3. Your CDN bill might explode (the CDN's usually charge by bandwidth, and they will be serving a lot more content if they are not able to cache it).

The solution for this problem can be:

1. If you don't really need the vary-by: user-agent header, don't send it! (.NET guys - if you use Microsoft BundleConfig to bundle and minify your CSS/JS, you will get this header automatically. You will need to get the System.Web.Optimization source code, remove the header from there, and recompile the DLL).

2. You can try to tell the CDN to ignore the vary-by: user agent header (in Akamai it is possible by configuration).

3. If your site requires distinguishing between mobile and non-mobile navigators and you are using Akamai, you can a use the "is mobile" flag which handles this in other ways.

Happy caching :-)