A Word About Google Cloud Platform
I am a big fan of using cloud platforms. In the case of Google's offering, I have been using it for test and production applications in a variety of scenarios since it was launched just for developers.
I have been using (and still am) using Google App Engine for mobile backends, web front ends, a backend that simply runs cron jobs and populates a database, a backend for chrome extensions, and even simply as a CDN.
When Google came up with Compute Engine, I was one of the early adopters, using it for hosting Node.js backends that needed customizations that PaaS could not offer (Heroku/Nodejitsu).
A lot has been said about weird design decisions on App Engine (a must read of you are considering app engine), but overall, when staring to develop a project on a low budget (or even on a high one), there are no better alternatives to date; fair pricing (free quotas can last for non-enterprise apps), great service (SSL, DB's, Memcached, Cron, DoS protection, Warmup Services and more), but the (arguably) the two most important things: Scalability (it's a big task, although other cloud platforms also offer that) and Monitoring - one of my favorite things about App Engine is that I always feel in control, the dashboards are accurate and easy to read.
When Things Go Wrong
But what happens when something weird happens? Google's infrastructure is said to be one of the most reliable and fast, but surely in the world of IT things sometimes go wrong.
In our case, we have had the "pleasure" of dealing with such an issue lately, and it scared the heck out of us.
At the company I work for, we are doing some research, sometimes trying out different apps in the market. In this case we have been using a Google App Engine Python backend as the backend for an Android App that we are testing out with users. Because the app is not actually an official company app (it's still in the research stage, the idea for the app came from one of the dev's in the team), we did not bother to purchase the expensive premier support package for the app ($400 a month).
Still, we are talking about an app that monetizes, and it has got several thousands of daily active users. Like every App Engine app that I do, billing was enabled. We definitely want the option to change the daily budget to handle spikes and maintain enough live instances that would guarantee a quick response time, even if this is just test app.
One bright day, we woke up to find that our backend starts to return 503 errors for all users: we have reached our quotas, and could not update the budget! We were losing money for evert minute that the app has been down, so naturally we were eager to get the issue fixed as fast as possible.
When we realized there's nothing we can technically do to solve the issue - the daily budget would just not get updated, and it's too late to shut down instances to reserve quota, we did everything we could to try and reach out to Google for help.
Reaching Out to Google To Solve The Problem
Our first step was to look for urgent support contacts. After doing some digging, we found out there is actually no way to reach Google in a reasonable time* when a problem comes up, unless you have previously prepared for this scenario by paying for an expensive gold-level support account.
If you are not paying for support beforehand, you can only try the following, using "official" channels:
1. Open up a billing support case here. I have tries this once and they never got back to me, so I was not surprised not to receive any response in our latest production issue.
That's basically it. This is the only official ways of handling an urgent production issue. The unofficial channels you should try are:
1. Try to call Google's support - won't work (you need a PIN - and to get a PIN, you need to register as premium customer - a process which make take up to a few days)
2. Try to subscribe to premium support and hope they will get back to you before the company shuts down your project and fires your entire team. If they get back to you, either be prepared to pay a lot of money, or beg them to fix the problem, even though you were actually abusing this communication channel just so that you could reach them.
3. Publish your plea for help on StackOverflow or Google Developers Forums, and hope that an engineer from the App Engine team will respond.
4. If you are lucky enough, you have some Google connections, and you can try to ask them to get App Engine team's attention in time.
For us, unofficial (4) came to the rescue, when I contacted a Google Developer Advocate I happen to follow on G+, and she was kind enough to help us by quickly forwarding the issue to GAE engineering that reseted our quotas after about 4 hours of being down, and fixed the App Engine budget update bug till the end of the day.
Conclusion
If you ever plan on using Google App Engine for any project that's more than a test project, you have got to purchase the rather expensive gold-level subscription. That is the only way you could reach Google with a guaranteed response time of up to 1 hour in case of a P1 issue.
This means that if you are a developer with an app that earns some money, but that app does not earn around $500 monthly, app engine might not pay off (the gold subscription starts at $400/month and you probably pay up to 100$ a month for app engine resources).
No comments:
Post a Comment