Which cloud platforms deliver reliable service?

Somebody needs to stress-test the cloud if it’s to be taken seriously as an IT resource for “real” business. Fortunately, a few Australian researchers have taken on the task, and their stress tests “have revealed that the infrastructure-on-demand services offered by Amazon, Google, and Microsoft suffer from regular performance and availability issues.”

The researchers created stress tests that simulated 2,000 concurrent users connected to applications hosted on the Amazon EC2, Google AppEngine, and Microsoft Azure cloud computing platforms. As always with these types of tests, there is some good news and some bad news:

• The good news is that the testing did confirm that these cloud computing platforms were able to scale as needed and responded dynamically to an increasing demand load. In essence, when the demand increased, the cloud computing systems dynamically provided the additional capacity required to support the demand.
• The bad news is that performance varied greatly. Indeed, according to the researchers, response times during the tests differed by a factor of 20, depending on the time of day the testing occurred. This is consistent with my experience and is perhaps due to the fact that multitenant, on-demand infrastructures are, well, multitenant, thus serving many users simultaneously, the number of which rises and falls during the day.

The testing also demonstrated that the various cloud computing platforms were individually suited for specific types of applications. For example, Google’s AppEngine appeared to work best for simple applications or tasks that take less than 30 seconds. Its monitoring tools were more developer-oriented and not such a great fit for business users.

I’m encouraged by the results of this testing, considering that the larger concern around cloud computing is its ability to dynamically scale, which is a core reason to leverage cloud computing in the first place. The ability to increase processing on demand, without having to go out and purchase waves of hardware and software upgrades, is really why we’re looking at cloud computing.

On the other hand, the performance issues are discouraging — but not a surprise. Anybody who uses the Web understands that remote sites have huge variations in response times, and in many instances there are any number of links in the chain outside the remote servers themselves, such as network saturation, that could cause the latency. However, the researchers stated they took that into account in their testing methodology.

I hope more testing is done and results reported. They are all good data points that should be considered along with the potential value that cloud computing can bring to your enterprise.