Blog entries

20Nov

More on Amazon’s SAS70 Type II

Amazon hasn’t been forthcoming since my last post on their control and control objectives, which is disappointing, but expected.  I still believe that transparency here is more important than security through obscurity.  Hiding the controls and control objectives doesn’t provide much in the way of particular security benefits, although I’m certain some will argue that it does.  Consider however, that while the SAS70 controls would tell what is being audited, that doesn’t necessarily translate to all of the controls in place.

Regardless, a bit more light has been shed on Amazon’s controls and measures in their recent security webinar.  You can access it here.

At a high level, CJ Moses, who presents the webinar talks to the core areas they covered in the control objectives, which are:

  1. Security organization
  2. Amazon employee lifecycle
  3. Logical security
  4. Physical security
  5. Environmental safeguards
  6. Change management
  7. Data integrity, availability, and redundancy
  8. Incident handling

This looks pretty reasonable at a high level.  Of course, it would be nice to see the actual controls and objectives, but at least they are covering the appropriate areas of security.  I do notice that there isn’t much around perimeter or related security.  I’m guessing they are trying to gloss over the AWS distributed firewall.  It would be nice if someone besides Amazon was vetting the way this was built.  They appear to consider it a piece of core intellectual property despite the fact it would be trivial to reproduce.  I’m not exactly certain why.

Post to Twitter

Comments
16Nov

Why is Amazon’s SAS70 Audit Bogus?

At first glance it seems like Amazon’s recent announcement of a successful SAS70 audit is grounds for celebration[1]. Certainly it has met with fanfare on Twitter and blogs.

Unfortunately, a SAS70 audit isn’t what most people think it is. Worse yet, Amazon’s reluctance to provide details of the audit provides a false sense of security with no tangible benefits.

Let me explain.

Understanding the SAS70 Audit
The SAS70 is a methodology for performing an audit, not the audit rules themselves. The SAS70 can prove whatever you decide it needs to prove. From taking the garbage out to turning the lights on.

From Wikipedia:

SAS 70 defines the professional standards used by a service auditor
to assess the internal controls of a service organization and issue a service auditor’s report.

Here’s how it works.

For a SAS70, you must specify a series of “controls” and “control objectives”. Like it sounds, you are asserting that a given ‘control’ meets a goal or objective. An example of a control might be the ‘new user creation process’ or a ‘firewall’. An example of a control objective might be the following[2]:

The new user creation process MUST guarantee that a user’s password
is at least 8 characters long and composed of a mix of at least one uppercase,
one lowercase, and one numerical character.

Once all of the control objectives are in place an outside auditor, like Deloitte & Touche, comes in and verifies that you are compliant with the stated control objectives over a period of time. If it is a Type 1 audit the period is 3 days. If it is a Type 2 the period is 6 months.

Now here’s the rub: Who decides what the control objectives are? An outside agency? A regulatory body?

None of the above. The company being audited decides and can make the control objectives anything they like. Here’s a SAS70 FAQ response on the topic right from the SAS70.com website.

Again, the SAS70 is just an auditing framework. Why then do so many think it’s useful?

Background on the SAS70 Audit
The SAS70 comes out of the financial industry and is a relatively generic framework for that reason. The financial industry has tons of different regulatory requirements that vary from state to state and country to country. Moreover, within the financial industry these kinds of audits are undertaken all of the time, the parties involved know what they are testing for, and how to negotiate it.

For example, a large bank might outsource work to a secondary institution and have a desire to see that institution provide proof they are following certain guidelines or regulations. A good example is the Bank Secrecy Act. The large bank in this case knows what the BSA requires and how to evaluate the secondary institution’s SAS70. This knowledge allows them to assess secondary institution’s level of compliance with the BSA. At the same time, the secondary institution is familiar with what its large partners will require and sets up its annual Type 2 to cover the ‘usual suspects’ of controls and control objectives.

So how did we get here?

Hosting Companies and the SAS70
In recent years as financial institutions began to outsource they required that various hosting (and other) businesses perform the audit as well. Unlike their usual partners it hasn’t been clear what hosters need to be compliant with. Because of this most folks have simply done these SAS70s as simple Type 1s that are one-offs. This allowed the hosters to keep their costs down while allowing the bank to outsource and the hosters to generate revenue.

Here’s the problem: Cloud computing is ushering in whole new ways of delivering IT services.

It demands greater transparency than ever, especially when it comes to security. If the average person doesn’t understand the SAS70 and if you don’t provide your control objectives so that others can vet the objectives you sold then you are creating a false sense of security.

You could have one control objective that simply says: “we must keep the power in the data center on” and successfully pass by fulfilling that over 3 days or 6 months.

The Need For A Cloud Security Standard
There are a couple of security and IT standards that can be used as the basis for a good SAS70 audit. For example there is CoBIT and the ISO27002 (formerly ISO17799). There are probably others I’m unfamiliar with. Unfortunately, most of these standards really focus on the Enterprise and not on a multi-tenant public cloud or hosting companies, who have some issues specific to their particular business models and architectures.

So, even if Amazon used one of these, it’s still not good enough for them to keep their controls and control objectives hidden from public view. How are we to be certain that they are sufficient? [3]

Summary
Until there is a security standard for running a cloud then SAS70 audits with unpublished controls and control objectives like the recent AMZN announcement are simply smoke and mirrors. They provide little or no real assurance to the average consumer of the AWS public cloud and serve only to provide a false sense of security.

UPDATE: @wpauley says he has a copy of the AWS controls, but I haven’t seen them yet. When I get a copy I will post them.
UPDATE2: Apparently @wpauley was a special case. AWS is keeping the controls under wraps. If you have a copy send them to me anonymously and I will get them posted.

[1] Even the recent refresh of the Amazon Security Whitepaper (PDF) does not include details on the controls or control objectives
[2] Been a while since I was involved in a SAS70 and there is a specific language they use that I’ve forgotten. Did not find any examples on the net. Appreciate clarifications in comments below if you have them.
[3] I think this raises a broader question, which is should any public cloud ever be allowed to keep their SAS70 controls and control objectives hidden? There is a very nominal argument for security through obscurity, but the reality is that many people will have to see them anyway, so why not shed some light?

Post to Twitter

Comments
27Oct

On Second Thought…How Big Is AWS Really?

We are trying something new at Cloudscaling and inviting a few of the more interesting cloud computing bloggers to provide some alternative viewpoints.  We thought we would start with Andy Schroepfer. His additional analysis of AWS revenue is thoughtful and worth consideration.  –Randy Bias, Founder and Cloud Strategist, Cloudscaling

Guest Author Andy Schroepfer is VP of Strategy at Rackspace. You can follow Andy’s content on www.NoMoreServers.com and via Twitter @shrepfur.

Determining the exact revenue size of Amazon’s Web Services (AWS) unit is akin to finding the exact server that an Amazon customer’s code is running on in the cloud. In both cases, only Amazon knows the specifics whereas the rest of us are left to guesstimate. I have previously praised a few blog researchers for their good efforts and wanted to contribute to the discussion with an analysis of a different type. My approach, using just the financial detail in Amazon’s SEC filings, suggests AWS might be smaller than the much more detailed assessments grounded in usage data. To that end, I suggest that all of AWS is below $200 million annually, and perhaps closer to $150 million.

Read more…

Post to Twitter

Comments
23Oct

State of the Cloud – Cloud Camp in the Clouds

I recently gave a short, 5-minute ‘lightning talk’ at Cloud Camp in the Clouds.  This is the first ‘virtual’ (online) event for the Cloud Camp folks.  I want to particularly thank the organizers Reuven Cohen, Dave Nielsen, and Sam Charrington for allowing me to give our perspective and also to let us tee up the entire event as the first presenter.  A recording of the entire event is in the works, but I wanted to reproduce my presentation here to get it out sooner.

So here is a recreation of the talk including my original deck, which has my fonts (the perfectionist in me, I can’t help it), and a professional (sort of) quality voice-over.  I’d also like to thank the Slideshare folks whose audio synchronization tool was a cinch to use.

State Of The Cloud – Lightning Talk

View more documents from Randy Bias.

Post to Twitter

Comments
01Oct

Amazon’s EC2 Generating 220M+ Annually

Today I’m going to tell you how much revenue Amazon’s Elastic Compute Cloud (EC2) is generating. After all, you, my regular readers, come to this blog for its insight, original thinking, and gems of wisdom. Today I have something particularly juicy for you: real-world numbers on Amazon’s EC2 size and revenue.

Recently Ruv Cohen talked about how cloud computing is taking off and Guy Rosen made an attempt to derive EC2 usage information. What I have today is actual verified EC2 numbers plus some guesses and a rough model of it’s current annual usage. Not only do I have a pretty good line on Amazon EC2’s size and usage, but we can use that to infer revenue to a fairly close approximation. Once we have EC2’s revenue numbers we also have a good notion of the overall size of the current infrastructure cloud computing (IaaS) market. Looking at Amazon’s most recent 10-Q we can even make a guess about the annual growth rate.

How Big is Amazon’s EC2?
Big. 40,000 servers. I have independently confirmed this with at least two sources close to EC2. Obviously, I can’t reveal them, but they are personally known to me and reliable. The first source gave me the 40,000 number and the second confirmed that the number is close. At most, we’re talking +/- 10,000 servers, so within 25%, but I’m guessing I’m very close.  More like +/- 5K. Regardless, within 25% is more than close enough for us to get a pretty good gauge. For our purposes today we’ll go with the 40K number.

Before we dive in, I have to tell you that it’s pretty amazing to see a real number for the size of the largest part of Amazon Web Services (AWS). What’s even more amazing is that I can tell you a whole lot more about EC2 and it’s revenue.

EC2’s Characteristics
Let me tell you a bit about EC2. There are roughly three buckets of certainty here and I’ll divide these appropriately. For the sake of brevity I’ll deliver these in a bullet list.

Known For Reasonably Certain

  • Amazon does not oversubscribe cores
  • Amazon uses Rackable servers, mostly 2U dual-socket systems, with 8 cores, probably the S2108 models
  • Some of their older systems are 2U dual-socket systems with 4 cores total
  • The AWS target cost for servers is roughly $2-2.5K
  • Initial overall target utilization rate for EC2 was 70-75%
  • Recently AWS has been having regular capacity issues (due to growth?)
  • Amazon is beginning to move some retail site capacity onto EC2
  • RAID is used for the disk subsystem, probably in a RAID-0 (striped) configuration[1]

Mostly Sure

  • Most of the servers are likely in the US availability zones vs. the EU zones, maybe 75-80% of total capacity
  • Amazon probably runs 8 500GB SATA drives per system
  • At 2U, Amazon runs 16-18 servers per rack, probably 16
  • Amazon uses the same, or a very similar, physical hardware platform for all instances sizes; it has changed over time and there are a few generations, but they don’t buy or ’silo’ the hardware if they can help it[2]

Inferred or Assumed

  • At 40,000 servers evenly distributed across 6 availability zones we know:
  • ___ 6,700 servers per zone
  • ___ 417 racks per zone, 2,500 racks total
  • ___ 2 * 30A circuits per rack (assuming no redundant PSUs) at a draw of probably 6.5kw
  • ___ That’s roughly 16.25MW of total power, not including that of the critical systems (HVAC, etc.) or the other AWS services
  • ___ 95,000 used sq ft, which probably means 200K total sq footage including support systems
  • Taking a good guess, I’d say as many as 20% of instances are ‘reserved‘, meaning purchased at a 70% discount one year in advance
  • Another good guess is that up to 20% of EC2 capacity is being reserved for the retail site (Amazon.com)

Instance Size Usage
Most of the above is either very close to the truth or an educated guess. In order to make any kind of real revenue projections I have to go out on a limb and make some kind of assumptions about what percentage of each instance size is in use. Of course, this number fluctuates significantly, but there is an average.

Before I go out on a limb, let me say that neither the educated guesses above and wild guesses below are completely without merit.  You can see more about my background here, but very quickly, I’ve been running large scale infrastructures since 1990, have helped build multiple 100,000+ sq ft datacenters, was formerly in a senior leadership position at one of the top IaaS clouds in the U.S., and am running what I believe is destined to be the pre-eminent gold standard in cloud computing consulting (Cloudscaling.com)[3].  So, the educated guesses are probably pretty good and the wild guesses might not be completely wrong.  :)

So here’s my wild-ass guess on AWS instance size usage rates:

  • m1.small 21%
  • m1.large 35%
  • m1.xlarge 20%
  • c1.medium 13%
  • c1.xlarge 11%

It’s hard to ballpark these, but I’m basically assuming something like the following:

  • Most folks start on an m1.small until they go to production, then they move up to the m1.large, because it’s got decent I/O characteristics (this is a well known phenomena in EC2)[3]
  • m1.small was the dominant size initially (100% of all instances), but continues to lose ground as more people use faster instances, but it’s still at a highly desirable price point for ‘testing’ so it remains under heavy rotation
  • m1.xlarge’s are where the ‘big’ production databases are and they are fairly steady state; people with larger sites ’shard’ their database to continue scaling
  • I’ve been told the high-CPU instances are under high demand, so while they are relatively new, I believe combined they are around 25% total, mostly for batch processing & high-performance computing (HPC) tasks[4]

I believe that m1.large is on track to be the new m1.small.  The de facto size instance that people use for most purposes.  With the m1.xlarge for databases.  That should account for roughly 55% (as you can see above), high-CPU instances account for a larger than expected share at 24% roughly (per my sources), and the remaining 21% is the the small instances, which will continue to decline in popularity, but ever so slowly.

Utilization Rates
Given that the initial utilization rate target was 70-75%, I’m putting the minimum rate at 70%, the max at 80%, and the average is 75% assuming AWS is hitting it’s targets. After several years in operation I’m sure they are managing it reasonably well.

EC2 Revenues
Putting all of the above together into a spreadsheet, assumptions, known facts, and my wild guess at instance usage, we wind up at average utilization and revenues as follows.

EC2 Instance Usage Numbers

  • m1.small 34,675
  • m1.large 28,896
  • m1.xlarge 8,256
  • c1.medium 5,366
  • c1.xlarge 2,270

EC2 Instance Revenue Run Rates

  • Hourly $25,264
  • Monthly $18,190,195
  • Annually $218,282,342

Please keep in mind that this number does not account for Elastic Block Store (EBS), bandwidth transfer in/out, or anything except EC2 instances.  The actual revenue number is likely to be a fair bit higher (+25%?).

You can download the spreadsheet yourself to play around with, but be forewarned, it’s barebones and I make no claims of being a business analyst.  I’m just clueful enough to be dangerous. ;-)

EC2 Revenues as ‘Other’ & Rate of Growth
What’s interesting about Amazon’s reporting of it’s Amazon Web Services (AWS) revenues is that it’s part of the ambiguous “Other” line in their financial reports. Looking at the most recent 10-Q shows us that the total revenue for “Other” for six months ending June 24th, 2009 was 260M. In talking to my original sources, a number of 109M (half my annual estimated run rate) as part of the 260M Other line sounds ‘reasonable’. Revenues for S3, a much lower margin business are ‘probably’ 1/10th the size of EC2, and the only other items in ‘Other’ of note are IMDB and A9, both probably in the 10-20M range.  The other large unknowns are AWS credit card revenues and revenue related to bandwidth in/out of EC2.

So, in essence we now know with a reasonable degree of certainty that EC2 is, in fact, the majority of the ‘Other’ line in Amazon’s revenues. I think we can safely assume that EC2 accounts for most of the growth as well. Looking at year-over-year from 2008, the change is +23M or roughly 10%. Which is impressive in the face of the general downturn. Of course, this number could be much higher if EC2 is replacing the loss of other revenues (A-9?), which seems quite possible.

Infrastructure Cloud Market Size?
From the grapevine, I understand that Rackspace (and probably others) are also growing very rapidly, but likely even Rackspace Cloud, the nearest competitor is around 10% the size of EC2 at best. Using that assumption and EC2 as a guidepost, the total Infrastructure-as-a-Service (IaaS) market is likely around 400-600M in size and growing at a clip of 10-20% annually in a major economic downturn. Pretty impressive given that the overall hosting market has been depressed.

Summary
It feels good to have more light shined on what has been an unknown. As you can see this is a real and rapidly growing market. We’re still in the ‘early’ phases of the adoption curve. If cloud computing gains more traction, a broader audience, and the economy recovers, it won’t take long before this is a billion dollar a year business.

–Randy

Ps.  For bonus points, multiply the target server cost for AWS by the number of servers, work out the CapEx costs, derive the monthly amortization and compare to revenue.  You’ll see why I’ve been telling people that this is not a ‘low’ margin business.

UPDATE: @ruv, et. al. helped me see a flaw in my calculations where I was using the 6 month 10-Q number for the annual Amazon ‘Other’ revenue.  Updated to make the comparison correct.  Still standing by a 220-260M EC2 run rate, which is in-line with my knowledge of the general size of other smaller clouds and more realistic in terms of the makeup of the ‘Other’ revenue line.

[1] Meaning that when the drive fails, the RAID is trashed and all instances on the system are gone
[2] By ’silo’ I mean they try to reuse the hardware across all instance sizes as much as possible; they do not buy and keep one piece of hardware for each instance size
[3] Look for more announcements soon; what was my personal consulting business has taken off like a rocket and we’re now 8 going on 20 people; interested?  email jobsnospam@cloudscaling.com (yes, that email address is a Turing test).
[4] We have clients who do this and I’ve seen it with just about every startup or large business we (Cloudscaling) has helped onboard to EC2

Post to Twitter

Comments

Twitter links powered by Tweet This v1.6.1, a WordPress plugin for Twitter.