Last night I presented at the Chicago DevOps Meetup along with @mattokeefe and @mfdii. The presentation went very well and was warmly received. It’s a bit of a revisit of some topics I have covered before, such as Pets vs. Cattle, the new kind of elastic cloud, and the architectural patterns involved with building these new kinds of clouds. I did a major refresh though and am weaving the threads a bit differently than before.
I hope you enjoy it.Cloud Computing | Leave a comment
Fall of last year I wrote a controversial whitepaper detailing my concerns about how distributed storage was being marketed. The blog introduction and the whitepaper were both entitled Converged Storage, Wishful Thinking, and Reality. There was a certain amount of expected blowback from folks at RedHat and Ceph as well as more thoughtful replies from Dreamhost and Solidfire. Some of the feedback was pure screed, some was worthy of considered thought, and much of it missed the mark. I realized however that it was *my* failure to communicate clearly that caused a considerable amount of the feedback to be erroneous.
So I wrote an update to that whitepaper that I think helps clarify my thinking in this area. Fortuitously, the recent Google outage really helps to emphasize what I am talking about. If you haven’t been paying attention, you can find out more about it here and from Google themselves. In a nutshell, Google, who runs a 3-4 nine operation for most services (meaning 99.9-99.99% uptime), had a 35-minute outage due to a software bug. It’s noteworthy that an event like this is so heavily scrutinized given that the vast majority of businesses struggle to achieve three nines of uptime.
More importantly, this outage highlights what I was trying to say in my original paper, which is that large homogeneous software systems are inherently dangerous to uptime. Systems fail because of hardware failure, operator error, or software bugs. Traditional high availability (HA) pairs solve for the hardware failure problem, but typically ignore operator error or software bugs. Newer distributed software systems *still* predominantly solve for hardware failure, ignoring the more likely failure scenarios of operator error and software bugs. This is a major problem.
There are a number of ways to solve these problems, including but not limited to: running more than one kind of software system (i.e. moving to a more heterogeneous set of systems), using sharding techniques, and applying disaster recovery techniques. All of these approaches are essentially doing the same thing, which is limiting the failure domain size. Not having fault isolation means you will see cascading failures as AWS has seen several times when it violated some of these basic principles. Operator errors combined with software bugs caused massive EBS failures, which spilled across Availability Zones because the EBS control plane spanned the AZes.
This is my problem with how distributed storage systems are being marketed. I love distributed storage. I think it has a place in the datacenter. Cloudscaling is currently evaluating our options for integrating open source distributed storage software in our product. The problem is that it’s place is not to run everywhere in the datacenter and the marketeers at various storage startups who believe so and market so-called “unified” storage solutions are really setting everyone up for failure. There is no spoon.
For more details, here is my updated storage whitepaper, now entitled The Case for Tiered Storage in Private Clouds, that hopefully furthers the conversation on the appropriate best practices and patterns to use for deploying storage in private clouds today.
In the interest of transparency, I will say that this new revision was kindly reviewed by the team at SolidFire, who have a similar viewpoint to mine. Much thanks to Dave Wright, Dave Cahill, and John Griffith (PTL for OpenStack Cinder) for their feedback.Posted in Uncategorized | Leave a comment
Voting is now open for the Spring 2014 OpenStack Summit in Atlanta!
Cloudscaling (and our friends at Mirantis) have submitted a variety of great session abstracts, from Randy Bias’ walkthrough of hybrid cloud landmines to avoid when architecting applications to Boris Renski’s discussion on updating the OpenStack mission statement to disrupt large player competitive barriers and keep the stack open for innovation.
Below, we’ve summarized each talk and provided a link to its page on the voting site. As always, you must be a member of the OpenStack Foundation to vote. If you’re not already a member, just click here to join as an individual member for free. You get to shape the OpenStack Summit agenda by voting up the sessions that you’d like to see.
We look forward to seeing you in Atlanta!
(Vote Here) Open Cloud System 3.0: OpenStack Architected Like AWS (Azmir Mohamed) - At Cloudscaling, we spend a great deal of effort operationalizing OpenStack for customers and ensuring it’s compatible with leading public clouds. In this session, we’ll detail the configuration and design decisions we make in real world OpenStack deployments.
(Vote Here) OpenStack Scale-out High Availability: Scaling to 1,000+ servers without Neutron (JC Smith + Randy Bias) - The default OpenStack networking models don’t make sense in a modern datacenter. Similar to the approach taken by HP, CERN and other deployments, we’ll detail how we replace the default networking model through standard plugins in order to deploy a fully scale-out networking model that can support 100s of racks with incredibly high performance.
(Vote Here) For Cloud Operators: Hands-on With OpenStack for Elastic Clouds (Sean Winn) - In this session, we will provide a condensed version of our 2-day OpenStack Bootcamp workshop with an operator-centric, hands-on approach.
(Vote Here) ZFS on Linux + OpenStack: Flash Performance at Spinning Disk Prices (Joseph Glanville) - Join us for this walkthrough of ZFS on Linux (ZoL) where we’ll cover the state of ZoL along with the related OpenStack plugins and drivers. You’ll leave with a clear understanding of why multi-tenant cloud workloads are perfect for ZFS as well as how to measure and tune performance.
(Vote Here) Voice of the User: Running OpenStack in Production (Drew Smith + Ryan Lane) - Join us for an analysis and interactive discussion of real world OpenStack deployment issues. Along with guest speaker, Ryan Lane, who is running an OpenStack based cloud, we’re going to combine a review of data from our field and support requests with war story examples of “when it goes wrong”.
(Vote Here) OpenStack Block Storage: Performance vs. Reliability (Randy Bias + Joseph Glanville) - In this presentation, we’ll talk about selecting and designing the right kind of storage for your use case. We’ll also walk you through how we built our block storage solution in Open Cloud System, its design principles, and explain why it is a better version of AWS Elastic Block Storage (EBS) in several key ways.
(Vote Here) iSCSI Performance Testing & Tuning (Joseph Glanville) - In this session, we’ll explore the pros and cons of different iSCSI implementations within OpenStack Cinder deployments. We’ll then look at performance tuning options for iSCSI as well as the effects of related architecture components such as using jumbo frames in the network, flash storage, hybrid storage pools and iSCSI multi-pathing.
(Vote Here) Emulating A Large Scale, Multi-Rack Cloud on One Server (Mike Lang) - In this presentation we’ll introduce a new open source tool that allows us to emulate a multi-rack, multi-server, OpenStack deployment on a single server, allowing complete cloud architectures to be tested. We’ll also recommend how this tool can be used to expand the OpenStack continuous integration system and possibly serve as a mechanism for testing a variety of OpenStack reference architectures within complex, multi-rack deployments.
(Vote Here) Hybrid Cloud Landmines: Architecting Apps to Avoid Problems (Drew Smith + Randy Bias) - Application portability? Cross-cloud replication and bursting? These are hard issues with serious implications that we’ll examine in detail, digging into the subtleties between simply implementing a cross-cloud API and actually using it to get real work done. We’ll provide a set of recommendations on how to architect your application from the ground up to be hybrid cloud friendly.
(Vote Here) Tinker: A Tool for Network Configuration Management at Cloud Scale (Abhishek Chanda) - Managing network configuration in a cloud deployment can be a major pain. In this session, we will introduce tinker, a tool to generate and manage network device configuration using existing metadata to automatically and reliably generate configurations for all network devices in a cloud.
(Vote Here) OpenStack and the Google Compute Engine APIs (Alex Levine + Randy Bias) - It’s a hybrid cloud world and OpenStack powered public clouds won’t be everywhere, so it makes sense that we have a rich ecosystem of compatibility APIs in addition to the native OpenStack APIs. Come learn about the Google Compute Engine (GCE) APIs we recently built and submitted back to StackForge, including how they were built, how to add them to your own OpenStack deployment, and how to test them.
(Vote Here) Configuring Live Migrations for Workload Transitions (Damian Igbe) – This talk will review approaches to three fundamentally different live migration methodologies: block migration, shared storage migration, and migrating instances booted from Cinder volumes (i.e., volume booted migration). It will compare these three approaches and discuss factors to consider when deciding which methodology to use in different scenarios. We will also review practical (and sometimes critical) steps to ensure that both the cloud and its workloads are properly configured for live migration, along with important factors that can make or break smooth implementation and operation of live migration.
(Vote Here) Step One, Launch Openstack; Step Zero, Pick your hardware (Gregory Elkinbard) – The power of OpenStack software implementation, deployment and operations all cross the critical path of hardware selection. In this presentation, we’ll present a distillation of the key factors we have applied at Mirantis through dozens of highly successful OpenStack deployments on how to select an optimal hardware platform for your OpenStack cloud.
(Vote Here) Source Based Packaging: The Missing Ingredient of Continuous Delivery? (Dmitry Borodaenko) – This talk will explore the gap between source-based CI used by the OpenStack community and package-oriented deployment favored by OpenStack distributions, and examine the state of tools aiming to bridge that gap. Attend and you’ll learn how to tick the OpenStack CI/CD checkbox without leaving the comfort of the package-config-service pattern and the safety of security updates from your favorite distro.
(Vote Here) Is it time to update the OpenStack mission statement? (Boris Renski) – This talk will take a closer look at various OpenStack community initiatives and projects that disrupt competitive barriers established by large infrastructure players. We’ll also examine how the OpenStack community is making OpenStack distribution vendors in their present form irrelevant in many respects.
(Vote Here) Preparing and Publishing an Application in the Openstack Application Catalog (Alexander Tivelkov) – This workshop will walk you through the steps required to prepare an application for publication. The presenters will pick a real-life application and demonstrate how to publish it in the Catalog. Attendees will see how to define the configuration parameters and dependencies, build the deployment and maintenance workflows, construct a user-friendly UI to control these workflows – and, eventually, how to publish all of it in the Catalog. And all of this without writing a single line of Python code or modifying a single bit of the application itself.
(Vote Here) How to Avoid Picking Problems that Openstack Can’t Solve (Evgeniya Shumakher) – To be effective promoters and developers of OpenStack, we need to be able to identify these cases (not all edge-cases, either) and have the firmness of conviction to guide prospective users away from OpenStack, when risks are so high that their results will be marginal or unsatisfactory. (And we need to look at these gaps, where they exist, and think hard about filling them.) Attendees will leave this presentation with a fuller understanding of how to pilot their organizations away from what may become lackluster OpenStack engagements, and towards opportunities that take best advantage of OpenStack as it now exists.
(Vote Here) Extending the Fuel Project to Accelerate and Simplify Deployment of OpenStack Drivers and Components (David Easter) – In this session, we’ll fully familiarize attendees with the (relatively simple) process of becoming an OpenStack contributor. We’ll go over the process roadmap, account registration, security and authentication, connecting with community online resources and documentation, OpenStack and related code repositories, projects and their taxonomies, configuring Git to work within OpenStack’s code-review process and committing changes, review and approvals. And we’ll also cover details of how and where to best patch Fuel and describe the Fuel API timetable, imminent component releases and Fuel’s project roadmap. Attendees – particularly developers – should come away from this session ready to join the OpenStack project, find resources, pick goals and quickly be productive, especially where contributing to Fuel is concerned.
(Vote Here) Rally – Benchmarking OpenStack at Scale Using Your Own Hardware (Boris Pavlovic) – Benchmarking a huge distributed system like OpenStack raises the complexity bar significantly. Add to that the fact that OpenStack may be a moving target – with Continuous Integration/Continuous Deployment (CI/CD) keeping things in flux – and benchmarking becomes, as Spock might say, “a non-trivial problem.” This session will explore Rally’s high level architecture, the current state of the Rally project, and the Rally Roadmap. We’ll also cover how to use Rally to improve the quality of OpenStack (Dev case), to ensure that OpenStack clouds pass SLA (QA/DevOps case) and to help build CI/CD. Attendees should leave this session with a solid understanding of Rally and most of the info they need to begin integrating Rally to existing OpenStacks and architecting it into new deployments.
(Vote Here) How Community can make OpenStack Customer Centric (Alex Freedland) – As a major OpenStack vendor, Mirantis has developed a set of best practices for welcoming and working with end-user customers, extending to them the benefits of OpenStack community participation, and conversely, obtaining from them guidance and actionable market intelligence. By discussing these best-practices, we hope to 1) help the OpenStack community reach out more consistently, and with better results, to real end-user customers 2) develop beneficial (but also non-burdensome) roles in the community, enabling and shaping participation by a range of parties whose main motive is to use rather than build the product and 3) explore structural tweaks that will enable better exchange of information between ‘style: end-user’ participants and members of the community who participate in more fundamental ways.
(Vote Here) Bare Metal Peer-to-Peer Provisioning at Scale (Herman Narkaytis) – In this talk, we’ll review the current state of bare metal provisioning in TripleO / Ironic and Fuel and model the scalability limits of alternate approaches and architectures. We will discuss the state of things in Icehouse, including what use cases are covered, the algorithm, and limitations. We will also explore whether plans for Juno can reverse diminishing scalability and drive bare metal provisioning closer to linear performance. Bare metal provisioning (versus creating virtual machines) has its own challenges, and after attending this talk, you’ll not only have a better understanding of the differences, you’ll also have a feel for possible ways to mitigate those challenges.
(Vote Here) Breaking the East-West Barrier: How to Combine a Flat Network, Security, Manageability and Killer Performance in Openstack (Pedro Marques, Przemyslaw Grygiel) – In this talk, we’ll outline a practical, step-by-step method for configuring and provisioning a 200-node OpenStack cluster using Fuel. Using a fully-functional physical infrastructure, we broke through very real bottlenecks in Openstack defaults and in tooling (the proof of a real bottleneck is that when you resolve it, it exposes the next bottleneck). In addition, we will enumerate typical mistakes and pitfalls.
(Vote Here) Securing OpenStack with Intel Trusted Computing (Christian Huebner) – Intel Trusted Computing allows comparing the current state of a compute host with a previously stored good state. This presentation will contain a general overview of Trusted Computing, OpenAttestation server and client deployment on Red Hat Enterprise Linux, the inner workings of trusted_filter.py and its attestation procedure against an OpenAttestation server and configuration and testing of OpenStack controllers and compute nodes for trusted execution. The presentation is based on a recent deployment of OpenAttestation with both trusted and untrusted compute nodes using RedHat Enterprise Linux 6.5 and OpenAttestation 1.6.
For Cloudscaling, the journey to the hybrid cloud begins (but does not end) with API compatibility between the OpenStack powered private cloud and leading public clouds such as Amazon Web Services (AWS). Cloudscaling customers run OCS, our OpenStack powered private cloud solution, because it enables them to burst to and repatriate from AWS. Leveraging an OCS private cloud as part of their hybrid cloud strategy provides agility, choice and flexibility to the application and infrastructure teams.
The AWS EC2 API has been part of the OpenStack Compute (Nova) since the project’s inception – Cloudscaling have been a beneficiary of this initial community investment and we have built a robust business around it. A typical AWS customer has invested heavily on AWS specific tooling and training so the cost of change is high. Cloudscaling OCS makes the transition to an OpenStack private cloud palatable for these enterprises because the same tools (Rightscale, Dell Cloud Manager, AWS CLI, etc) and training are fully compatible. Customers do not have to start from scratch when transitioning from AWS to OpenStack using OCS.
However, we have always felt that the AWS API was not the end all be all for our customers and the broader OpenStack community. Today, we are changing that by making the GCE API available to the OpenStack community via StackForge. We believe the GCE API provides the same benefits as the AWS EC2 API to the community – agility, choice and flexibility .
More About StackForge
For those not familiar with StackForge, it’s the OpenStack incubator – it allows the broader community to see contributions and kick the tires on OpenStack related projects. StackForge is also the way that OpenStack related projects can consume and make use of the OpenStack project testing and continuous integration (CI) infrastructure. The ultimate goal of any StackForge project is the same – inclusion into OpenStack. We are passionate about GCE and why we have expended considerable energy to make the API set available on StackForge.
How passionate? For starters, take a look here. As you can see, Cloudscaling is keeping great company in StackForge – IBM, HP, RedHat and Mirantis are among the stable of companies that have contributed code to it. Our GCE API contribution is also significant – more than 27,000 lines of code broken across the GCE API, JSON with API definitions, OpenStack Commons, Unit and Tempest Tests. Finally, our GCE API contribution is covered under the Apache Project Licence (APL) v2, just like OpenStack.
Take GCE API For A Spin
Get started with GCE API here – simply enable it as a service on your Grizzly or Havana based OpenStack cloud. It’s also enabled on OCSgo, our evaluation private cloud where you can also experience all the functionality available in the latest OCS release. We are initially validating functionality using gcutil and will include other GCE compatible tools in the future. The Cloudscaling team believes we have done all the right things to make GCE API another choice in OpenStack. What do you think?
UPDATE: Broken link fixed to the repos.Posted in OpenStack | Leave a comment
OpenStack is a powerful way of providing and consuming IaaS to achieve scale and agility, but its very nature as an open source project raises IT management challenges. Removing some of the implementation complexity and management obstacles from the equation is vital to enterprise adoption. Cloudscaling is committed to advancing OpenStack to meet the needs of the enterprise through advanced features in our core product Open Cloud System (OCS).
To that end, we are excited to announce that our latest release – Open Cloud System (OCS) 2.6 – is available today. This release marks a key milestone in the evolution of OpenStack, making it easier to deploy and consume.
The major features in this release include
Initially we are addressing two key use cases with Dashboard:
But it does not end there. We are investing heavily on this graphical user interface (GUI) because it is a critical component of how enterprises can realize the promise of life cycle managing hundreds of cloud nodes per cloud administrator. You can expect the list of capabilities and use cases to expand in the coming months. Visit our Resources page to learn more about our latest release.
In addition to OCS 2.6, Cloudscaling’s value extends through our Professional Services and Support teams. Visit our Cloud Concierge and Training pages to learn more about how we can help you put OpenStack to work for your business.Posted in OpenStack | Leave a comment
Recently, Barb Darrow wrote a note to Google Compute Platform, suggesting “8 Things it Could do to Freak Amazon out.” As long term readers know, I called out Google’s entry into the public Infrastructure-as-a-Service (IaaS) a while ago. My extensive experience at GoGrid and in the service provider and hosting business, provides me with a deep understanding of the public cloud market’s underlying business models and technologies. Cloudscaling also created the Google Compute Engine (GCE) APIs for OpenStack, which were finally submitted and made available via StackForge just today (more on that later).
Barb’s recommendations to Google were a good start, but I would like to expand a bit on her original article. Here are my thoughts.
8 Things GCE Could do to Freak out Amazon
#0 – sub-hour payment increments
This is not in the official list as it seems to be taken for granted that GCE already does this (they do) and that AWS should be concerned (they shouldn’t). While sub-hour payments sound good as it provides greater granularity in billing, it’s doubtful that most people will shave off any significant dollars. The main reason is that an hour of granularity is reasonably good and that most workloads expand during the day and contract at night following the general pattern of daily Internet traffic. The secondary reason is that when you combine server failure detection times with new VM spin up times and initial provisioning times, you are going to be well into a full hour. We simply aren’t at a point where the majority of apps spin up VM capacity rapidly (<1 minute) and spin it back down quickly. Elasticity is measured in hours, not minutes right now. That will change over time, but sub-hour payment increments are a premature optimization and marketing differentiation more than a reality for most customers right now.
#1 – Provide Reserved Instances
Makes sense. Couldn’t agree more. I’m sure they are hard at work thinking about it, but this is not a very big challenge so I’m sure Google will get to it when it makes sense to them and as Barb says it’s likely GCE is doing this on a case by case basis already.
#2 – More Managed Service-y Services
Google was churning out developer friendly services before Amazon even launched EC2 and it’s been cranking them out just as consistently. If you actually look at the Google dashboard you will be stunned at their ecosystem of services. It already eclipses AWS. I think it’s a mistake to look at Google as having to “catch up” to Amazon. What really happened is that Google started at the application layer and has been working down towards infrastructure, while Amazon started at the infrastructure layer and has been working up towards applications. Neither is really “ahead” of the other in the broader sense, although if you pick specific services, clearly their respective “leads” are relative to where a given services is in their “stack”. If it’s closer to the application layer, Google’s ahead, closer to infrastructure, Amazon is ahead.
#3 & #4 – Parlay Search / Get Louder
Not much to say here. These seem self-evident and I’m sure we’ll see more from Google given how they have been accelerating recently.
#5 – Offer More and Different Types of Instances
Makes sense. Customers want choice and variety. I am certain we will see choice increase over time. I will respectfully disagree with the smart and talented Sebastian Stadil however. It’s actually a mistake to provide live instance resizing. This is one of those kinds of suggestions that makes a *ton* of sense from the cloud end-user perspective, but actually has a negative impact for the public cloud provider. Live instance resizing isn’t strictly technically hard. Guest OSes support “hot plug” of RAM, CPUs, and other virtual hardware components and most hypervisors also support this. The problem however is that if you allow arbitrary instance resizing it’s very difficult or impossible to provide QoS and resource SLAs on a per instance basis. You also stand a very good chance of blowing up your business model by not having an even “bin packing” of instances on hardware. Most of these clouds need to run at around 80% sold capacity for the business model to work out. Instance resizing could cause that to go as low as 40% if you try and maintain QoS/SLAs, which would destroy the profitability of the public cloud itself. This is also more of a “pets” approach than a “cattle” approach.
For example, imagine you have a piece of hardware 4 cores (4c) and 4GB RAM (4g) that hosts 4 VMs, each of which get 1c/1g a piece (I’m simplifying here so bear with me). Now, each VM is roughly guaranteed one core each. This is an oversubscription ratio of 1:1 (AKA “no oversubscription”). If each customer suddenly resized these VMs to 4c/1g, that oversubscription ratio would change to 4:1, meaning that if all were banging away with their CPUs they would still only be getting roughly 25% of what they asked for, or one core. You can’t provide strong performance guarantees if you allow instance resizing. Now, some of this could be addressed by dynamically reallocating workloads via live migration on the backend and building a sophisticated scheduler to allow “re-packing”, but it would get very challenging in the situations where there simply wasn’t enough immediate capacity to service a request. The reality of the situation is that this is more trouble than it’s worth and customers would be better off learning how to dynamically spin up a new VM and scale horizontally. The new model is cattle, not pets.
I wrote up a detailed example of how oversubscription impacts cloud performance in late 2009.
#6 – Add More Datacenters
No comment. Google has a global reach and will use it. Nuff said.
#7 – Offer Virtual Machine Image Import/Export
Here here. Every cloud needs more of this. I’ve long said that even OpenStack Glance should be working more in this direction.
#8 – Do All of the Above But Faster
Google’s development engine is as fast as Amazon’s and in fact, you could argue that Google pioneered the agile webscale development model. This really comes down to a question of semantics. If you say that public cloud starts with infrastructure and then expands, then Google is chasing Amazon. If you say that public cloud starts with application services like Google Mail/Search/Apps, then *Amazon* is chasing Google, moving towards application level services. The reality is that either entry point is still “cloud” and that Google and Amazon have been inexorably moving towards each other. Google is the only major competitor to Amazon in this regard already. The only possible exception I would make being Microsoft, but they have their own internal problems to resolve before they are a serious contender.
Look at the number of developer centric APIs Google already has (this is a partial screenshot as the complete list is too long to fit):
There are over *75* developer APIs across all of Google’s cloud services. Perhaps Amazon should start moving faster?
What Was Missed: Google’s Networking Advantage
Interestingly, one of Google’s biggest technical advantages was overlooked. For the past decade Google, Yahoo!, and Microsoft have been buying up dark fiber and deploying it between their datacenters. These folks, particularly Google have massive nation-wide networks capable of Terabit speeds (1000s of Gigabits). These networks interconnect Google datacenters. By some measurements Google is the 2nd or 3rd largest ISP in the USA.
Dark fiber doesn’t grow on trees. It’s incredibly costly to deploy and as what was in the ground has been sold off, it has become progressively more expensive. If you own the dark fiber and light it yourself, you can continue to push more bandwidth across the same strands by using new DWDM gear on either side. If you *don’t* have dark fiber, you only have access to “lit fiber” that limits how much traffic you can push across.
Google and Microsoft have massive structural advantages in their dark fiber networks and can not only have massive bandwidth between datacenters, but also have it at a very low cost. And Amazon cannot “come from behind” to make up the difference as supplies of dark fiber are now severely limited until the next wave of fiber buildouts and no one has any idea when that will happen.
What can Google do with this? Imagine the ability to have all Google Cloud Storage geo-replicated for free bandwidth costs. With so much bandwidth, moving huge VM images (100+ GB) across the network no longer sounds crazy. What about having your “regions” be as big as the entire eastern or western seaboards? All of this is possible if you have a big, fast, cheap WAN, which Google has and Amazon will never be able to achieve.
It’s Going to be a Two Way Horse Race
Exciting times in cloud land. Amazon Web Services finally has a credible competitor in Google’s Cloud Platform. I’m glad that others are beginning to see the light and I am still hopeful that Microsoft comes from behind and shows up to the party. In the meantime, watching Amazon and Google compete should provide plenty of entertainment and collectively we can all help the cloud revolution happen faster.Posted in Cloud Computing | Leave a comment
In the modern world of IT the need for scalability and agility is undeniable. Cloud computing with OpenStack is a powerful way of providing and consuming IaaS to achieve scale and agility, but its very nature as an open source project raises IT management challenges. Removing some of the implementation complexity and management obstacles from the OpenStack equation is vital to enterprise adoption.
For enterprises to confidently migrate – as a matter of course – to OpenStack-powered IaaS, six requirements must be met. 1) High Availability, 2) Robust Management, 3) Open Architecture, 4) Scalable and Extensible, 5) Hybrid Cloud Federation, 6) Global Support and Services
The Cloudscaling webinar series will shine the light on the essential technical and business components required for organizations to confidently adopt enterprise-grade OpenStack powered clouds. Each webinar will include a live Q&A with the individuals at the center of the OpenStack movement. This four-part webinar series is now open for registration.
Date: Thursday, January 23, 2014
Date: Thursday, February 6, 2014
Date: Thursday, February 27, 2014
Date: Wednesday, March 12, 2014Cloud Computing | Leave a comment
It’s useful for all in the OpenStack community to remember that many believe OpenStack was not ready for prime time until Essex or possibly Folsom. Some might even claim Grizzly was the first true production release. My money is on Folsom for what it’s worth.
What this means is that 2013 was YEAR ONE for OpenStack in terms of potential enterprise adoption. Think of 2010 and 2011 as the ALPHA and BETA years. Similarly, 2012 can be thought of as the 1.0 release and Grizzly as 1.1.
As you know, enterprise products don’t have thousands of customers in year one. In fact, according to a friend, Michael Mullany, previously VP of Marketing at VMware now CEO of Sencha, “Server units were … very slow for ESX in year 1.” Given that we are year one for OpenStack enterprise adoption, the recent user survey can be seen in a very different light:
387 deployments, doubled from 6 months ago
almost 200 of these deployments are for production workloads
47 of these deployments are over 100 physical servers in size
continued doubling every 6 months would result in ~3,000-6,000 deployments by year three, on par with VMware’s announced 5,500 server customers in year three
So, how did we do in year one of enterprise adoption?
The Year in Review
OpenStack began this year with three major discussion themes:
rapid expansion of service provider and web app use cases
ecosystem wars (OpenStack v Eucalyptus v CloudStack)
anticipation of Grizzly and the maturity it would bring
Fast forward 12 months and you can see just how much ground the OpenStack community has covered. And, when you consider that the Foundation is a little more than a year old, you begin to appreciate the progress that’s been made.
1. Growth, growth, growth
The OpenStack Foundation’s second user survey shows 822 respondents from 539 companies running 387 deployments, 165 of which are in production. The majority of these deployments are on-premise private cloud (58%), and while a bunch of them are less than 500 compute instances, there are a significant number in the 500-5,000 range.
Compare these numbers to the first user survey and extrapolate the trend (dangerous with two data points, granted), and you see impressive growth across the board. In six months, we’ll have another survey and can more confidently project growth. Until then, it’s safe to say OpenStack is growing where it counts: deployments.
2. Enterprise traction
Enterprises achieving success with OpenStack fall into two camps: those who already have substantial open infrastructure expertise in house, and those who hired that expertise via a vendor with a supported distro or product. There are few enterprises that fit the former case, and commercial products based on OpenStack are only now maturing to serve the latter case.
OpenStack is a less mature analogue to the Linux kernel, not a Linux operating system. It’s a framework and a centerpiece for a full cloud operating system. Most enterprises are not going to have the patience or risk tolerance to build their own OpenStack-based deployment from raw code, then bear the cost of supporting it. Doing so is much like rolling your own Linux operating system distribution, which few do.
We see enterprises embracing OpenStack in our sales pipeline and in boot camps like the one we ran at Cloud Connect Chicago. It’s going to be a year or more before those enterprises work their way from deployment strategy through standup and on to production. Then, they’ll have to decide how much they want to talk about it.
3. OpenStack spreads its umbrella
OpenStack, as of Havana, is now 13 separate projects, including the four new projects in incubation status. These new projects provide key capabilities requested by the enterprise such as support for bare metal instances for better support of big data and advanced hardware. Database as a service is another new feature of interest to enterprise buyers.
4. Rackspace diminishes in OpenStack influence
Rackspace is now the #5 contributor and we are seeing their community influence diminish. With all due respect to Rackspace, this is good for the project: it shows the maturation of our democratic meritocracy, where it’s crucial that no one company is overly influential.
5. Demand for hybrid cloud is real
Hybrid cloud is the near term future. Even Andy Jassy of AWS has admitted it at the recent re:Invent conference. RightScale’s State of the Cloud presentation shows extensive interest from enterprises in multi-cloud and hybrid cloud solutions.
OpenStack is the best positioned project to deliver hybrid cloud. It’s got the most flexible architecture, extensive support for AWS APIs, growing GCE API support, and obviously, Rackspace API support is already there. That means that OpenStack is the only open source project with support for the top three public clouds. Most importantly, the community seems committed to working harder than ever on delivering public cloud compatibility.
There are challenges for OpenStack. We must come to terms with architecture and API frameworks. Technical and Board leadership must determine what is “core”. The community has to become disciplined and avoid the distractions of pet projects. And the disconnect among users, operators, and developers needs to be addressed. All of this will happen within the structure of an open source project that’s governed by a technical meritocracy, not a benevolent dictatorship. A tall order for 2014 to be certain.
Time to get back to work. Onward to Icehouse and the J release. See you in Atlanta next May.
Posted in Cloud Computing, OpenStack | Tagged OpenStack, predictions | 4 Comments
At Cloudscaling, we believe the future of IT platforms rests in the cloud. Predictions are always a gamble, but the following synthesis of insights from our interactions with customers, partners, analysts and our own prescient point of view, point the way to a compelling case for how cloud-enabled IT will continue its evolution in 2014.
(1) No More IT as Usual
The change that many of us have been talking about for 3+ years is finally coming. Enterprises are finally realizing that “cloud computing” does not mean server consolidation or automation. It’s a fundamental rethink of the IT stack, as well as the processes that deliver services to internal and external customers.
(2) OpenStack is Coming to an Enterprise IT Portfolio Near You
Related to the point above OpenStack will be the platform of choice that enterprises use to build their new IT infrastructure. It’s coming, whether centralized IT likes it or not.
(3) Public Cloud is No Longer the Enemy
2014 will be the year in which enterprise IT begins to figure out how to adopt the public cloud and use it to their advantage. Amazon Web Services’ myriad platform-as-a-service offerings for developers and higher value services for businesses will force traditional IT suppliers to rethink what they are doing. Google will join in the fight in earnest as it realizes the risk of being boxed out of a market where it should be vying for leadership.
(4) Public Cloud Becomes the Default Backbone for Enterprise IoT
The explosion of IoT activity in 2014 and beyond will be driven by the nexus of low cost sensors, connectivity networks, cloud computing, advanced data analytics and mobility. Aggregating this data and acting on its findings will best be achieved by capturing, analyzing and responding from the public cloud.
If you want to analyze billions of inputs in real-time or near real-time, you won’t want to drag the data all the way back to your data center. A longstanding mantra in business intelligence (BI) and data science has been that it’s easier to move the compute to the data, than the data to the compute. With cloud-based Hadoop and SaaS-based BI solutions proliferating, it’s becoming hard to justify bringing this data down to analyze it.
(5) Big Data Becomes the Leading Use Case for Hybrid Cloud
Related to the point above, the problems associated with moving large data sets make a compelling case for hybrid cloud in big data scenarios. Of course, we’re talking about hybrid clouds in which the public portion is AWS or GCE.
(6) Software Defined Everything Marches Forward
The steady progress of software defined everything (SDE) will continue. It’s no longer a matter of if, but when and how. The over-riding objective in the move to software-defined solutions is to run the infrastructure at higher levels of utilization and support faster changes to support a more agile business environment? Who doesn’t want that?
(7) Brands Become Open Platforms for Collaboration and Innovation
Across industries, businesses will be disrupted by open architectures. They will compete less on the features of their products and more on the breadth and quality of their connections. Increasingly the most successful brands will have open architectures that not only allow, but encourage active participation.
(8) Cloud spending will surge by 25%, reaching over $100B
According to IDC, the $100B figure includes software, services and cloud infrastructure. Workload specialized cloud infrastructure services will be the new form of differentiation among cloud service providers. A pitched battle will be joined for the developers that can create the cloud-based applications and solutions that will fuel the market’s growth. IDC predicts that by 2017, 80% of new cloud apps will be hosted on six PaaS platforms.
Where do we go from here? Tell us what you are prioritizing and which trends are top of mind.
Happy New Year !!!Posted in Cloud Computing | 1 Comment
Andrew Tahvildary joined Cloudscaling last month as Vice President of Engineering. He joins us most recently from newScale, where he led the engineering team for more than six years as the company effected a successful exit via acquisition by Cisco. He was a member of the executive leadership team, responsible for product development and architecture, quality assurance, support, release management, program management and lifecycle budgeting for the engineering organization.
Before newScale, Andrew was engineering VP for Primavera Systems, prior to its acquisition by Oracle. In total, his engineering leadership experience stretches back nearly 20 years, covering a range of technical roles.
I spent a few minutes with Andrew discussing his experience and his philosophy of leading high-performing engineering organizations.
RC: Let’s start with your experiences at newScale, Primavera and Evolve. What did those technical leadership roles teach you about building engineering teams at high-growth startups?
AT: The key thing to any engineering team – startups or non-startups – is the people. If you don’t have the right people with the right level of skills, the right attitude and the right desire, you’re not going to be successful.
Second thing is that great teams typically have fun producing good results. I never had a team that didn’t have fun and produce a product that was any good. So, it really starts with the team in my opinion.
After that it’s getting the right processes in place, the right technology, and the right strategic direction to be successful. My job is to create an environment for the teams to be successful.
Building a productive interface between engineering and product development/management is critical in the environment of rapid iteration that software startups live in. Talk about your philosophy on building solid connections between engineering and product functions.
To me it’s all about teamwork, especially when you’re iterating really fast. You’re examining different ideas and different options and really going to into unknown areas every day. This is especially so in early stage markets. You really have to work very closely as a team; product management plus engineering. I see them as just one team quickly iterating and getting feedback to define the right solution for the market that we’re going after. This has been the case at every successful startup I’ve been with.
Is the role of engineering leadership for a company leveraging an open source project different than the same role in a walled garden?
If I look at it from 30,000 feet, it’s almost the same.
At the end of the day you need to build a product that people find useful and useable, one that’s unique and a lot of people want to use. The market demands products that are simple to use, simple to implement, simple to get value from. But the key thing is in both areas – open source or proprietary software – you have to build things that are unique and that meet a broad market demand to be successful.
When you come to details there are different processes that you have to go through obviously, like the go to market strategy. But as far as building the products, it’s very similar. Of course, today, nobody builds everything from scratch anyway.
Hybrid cloud is becoming a preferred choice for enterprises that want to embrace ideas like elastic cloud and agile development. That’s a disruptive shift for most enterprises. What are the biggest concerns you’ve heard from enterprise IT when evaluating a new technology that will transform longstanding processes?
Most IT organizations are risk averse, but I think the market is moving more in the direction of open source, so enterprises are learning how to evaluate that risk.
You also have to deal with different skills sets for two different types of cloud. Typically if you have one side of the IT organization managing the public cloud like AWS, another group will manage the private cloud, whether that’s VMware or someone else.
The question becomes, how do you bridge that skill gap? How do you deal with SLAs across these things? How do you move applications? Before they knew how to manage their data center. But now when you open it up and you go to public cloud and hybrid cloud, are the two technologies the same? Are they working on the same set of APIs? How do you manage them? How do you support them? How you maintain things across?
Those are the concerns people have – the complexity of how you manage two very different cloud paradigms.
What is your first priority at Cloudscaling?
My first priority is to get to know the team, then get to know the product a little bit better. Then, we’ll develop an environment for us to be ready to get to the next step as an engineering organization. I think we’ve done a great job to get to this point and there are tons of things we can do that accelerate our velocity, our innovation as well as making it a fun place to be.
One of the key things that I’ve learned in different startups is that you should aim for success, not perfection. That’s really key thing.
Aim for success, not perfection.
Thanks, Andrew. And welcome to Cloudscaling.
You’re welcome. It’s great to be here.
Posted in Company | 2 Comments
← Older posts