Infinite Undo! RSS

Falsehoods About Time · Code Coverage · Interview Questions · Functional Testing


 
 

Hello, I'm Noah Sussman

Dec
29th
Sun
permalink

Continuous Delivery is Mainstream

Google, Amazon and Facebook all are using very aggressive Continuous Delivery workflows and have been doing so for years.

UPDATED January 21, 2014

I meant the adjective “mainstream” in the sense of “not dangerous.” For example: “The Ramones are so mainstream, I only listen to Norwegian Death Metal.” Based on the large amount of feedback I have received, this is not everyone’s default definition of “mainstream.” Hopefully adding this paragraph to the post will clear up such ambiguity for future readers.

Once again, I did not mean to imply that everyone is doing CD. Everyone is not doing CD! But, CD is no longer the risky experiment it was in 2010 when Chad Dickerson hired me to help scale the CI system at Etsy (which is how I got involved in this whole discussion in the first place). Today CD is a mature option and I think it is the best option available. But there are certainly other ways to build software and lots of people use those other methodologies to great success.

I apologize for once again having slightly confused a small portion of the Internet and ultimately starting a couple of interesting if off-topic conversations. Now here is a kitten GIF to atone for my actions and let us never speak of this again.

Wow that kind of turned into a rant. Sorry. Anway — as I was saying: Google, Amazon and Facebook all are using very aggressive Continuous Delivery workflows and have been doing so for years.

Weekly, daily or even hourly production deployments are a commonly-practiced, enterprise-scale software development life cycle. (Editor’s note: in retrospect we can see how “commonly” might be taken to mean “universally” in this sentence. That was not our intent. In our opinion, no methodology is universally practiced. Many successful and competitive organizations still practice Waterfall. See also diseconomies of scale.)

Today, numerous “Web-scale” organizations (notably Google, Amazon and Facebook) practice continuous deployment, with real-time production monitoring and a lot of exploratory testing taking place in production as well.

Here are a dozen articles about Web-scale continuous deployment in the enterprise

After reading these links it should be clear that Continuous Delivery is Yet Another Mainstream Software Methodology, just like Agile and Waterfall. Please reach out to me on Twitter if if you think you can still make the case — in light of this evidence — that Continuous Delivery is still “untried” or especially risky ;-)

One other thing to take note of here: almost all these articles are old. Some were written 2 years ago or more. Rapid iterative development may or may not work for a specific organization — but the general problem of implementing Continuous Delivery is clearly well-solved and has been so for a while now.

Google practices Continuous Delivery

  1. At Google, 15,000 engineers work from the HEAD revision of a single Perforce trunk.
  2. 50% of the code will be changed in any given month.
  3. Google’s test infrastructure is legendary and they’ve written a comprehensive book about how they perform QA while continuously releasing.
  4. They’ve also put a lot of effort into scaling Perforce.

Here is a fantastic deep-dive into Google’s deployment pipeline:

Amazon practices Continuous Delivery

  1. At Amazon, new code is deployed to production at a staggering rate of once every 11.6 seconds during a normal business day.
  2. That’s 3,000 production deployments per day.
  3. They’ve invested an enormous amount of time and money into creating an architecture that facilitates small, orthogonal, frequent code pushes.

Here’s Amazon’s Jon Jenkins breaking down their deploy stats at the O’Reilly Velocity conference:

Facebook practices Continuous Delivery

  1. At Facebook, each of 5,000 engineers commits to trunk HEAD at least once a day and the code at trunk HEAD is pushed to production once daily.
  2. Facebook has no dedicated QA team. All responsibility for testing rests with the software engineers.
  3. They’ve invested heavily in infrastructure that provides zero-downtime deployment at Facebook scale.

Here’s Facebook’s release manager, Chuck Rossi, going into detail about how Facebook engineers balance their experiments against the risk of breaking some fundamentally important part of the site: