Google, Amazon and Facebook all are using very aggressive Continuous Delivery workflows and have been doing so for years.
Wow that kind of turned into a rant. Sorry. Anway — as I was saying: Google, Amazon and Facebook all are using very aggressive Continuous Delivery workflows and have been doing so for years.
Weekly, daily or even hourly production deployments are a commonly-practiced, enterprise-scale software development life cycle. (Editor’s note: in retrospect we can see how “commonly” might be taken to mean “universally” in this sentence. That was not our intent. In our opinion, no methodology is universally practiced. Many successful and competitive organizations still practice Waterfall. See also diseconomies of scale.)
Today, numerous “Web-scale” organizations (notably Google, Amazon and Facebook) practice continuous deployment, with real-time production monitoring and a lot of exploratory testing taking place in production as well.
Here are a dozen articles about Web-scale continuous deployment in the enterprise
After reading these links it should be clear that Continuous Delivery is Yet Another Mainstream Software Methodology, just like Agile and Waterfall. Please reach out to me on Twitter if if you think you can still make the case — in light of this evidence — that Continuous Delivery is still “untried” or especially risky ;-)
One other thing to take note of here: almost all these articles are old. Some were written 2 years ago or more. Rapid iterative development may or may not work for a specific organization — but the general problem of implementing Continuous Delivery is clearly well-solved and has been so for a while now.
Google practices Continuous Delivery
- At Google, 15,000 engineers work from the HEAD revision of a single Perforce trunk.
- 50% of the code will be changed in any given month.
- Google’s test infrastructure is legendary and they’ve written a comprehensive book about how they perform QA while continuously releasing.
- They’ve also put a lot of effort into scaling Perforce.
Here is a fantastic deep-dive into Google’s deployment pipeline:
Amazon practices Continuous Delivery
- At Amazon, new code is deployed to production at a staggering rate of once every 11.6 seconds during a normal business day.
- That’s 3,000 production deployments per day.
- They’ve invested an enormous amount of time and money into creating an architecture that facilitates small, orthogonal, frequent code pushes.
Here’s Amazon’s Jon Jenkins breaking down their deploy stats at the O’Reilly Velocity conference:
Facebook practices Continuous Delivery
- At Facebook, each of 5,000 engineers commits to trunk HEAD at least once a day and the code at trunk HEAD is pushed to production once daily.
- Facebook has no dedicated QA team. All responsibility for testing rests with the software engineers.
- They’ve invested heavily in infrastructure that provides zero-downtime deployment at Facebook scale.
Here’s Facebook’s release manager, Chuck Rossi, going into detail about how Facebook engineers balance their experiments against the risk of breaking some fundamentally important part of the site: