--- title: "Failure happens: Artur Bergman on 365 Main" description: "Artur Bergman welcomes Jesse Robbins to O'Reilly Radar with a 2007 post on the 365 Main outage, opening the operations series on designing for failure." canonical: "https://jesserobbins.com/mentions/failure-happens-artur-bergman/" jsonld: "https://jesserobbins.com/mentions/failure-happens-artur-bergman.json" image: "https://jesserobbins.com/_astro/avatar.CCwnroSW_FTaXX.jpeg" doc_version: "1.0" last_updated: 2026-02-20 slug: "failure-happens-artur-bergman" outlet: "O'Reilly Radar" author: "Artur Bergman" date: "2007-07-25" url: "http://radar.oreilly.com/archives/2007/07/failure_happens.html" type: "Article" excerpt: "Artur Bergman welcomed Jesse Robbins to O'Reilly Radar with this July 2007 post on the 365 Main datacenter outage, which opened the operations series that became Jesse's column and set the case for designing for failure." quote: "I want to welcome Jesse Robbins to Radar. We are kicking off a series of articles exploring the depths of the dark and forgotten world of operations." quoteAuthor: "Artur Bergman" tags: "Jesse Robbins, O'Reilly Radar, Artur Bergman, Web Operations, DevOps, 365 Main, Failure" --- # Failure happens ![Artur Bergman](/images/entities/artur-bergman.png) O'Reilly Radar by [Artur Bergman](https://www.linkedin.com/in/crucially/) · July 25, 2007 · Article > "I want to welcome Jesse Robbins to Radar. We are kicking off a series of articles exploring the depths of the dark and forgotten world of operations." Originally published in **O'Reilly Radar** on July 25, 2007. [View on Web Archive ↗](http://radar.oreilly.com/archives/2007/07/failure_happens.html) Artur recruited me to write for O’Reilly Radar, and this is the post he did it in. It went up in the middle of the 365 Main outage, when a power failure took a large slice of the web offline, and he used the moment to make an argument I had been making for years: that failure is a design constraint, not an exception, and that operations had been treated as the poor cousin of engineering for too long. He closed it by introducing me and opening a series. _Artur Bergman, “Failure happens,” O’Reilly Radar, July 25, 2007. The original URL no longer resolves; the text below is preserved from the [Internet Archive capture](https://web.archive.org/web/20080216011341/http://radar.oreilly.com/archives/2007/07/failure_happens.html)._ What an exciting day, as services for hundreds of thousands of users and millions of readers disappeared from the internet. In a stunning but unsurprising event, a repeated power cycling caused by a blown power station disrupted the 365 Main datacenter, causing them to lose all power to two colocation rooms. I jokingly refer to 365 Main as the “Web 2.0” datacenter; of course, there is nothing Web 2.0 about the datacenter itself. But it does host a remarkable number of such properties, including Craigslist, Technorati, and Red Envelope. Someone could make a lot of VCs cry by taking it out, or so the running joke goes. And ironically enough, this morning 365 Main (together with Red Envelope) put out a press release announcing 2 years of 100% uptime; one may also note that they have now removed the press release from their site, as [http://www.365main.com/press\_releases/pr\_7\_24\_07\_red\_envelope.html](http://www.365main.com/press_releases/pr_7_24_07_red_envelope.html) returns a file not found. We can draw the conclusion from today’s event that the return to the mainframe world is precarious. Maintaining our own systems is harder than paying someone else to do it; trading on this fact, we expect reliability in exchange for cost. Having entrusted our data, services, and therefore income to these companies, we trust them to keep it safe and available. When this trust is broken by extended downtime, clearly the situation is not a sustainable solution. But that doesn’t mean that running your own operations is the solution either. A while ago I wrote about disaster recovery with Amazon Web Services, and some of you astutely pointed out that problems can occur if you run your own operations. And indeed they can! And they do! Clearly the need for disaster recovery plans are just as important if you are hosting it yourself. These plans exist on a different continuum, affecting not just operations but also your entire organisation’s response to disasters. Planning response to disasters cannot be avoided, particular in San Francisco’s position between two quite dangerous fault lines. An earthquake is a question of when, not if. Are the startups ready for this? How long will we expect them to be gone? Possible answers to these questions look grim given a cursory analysis of today’s events. Several of the world’s largest websites went down. None of them were ready for a datacenter outage. None of them had backup datacenters or fail over that worked. None even had a coherent strategy for communicating the situation to the rest of the world. Google, Yahoo, Amazon and EBay are companies that have invested money in redundancy. Because of this investment they can build systems that survive outages, a significant comparative advantage for these companies. This advantage presents a tradeoff I am willing to make: even if I don’t trust Google with my data’s privacy, I do trust that they will keep my data safe. (Or at least safer than I or most people could.) But the tradeoff doesn’t pay off if the company cannot do that, and my trust in a whole swath of companies dropped completely. I wrote earlier about the discussion about the Open Source Developer Toolkit and the need for frameworks, tools and patterns that improve scalability. But in light of recent events, perhaps reliable and fault tolerant systems are more important than scalable ones. Events like this make people suddenly realise that mysql replication is not an adequate disaster recovery strategy. Or that tight coupling between the database and the app might cause a bit of a problem when your database might be moved to another city. Or that the memcache you are accessing suddenly is several milliseconds away. There is a small group of people who know this; some may call them jaded and cynical, others may call them experienced. But the vast number of developers and operations people are completely out of their depth here. I want to welcome Jesse Robbins to Radar. We are kicking off a series of articles exploring the depths of the dark and forgotten world of operations. Operations has too long been hiding in the shadows, treated as the poor cousin to engineering and development. It is time to share our horror stories, experiences and ideas in hopes of collectively pushing our profession to a higher level. _Originally published by O’Reilly Radar on July 25, 2007. Reposted here as part of the record of how the operations column began._ ## Also Mentioned - [Artur Bergman Founder and former CEO of Fastly](https://github.com/artur) - [O'Reilly Radar O'Reilly Media's technology blog, the publication Jesse wrote for and was covered by most often in the DevOps era](https://www.oreilly.com/radar/) ## Topics Jesse RobbinsO'Reilly RadarArtur Bergman [Web Operations](/topics/web-operations/) [DevOps](/topics/devops/) 365 MainFailure ## More Mentions [O'Reilly Radar You Become What You Disrupt October 2, 2007 What happens when a disruptive technology wins a platform play and inherits the obligations of the system it replaced? I wrote this on O'Reilly Radar in 2007. The questions are still open. “You become what you disrupt. What changes occur when you win a platform play, when you go from disruptive technology to a public utility?”](/mentions/you-become-what-you-disrupt-oreilly-radar/) ## Sitemap See [sitemap.md](https://jesserobbins.com/sitemap.md) for the full list of pages on this site.