Jesse Robbins

What the Apple Neural Engine and Google's TPU tell us about the next decade of inference

Jesse Robbins — Mon, 08 Jun 2026 00:00:00 GMT

A deep-research report on neural processors. What they are, how they differ from GPUs, and what Apple and Google have built across the Apple Neural Engine (ANE) and the TPU lineage.

Originally: Lab, essay, 2026.

How to run vector search on a website in the browser with no server (like I do)

Jesse Robbins — Sat, 06 Jun 2026 00:00:00 GMT

Here's how to do semantic search on your website without a server. Open the search box and the page downloads a small embedding model and a 160 KB vector index; from then on every query is answered locally in the browser, fused with keyword search. What I built, what to install.

Originally: Lab, code-story, 2026.

Good Fences make Good Agents: sandvault + /sv skill (part 1 of n)

Jesse Robbins — Sun, 24 May 2026 00:00:00 GMT

A simple solution to working with agents that cannot be trusted to run 'ls'. Install with brew. Works in seconds. I added a few features.

With Patrick Wyatt, Mike McQuaid, Eran Sandler, Jesse Vincent, and Nat Torkington for code and editorial review.

Originally: Lab, code-story, 2026. Read the original.

Sunil Dhaliwal, Founder of Amplify Partners, on Lessons from 27 Years in VC

Suffiyan Malik w/ Sunil Dhaliwal — Thu, 21 May 2026 00:00:00 GMT

Fastly was an introduction through a former CEO of mine, Jesse Robbins … Jesse said whenever Artur Bergman quits his job, he's going to start something. I would back that guy … Artur is fucking brilliant. Believe what he says … And that is probably the single biggest success in the history of the firm.
Sunil Dhaliwal, on We The Builders

This is an extraordinary two-hour conversation with my friend Sunil Dhaliwal, who has proven to be one of the most successful investors in our space. He goes far and wide here, from the dot-com bubble to the AI boom, early-stage firm investment dynamics, and why he built Amplify in the first place. It is a really great interview, and worth a listen or a watch.

Along the way he tells some stories about a few introductions I made for him during a particularly exciting time in infrastructure. Sunil led my Series B at Chef while I was CEO and also still Chair of the Velocity Conference. Sunil made it clear he wanted to run (and pay for) the bar at my speaker parties, and expected me to introduce him to the best people.

That night I introduced him to my friends Artur Bergman and Simon Wistow who were just founding Fastly. This, of course, turned out to be one of Sunil's (and my) best investments. (Sunil also met the founders of Datadog at Velocity the same way.) Definitely a trick I learned to use myself over the years.

One painful moment from this part of the interview was Suffiyan being surprised that "O'Reilly did conferences". Yes, they did... and we all miss them.

Long and excellent conversation for anyone who is interested in venture. Worth every minute. Go listen.

Originally: We The Builders, Podcast, 2026. Read the original.

The Seed 100: The Best Early-Stage Investors of 2026

Mon, 18 May 2026 00:00:00 GMT

I look for founders who want to build the operating system for entire industries. This requires extraordinary taste, grit, drive, and a vision for the future.
Jesse Robbins

Truly honored to be included for a second time in the Business Insider Seed 100 alongside so many investors who are dedicated to supporting founders.

Grateful to the founders, my partners and co-investors, and the many people who build extraordinary companies.

Originally: Business Insider, Article, 2026. Read the original.

Call a savepoint

Jesse Robbins — Thu, 07 May 2026 00:00:00 GMT

The pull you feel toward 'just one more iteration' is not evidence that the work needs more time. It is evidence that the schedule of small wins has trained your brain to expect another one.
Jesse Robbins

Hey... feeling depleted after working with AI?

It is not a flaw in you. It is the predictable result of two things: this work has the reward loops of a game and the cognitive cost of supervising it is much higher than it looks. Together they are an addictive loop to keep you in the seat past the point where you should have walked away wondering why you feel hollowed out by a day that produced so much output.

I learned this after hitting 2.3B tokens for the week & #11 on the tokenmaxxing leaderboard. Everyone doing extraordinary things with AI right now, the tokenmaxxers, running multiple billions of tokens a month across Claude, Codex, and Gemini... they all know this. They are the ones building the most ambitious systems, shipping the most code, producing the most remarkable output.

Working with AI engages the same dopamine machinery as slot machines, video games, and social feeds. You write a prompt, you wait, something arrives. Sometimes it is brilliant. Sometimes it is wrong in an interesting way. Sometimes it is wrong in a boring way. The loop is short: five minutes, sixty minutes, the cache windows you are unconsciously syncing to and the reward is highly variable. That is the textbook recipe for what they call "compulsive engagement". It is not an accident that you keep going past the point where you should stop. The loop is doing exactly what loops like this do.

The pull you feel toward "just one more iteration" is not evidence that the work needs more time. It is evidence that the schedule of small wins has trained your brain to expect another one. Video games and slot machines feel productive in exactly the same way. The difference is that with AI, sometimes you really did just ship something which makes the pattern harder to see and harder to break.

Call a savepoint. Have the agent capture the current state. All your open threads in a structured note and then close the conversation. Walk away. When you come back, fresh model and fresh you, you load the savepoint and continue. You can call for a savepoint at any time. You need one and so does your coding assistant. The work will be there when you get back.

Originally posted on LinkedIn on 2026-05-07.

Originally: LinkedIn, LinkedIn, 2026. Read the original.

Musical AI Raises $4.5M: Jesse Robbins Joins Board

David Nusbaum — Wed, 04 Feb 2026 00:00:00 GMT

Originally: Los Angeles Times, Article, 2026. Read the original.

What I look for when I invest

Alex Wittenberg — Wed, 28 Jan 2026 00:00:00 GMT

Alex Wittenberg ran a long conversation with me on airCFO's Funded podcast covering my personal and current firm investment criteria, how to get on a VC radar, and the founder mistakes I see killing pitches before they start.

Leverage and Focus

The biggest advantage early-stage founders have is speed. Anyone on the team can make a decision and ship it the same day, without a process or an approval chain to clear. A good plan executed immediately beats a perfect strategy debated for months. The founders who spend that advantage on real customer work are the ones who build real businesses. The ones who spend it on optics like big-name partnerships do not.

"We really, really, really like to see traction: early, repeatable, measurable, personal, where you can say, this is why this customer is using it."

The Big Lie About Investment Stages

There is a big lie in venture: the published stage gates that say you can raise a Series A at $500K ARR. Those numbers are the floor for getting a coffee meeting. The bar for a term sheet sits much higher, and founders who plan against the low end set themselves up for heartbreak.

Pre-seed investing means testing a concept with referenceable design partners at zero revenue. Seed means proving there is a repeatable business, typically $500K to $1.5M ARR. AI is compressing how fast founders reach those numbers, but the bar for a term sheet has not moved with it.

Market Size Is the First Filter

Of the four buckets we evaluate (team, market, product, traction) market is the one I will not bend on. A great team with real traction in a small market is still a pass for us. The cleanest example I have is LaunchDarkly. Every VC dismissed feature flagging as too small a market, and Edith Harbaugh reframed it from a developer tool to a business configuration platform. Same product, bigger market, and the round happened.

"We can love you. We can believe you are building the best product imaginable. We can see early traction that's super compelling. And if it's not a big enough market, it's not a venture scale business."

Co-Founder Dynamics

I pay close attention to how co-founders treat each other in unguarded moments. The Gottman frame of "turning towards" responses is the one I use: even in conflict, are the founders still in each other's corner? The Sanity founding team (Magnus, Simon, and the rest of the co-founders) has worked together for over a decade and visibly likes working together. That is the version that holds up when things get hard.

The inverse is devastating: "When founder relationships fall apart, it is the most value destroying, destructive thing that can happen to you professionally."

Founder Grit

The trait I care about above everything else is personal grit. The ability to grind in the absence of positive feedback, on purpose, because the work matters to you. I trained as a firefighter, and when someone calls 911 the team shows up and solves the problem no matter how big it gets. That is the same job description for a founder CEO.

It does not require a Type A personality. Mitch Hill, the CEO I hired at Chef, was a quiet introvert who had previously scaled a company from zero to a billion in revenue, and his resolve was extraordinary.

How to Get on VCs Radar

The playbook: attend startup and VC events, engage with the community, get a warm introduction, or fill out the application form. What does not work: cold LinkedIn requests, AI-generated SDR emails, and launching into an elevator pitch the moment you meet a partner.

"All I need to believe at first is that you're the kind of person I want to talk to for an hour or more."

The Investment Process

My personal standard is that every founder should leave thinking it was one of the best VC meetings they have ever had, regardless of outcome. They should actually benefit from the meeting. The pet peeves that kill momentum for me: view-only pitch decks (send downloadable artifacts), confusing SAFEs with term sheets, and trying to manufacture urgency with fabricated interest.

This is an Adventure!

A personal mantra "a journey with certain adversity, uncertain outcome, and good companionship."

Originally: Funded Podcast (airCFO), Podcast, 2026. Read the original.

Musical AI Bags $4.5M to Scale AI Attribution Tech: Jesse Robbins Joins Board

Mandy Dalugdug — Tue, 13 Jan 2026 00:00:00 GMT

Musical AI's attribution technology is essential infrastructure that will enable and accelerate every media-focused AI product.
Jesse Robbins

Originally: Music Business Worldwide, Article, 2026. Read the original.

Investing in Vibrant Labs: AI Agent Simulation Infrastructure

Jesse Robbins — Wed, 03 Dec 2025 00:00:00 GMT

Vibrant Labs opens a new frontier in AI infrastructure: production-grade, RL-ready simulation and verifier-driven evaluation built for long-horizon agents.
Jesse Robbins

I wrote the Heavybit announcement when Vibrant Labs joined the portfolio. Shahul ES and Jithin James, the team behind Ragas.io, are building production-grade simulation environments and verifier-driven evaluation for long-horizon AI agents.

The operational gap is specific. Agents take on multi-step planning and execution, and developers need safe, repeatable environments to measure and improve agent behavior before deployment. Vibrant Labs ships three pieces of infrastructure:

RL-ready simulation worlds for training agents against realistic scenarios.
Verifier-driven evaluation that returns explanatory feedback alongside pass-or-fail signal.
Training pipelines that turn evaluation data back into supervision signal.

"Vibrant Labs opens a new frontier in AI infrastructure: production-grade, RL-ready simulation and verifier-driven evaluation built for long-horizon agents."

Originally: Heavybit, Article, 2025. Read the original.

What Investors Look For in AI Startups: Builders with Taste

Marco @ Shift — Thu, 12 Jun 2025 00:00:00 GMT

At Shift Conference Miami 2025, Marco interviewed me on stage in a rapid-fire format covering what makes a developer-tools startup investable, where AI disruption is actually happening in developer workflows, and why the arc of software engineering keeps bending toward natural language.

Test and Taste

My personal investment filter is visceral: does the product change how you think about the problem from the very first time you see it? Two portfolio examples from my time at Heavybit: Tailscale, after building VPNs throughout my career, the first time I used it "made me mad. It was so easy." And Continue, which made me switch from vi to VS Code, "which is sort of a crazy shift for an old school systems engineer."

The broader criteria I use: builders with great taste, desperate to solve a problem they need to fix in the world, who can get early developer communities excited. Open-source software with a sophisticated business model. Tools that are hard to build without specialized knowledge.

Still Automating Toil

We are at the very beginning of this. Code generation and basic test automation are low-hanging fruit: "the work that none of us ever really want to do in the first place. That's what we're automating right now. We're automating toil."

The real opportunity lies ahead: observability, CI/CD workflows, UI/UX generation on the fly, customization, personalization. Every category of developer tools is getting changed, but the surface covered so far is small.

The Next Abstraction Layer

I think about AI code generation as the latest step in computing's longest arc: from hand-wired circuits to assembly language to increasingly natural programming languages. Each layer of abstraction moved humans closer to expressing software concepts in natural language, with 10x–100x productivity gains each time.

"We've arrived at the point where we're able to express software concepts and have another layer of abstraction that generates code, just like every cycle prior to that point."

The result is that more people can build software, which creates more complexity to manage, and more developer tools to invest in. "Every time we add a few million more people to building things, well, they get more complicated. There's more of it to manage. And so there's more things for me to build and invest in."

Agents Are Just Another Developer

The reframe I keep coming back to: "Agents are just another type of developer." They need the same things human developers and previous "non-human users" like Google's crawler needed: good documentation, clearly defined APIs, and affordances that make interfaces easy to program against.

There is a direct line from accessibility (screen readers) to SEO (Google's crawlers) to agents: each is a non-human user that benefits from the same infrastructure investments. The agentic layer builds on prior work and drives personalization, dynamic UI, and adaptive experiences.

"Does it mean that any of the work we were doing before goes away? No. Hopefully it just causes people to do more."

Originally: Shift Conference, Video, 2025. Read the original.

The Future of Dev Tools Is Autonomous: Engineers Will Become Fleet Generals

Marin Pavelić — Thu, 22 May 2025 00:00:00 GMT

Marin Pavelić at Shift Magazine surveys how developer experience is evolving from desktop-era IDEs to AI-collaborative environments where engineers increasingly manage fleets of autonomous agents. The piece walks through the category (Cursor valued at $9 billion, Windsurf acquired by OpenAI for $3 billion) and asks what happens when AI operates autonomously inside development workflows.

Designing for agents

Marin pulled three lines from the Shift Miami panel "Investing in Dev Tools in the Age of AI." The first:

"If you're building software now, you're not just designing for humans. You're designing for agents, too."

This reframes developer experience as a dual-audience problem. The tools, APIs, and documentation we build now serve both human developers and the AI agents working alongside them. It echoes the longer argument from my Shift Conference interview that agents are "just another type of developer" who need the same affordances as human users.

Open source reignited my joy

The second line is personal:

"Because of this open-source ecosystem, I started writing code again. It felt joyful."

The tools finally feel right. I came back to hands-on coding because Continue and the AI-native open-source community made it worth doing.

Collaboration over fighting

The third line is the one I keep returning to:

"Experiencing joy in collaborating with tools instead of fighting them may be the most important change."

The article places this alongside the broader frame I have been making everywhere: delegation as the new automation, engineers as fleet generals orchestrating AI agents, and open-source tools like Continue providing the transparency and control that make the collaboration possible.

Originally: Shift Magazine, Article, 2025. Read the original.

Experimentation, Causal Inference, and AI: Sean Taylor of OpenAI and VC Jesse Robbins at Data Council 2025

Jesse Robbins and Sean Taylor — Tue, 01 Apr 2025 00:00:00 GMT

AI provides an opportunity to radically improve how we do things.
Jesse Robbins

I sat down with Sean Taylor of OpenAI at Heavybit to preview the Data Science and Algorithms track for Data Council 2025. Our conversation focused on experimentation, causal inference, and the practical frameworks data teams use to drive product and business decisions.

We covered the featured speakers: Hadley Wickham on generative AI in data science workflows, Timothy Chan of Statsig on experimentation at scale, Joe Powers of Intuit on Bayesian A/B testing, and Bryan Bischof of Hex on ML engineering. I framed the conference as a rare opportunity for isolated data science practitioners to validate their work by "peeking over each other's shoulders."

Originally: Heavybit, Article, 2025. Read the original.

Next in Tech Ep. 197: Data Pipelines for AI

Eric Hanselman — Tue, 10 Dec 2024 00:00:00 GMT

Data pipelines are having a DevOps moment, starting with a cultural and technical shift toward continuous integration and delivery.
Jesse Robbins

Eric Hanselman hosted me on S&P Global's Next in Tech to argue that enterprise AI success depends on data pipeline quality more than model scale, and that the infrastructure patterns for getting it right already exist in the DevOps playbook.

The DevOps moment for data

Enterprise AI data infrastructure is in the same place software delivery was a decade ago. DevOps introduced continuous integration and delivery to replace brittle manual deployment, and data teams are now building pipeline automation for model training, evaluation, and governance. The cultural shift matters as much as the technical one. Organizations have to treat data delivery with the same rigor they eventually brought to code delivery.

Start small, iterate locally

I advocate starting with smaller models and localized datasets. The approach is the same one I have run since Chef and Amazon: prove the pattern works at small scale, measure what matters, then expand. Enterprises that try to solve the data pipeline problem at full scale first build fragile architectures and burn through budgets before they learn what actually works.

Pipeline quality as competitive advantage

The companies that win at enterprise AI are the ones that build the best data pipelines. Clean data in. Reliable inference out. Governance and cost controls at every stage. The data delivery layer is where the next generation of developer tools and infrastructure companies will be built.

Originally: S&P Global Market Intelligence, Podcast, 2024. Read the original.

The Data Pipeline is the New Secret Sauce

Jesse Robbins — Mon, 16 Sep 2024 00:00:00 GMT

The biggest challenge emerging is building and operating the infrastructure both for creating and running the data pipelines to build, manage, and maintain a robust, secure body of proprietary data.
Jesse Robbins

The data pipeline is the bottleneck in enterprise AI.

I wrote that at Heavybit in September 2024. The argument: enterprises that will get real work out of AI are the ones whose pipelines produce a secure first-party dataset and stay correct as the underlying systems change. Buying access to a model is one part of shipping AI into production. The harder work is operational.

I have lived this pattern. The conditions enterprises were grappling with in 2024 looked 1:1 with what I saw at Amazon as DevOps practices took shape. Slow-to-evolve organizations, regulatory pressure, customer risk, the same handful of operators carrying the weight. The starting point is different this time. DevOps had to drag the field toward continuous integration and delivery. Data pipelines for AI start there. I made that argument at Data Council earlier in 2024, before writing it up.

In September 2024, roughly 40% of enterprises surveyed said they had deployed an AI program or were actively exploring one. Microsoft was reporting 53,000 organizations using its AI offerings via Azure. Gartner had 87% of "mature organizations" carrying dedicated AI teams. These programs are not bought off the shelf. They produce an artifact: the internal dataset. It is the end result of a complicated toolchain run by a team that does this work full time. The data pipeline is that artifact's production line. It is the thing that does or does not compound.

The piece maps four inference hosting models and four enterprise maturity phases. Most of the value is in the phase model. Phase 1 looks like real experimentation against a hosted API. Phase 2 is the moment teams realize they have to stand up an internal pipeline to extract durable value from the use cases they have proven. Phase 3 is cost shock. The bill from the API provider becomes the line item that gets the AI program in front of the CFO. Phase 4 is the only one that requires judgment, because the right answer for one workload is rarely the right answer for another. Mature enterprises end up running mixed inference configurations and treating optionality as the durable asset, not any single hosting choice.

The operational point underneath the taxonomies is the one I care about. A data pipeline is a continuous process that begins when the model ships to production. It needs the same monitoring, validation, and team discipline as any other production software, plus the security and privacy work that stops personally identifiable information from leaking on the way through. Without those practices, enterprises either fail to build their internal dataset at all or ship real business risk: privacy leaks, model drift, the cost of re-training on bad data. Cost-control work belongs in this phase too, including model merging and mixture-of-experts as alternatives to retraining on the entire dataset.

In 2025, Heavybit backed Recce. I joined the board. CL Kao and his team are building data validation for the moment a pipeline change ships to production, which is exactly the discipline this article said enterprises would have to develop. Recce is the practical answer to a question this piece could only frame: how do you know your pipeline is still correct after the change?

The biggest challenge emerging is building and operating the infrastructure both for creating and running the data pipelines to build, manage, and maintain a robust, secure body of proprietary data to train, fine-tune, and orchestrate LLM operations, and for running inference, the actual process of models running calculations on inputted data.

The piece runs through two frameworks.

Four inference hosting models.

Hosted API. Calling OpenAI, Anthropic, and the rest. The provider absorbs the cost and operational burden of running large models. Enterprises pay in tokens.
On-device edge. Smaller models running locally, often on high-end laptops, sometimes paired with three-billion-parameter open-weight checkpoints. Lower latency, better data locality, an unclear scaling story for larger teams.
On-premise data center. Everything behind the firewall. Most enterprise IT workloads moved off-prem years ago for total-cost reasons. AI inference is following the same path outside heavily regulated workloads.
Off-premise cloud via third-party data center. The managed-inference layer. Resembles traditional cloud computing more every quarter. Introduces network latency and dependency on the provider's reliability posture.

Four enterprise maturity phases.

Phase 1, off-the-shelf cloud start. Most enterprises begin here, against a hosted API. Data science and operations teams focus on identifying valuable use cases. The hosting question is abstracted by the provider contract.
Phase 2, scaling what works. Teams have a pipeline that is good enough to deliver value on specific workloads. They harden privacy posture for the data types and jobs that matter. The bill starts to register.
Phase 3, cost shock and optimization. API spend hits a number that gets noticed. Teams reassess: continue paying for hosted inference, or invest in an internal model and the inference configuration to run it. Tooling investments here include pretraining datasets, data filtering, model evaluation.
Phase 4, specialization. Mature teams run mixed configurations and stop treating any single hosting choice as the answer. They prize optionality over vendor lock-in. They consider model merging and mixture-of-experts as alternatives to retraining on the entire dataset every time.

The piece names a specific operational point. The pipeline itself is the artifact, and building one demands operational effectiveness plus the security and privacy work that stops PII from leaking. Without that, enterprises fail to build their internal dataset at best, or ship real business risk from privacy leaks, poor model performance, and the cost of retraining on bad data.

The article also references Chaoyu Yang of BentoML on specialized AI systems built for specific use cases as a likely source of durable advantage. Specialized infrastructure is where serious teams have ended up in 2025 and 2026.

Read the full article at Heavybit.

Originally: Heavybit, Article, 2024. Read the original.

AI Investor Jesse Robbins on NYSE Floor Talk

New York Stock Exchange — Mon, 05 Aug 2024 00:00:00 GMT

I am focused on investing in pre-seed and seed companies using AI to enable new ways of writing software, of managing and deploying the software and infrastructure that powers everything.
Jesse Robbins

Full Interview Transcript

I invest in early stage AI companies focused on developer tools, productivity, and infrastructure. I like to think I invest in companies that power the companies that you hear of every day.

Q: Jesse, as you look to the next 12 months what are your priorities right now?

I'm focused on investing in pre-seed and seed companies, often where there is a strong AI component to enabling new ways of writing software, of managing and deploying software, of managing infrastructure that kind of powers everything. The last 12 months have shown us there's this incredible shift occurring, and so the next 12 months is going to be about helping our companies scale and grow and hire and raise more Capital and to reach a lot more Enterprise companies that are our amazing customers.

Q: Jesse tell me who are some of your companies?

Some of our larger late stage companies or companies like PagerDuty that was listed on NYSE. Other ones like LaunchDarkly and Snyk and Sanity, as well as smaller companies that are sort of at the pre-seed and Seed stage including Shipyard and Radar (a New York company), Mobot (another New York company), and many others.

Originally: NYSE, Video, 2024. Read the original.

Jesse Robbins Named One of the 30 Most Successful Early-Stage Startup Investors

Ben Bergman, Samantha Stokes, Rebecca Torrence, and Leena Rao — Wed, 17 Jan 2024 00:00:00 GMT

Business Insider named me to its 2024 list of 30 most successful early-stage startup investors. Grateful to the founders, the practitioner community Velocity and Chef were built with, and my partners at Heavybit.

Originally: Business Insider, Article, 2024. Read the original.

Heavybit Welcomes New Member: Continue

Jesse Robbins — Tue, 14 Nov 2023 00:00:00 GMT

I'm excited to welcome our newest portfolio company, Continue, which gives software engineers the power to streamline their development process using large language models (LLMs) and hit flow state faster and longer.
Jesse Robbins

I wrote this announcement when Continue joined the Heavybit portfolio. Continue integrates large language models directly into VS Code and JetBrains, so engineers do not have to copy code between applications, make edits, and re-prompt separate AI tools.

The reason we backed the team is the IDE-native, open-source design. Developer tools succeed when they meet engineers where they already work. Continue does that.

Originally: Heavybit, Article, 2023. Read the original.

Cloud Native StartupFest 2023

Jesse Robbins — Mon, 06 Nov 2023 00:00:00 GMT

Open source is not a business model. Open source is a movement. We're still figuring out the business models.
Jesse Robbins

The full transcript of my opening remarks is below. Erica's data slides and the founder panels are on the CNCF YouTube playlist.

Originally: CNCF / KubeCon, Panel, 2023. Read the original.

Generative AI in DevOps and Incident Response: What the Experts Actually Think

Jesse Robbins — Thu, 12 Oct 2023 00:00:00 GMT

GenAI is good at confidently delivering text that is pleasant to read, but not always complete, or correct.

By late 2023, every vendor had an AI pitch for incident response. Autonomous remediation. AI-powered root cause analysis. MTTR cut in half, automatically. I went to the people who actually run systems at scale to find out what was real. I interviewed Nora Jones (founder of Jeli, formerly Slack and Netflix), Jeremy Edberg (Amazon Alexa, formerly Netflix and Reddit), Mandi Walls (PagerDuty, formerly Chef), and Brent Chapman (Great Circle Associates, formerly Google and Slack). What came back was more specific, and more cautionary, than the vendor hype.

Key Themes

Where AI actually helps: summarization

The consensus is narrow but real. AI is useful for incident summarization: drafting post-incident reports, generating status updates, and catching latecomers up to speed during an active incident. These are tasks where a plausible first draft is valuable, and where a human will verify before it matters.

Nora Jones frames it precisely: "I think what we really want to do is use AI to get people more curious about what's happening in their incidents." The value is in the question it opens, not the answer it provides.

Why Hallucination Disqualifies Autonomous Remediation

The harder truth is that AI hallucinates, and in incident response, hallucination is dangerous. Brent Chapman's observation cuts to the bone: "LLMs are sometimes wrong, but never uncertain." A system that is confident and occasionally wrong is the worst possible profile for taking autonomous action in high-stakes, time-compressed situations.

Jeremy Edberg draws the line explicitly: "Right now, GenAI is something of an advisory tool. We're not to the point where we trust it enough to take actions." The human-in-the-loop is not a temporary limitation waiting to be engineered away. It is the right architecture given current reliability. Every AI-powered RCA and autonomous remediation pitch in 2023 ran into this constraint. The serious practitioners all landed in the same place.

The Junior Developer Analogy

Multiple practitioners reach for the same frame independently: working with AI tools is like working with a very junior programmer. You still have to check everything. You still have to ask whether the output is right, useful, and complete. Alert correlation, automated runbooks, AI-assisted postmortem generation all require the same oversight.

Edberg uses this frame to address career anxiety directly: "If you are good at logic and want to learn how to reason about computer systems, software engineering will still be a great place to be." AI will accelerate development, particularly for engineers who already know how to think about systems. It will not replace that judgment.

The AI SRE Career Question

Mandi Walls draws a useful distinction for roles: SREs, who are deeply integrated with engineering practice, will see more direct benefits from AI coding and observability tools than operators in more traditional roles. The productivity gains land where the work is closest to the code. MTTR reduction through AI-assisted triage and automated runbooks accrues to engineers who understand what the AI is actually doing.

The overall career picture is a shift toward strategic work and away from the mechanical. Practitioners who understand systems reasoning will find AI accelerates their output. Those optimizing for task execution without deeper systems understanding face more pressure.

The Human Judgment Floor

What runs through every perspective is a shared conviction: AI does not yet have contextual or collaborative judgment. It cannot read the room during an incident, weigh the organizational history that shapes which escalation matters, or decide what to prioritize when the situation is genuinely novel. Autonomous incident investigation fails at exactly the moments when the incident is most unusual, which are precisely the moments when it would matter most.

Nora Jones is direct: "I don't think generative AI is going to fix the incidents for you." Someone still has to verify, lead, and decide. The tools make some of that work faster while keeping the responsibility in human hands.

DevOps is dead? Nope, it is maturing ft. Jesse Robbins

Rob Zuber — Fri, 07 Apr 2023 00:00:00 GMT

Organizations evolve like cities. You start with a few shacks in the woods. Eventually you have enough at stake that you need building codes, fire codes, a fire department, and someone who actually tests the sprinklers.
Jesse Robbins

Originally: The Confident Commit, Video, 2023. Read the original.

What to Know About the Modern Incident Response Lifecycle

Andrew Park — Fri, 11 Nov 2022 00:00:00 GMT

Teams only get good at this when they embrace the whole process and each of its steps.
Jesse Robbins

Andrew Park's guide for Heavybit on modern incident management quotes me on why teams only get good at this when they treat the full lifecycle as a single discipline. The piece walks readers through the practical implications: normalize incidents by talking about them often, be honest about the state of the infrastructure, and treat the practice itself as the source of mastery. The line he pulled from our conversation is the one I have been saying since Amazon: skip any step in the cycle and you never fully develop the muscle for any of them.

Originally: Heavybit, Article, 2022. Read the original.

Fireside Chat with Jesse Robbins and Kolton Andrus • Failover Conf 2021

Gremlin — Thu, 29 Apr 2021 00:00:00 GMT

At Gremlin's Failover Conf 2021, Kolton Andrus and I sat down for a fireside chat on GameDay's origins at Amazon, how deliberate failure injection evolved into the discipline the industry now calls chaos engineering, and where reliability practices needed to go next.

We covered the early days of breaking production systems on purpose, the cultural resistance you hit when you first propose simulating catastrophic failures, and how those exercises changed the way Amazon thought about availability. We traced the shift from ad-hoc failure testing to systematic chaos engineering platforms, and dug into what separates teams that recover well from incidents from teams that struggle.

The session closes on the future of SRE: what engineering leaders should prioritize and how the chaos engineering community can continue raising the bar on production resilience.

Originally: Gremlin, Video, 2021. Read the original.

The Seed 100: The Best Early-Stage Investors of 2021

Melia Russell, Michael Haley, Candy Cheng, and Margaux MacColl — Fri, 02 Apr 2021 00:00:00 GMT

Robbins is the right investor to call in an emergency.
Business Insider

Business Insider named me to the 2021 Seed 100, a ranking of early-stage investors built with Tribe Capital from data on about a thousand seed investors.

Originally: Business Insider, Article, 2021. Read the original.

An oral history of #hugops: How tech's first responders built a culture of empathy

Tom Krazit — Thu, 25 Feb 2021 00:00:00 GMT

I've got to change the way that I approach this entirely and make it safe to experiment.
Jesse Robbins

A note from Jesse

Tom Krazit wrote this for Protocol in February 2021. Protocol shut down the following year, so I am preserving it here. This piece captures something real. The people in this story built a culture that did not exist before we showed up, and #hugops started as a sarcastic joke to troll one of my best friends. That it became a genuine signal of empathy across the entire industry is one of the things I am most proud of. The tools mattered. The movements they created mattered more.

When something breaks on the internet, the people who know how to fix it just want to give their colleagues a hug, even if they're a rival.

The #hugops community in its happy place: the Velocity conference. Photo: James Duncan Davidson/O'Reilly Conferences

In almost every profession, it seems like there are two types of workers: the ones who get the glory, and the ones who do the essential work no one ever sees, unless something goes wrong.

In enterprise computing, those overlooked people are known as operations engineers. They're the ones who keep the rickety Rube Goldberg machine that is the modern internet from falling to pieces every day, while their glamorous counterparts, software developers, get to bask in the recognition that comes with shipping a new feature or creating a new service.

A little over 10 years ago, a group of operations-oriented engineers decided they were fed up with software developers who didn't care if their code actually worked, so long as it shipped. They were tired of abuse at the hands of management who forced their teams to be on call 24/7 with little to no internal support, let alone recognition.

Those engineers created the Velocity Conference in order to band together: to share their lived experiences including the intense pressure to keep Fortune 500 companies up and running, to discuss tips and tricks for navigating tricky problems and to come together as a community of people who know what it's like to be at the bottom of the food chain when everything has gone to hell.

That community sparked a revolution known as DevOps, the idea that software developers and operations professionals needed to work together more closely to support the ever-more complex task of running sophisticated software over the internet. Big companies such as Amazon and Google started to develop the operations career path with incentives and rewards parallel to those on the development side, while acknowledging that these people needed support from the highest levels of the company to do their very difficult jobs.

And out of this community came a Twitter hashtag, an in-group signal to their peers during the most stressful moments of their careers that a team had their back. When a major cloud service goes down, such as during Slack's early January outage, most people on Twitter see an opportunity to vent their frustration and score points at the affected company's expense.

At those moments, the people who know what it takes to keep these services afloat spread a hashtag: #hugops.

This is the story of the engineers who keep the cloud running, and how they created their own culture of empathy when nobody else cared.

Life of a sysadmin

Adam Jacob, CEO of The System Initiative, co-founder and former CTO of Chef: Systems administrators, a now almost basically nonexistent job title, were not the most beloved humans in the technical world. We didn't get a lot of respect.

We were sort of in the same bucket like secretaries; we had a System Administrator Appreciation Day. The people who do the stuff you don't see get appreciation days because, by definition, it means I'm not being appreciated every other day.

[One team leader] took us and my whole team, there were like 20 systems administrators, and he took us all out for beer on System Administrator Appreciation Day. And he sat down with the pitchers of beer and the first thing he said was, "Here's your guys' beer. Too bad none of you are smart enough to be engineers. Cheers."

My response to that was to just be mean to him.

Jennifer Davis, developer relations manager, Google: I don't know if you ever heard of the BOFH sysadmin? There was this mentality of like, how cruel and evil can we be to our users.

Werner Vogels, CTO, Amazon: I think sysadmins mostly came out at a time when most companies were buying software. Traditionally at those operations, [software] development is on one side. Then there's this wall, and you throw software over the wall; and you don't care anymore.

Tim O'Reilly, founder, O'Reilly Media: There were all the, effectively, software janitors who were cleaning up after them. And the software janitors were kind of going: That doesn't really work.

Jesse Robbins, a former firefighter and present-day hugger, at the Velocity conference in 2010. Photo: James Duncan Davidson/O'Reilly Conferences

Kolton Andrus, co-founder and CEO, Gremlin: At Amazon, I was one of 10 people that was paged when the website went down. And I took and managed the resolution of those calls from the side of the freeway next to my motorcycle because I had to pull over, call in and handle it immediately; it couldn't wait 10 minutes until I got home.

There was an Amazon Christmas party that I was at where I got a page, I had to run out to my car, get my backpack, come into a war room, sit down and resolve an incident before going back to the party. There's a lot of work that the engineers and the ops folks do behind the scenes, a lot of thankless work to help make sure things go well and get fixed.

Nathen Harvey, developer advocate, Google: What do we celebrate in technology? We celebrate new; new features, shipping new capabilities that we're delivering to customers. And we get angry when systems fail. Basically what you're saying is: We celebrate the developers, and we recognize the operators when everything goes to shit. That's not great.

Jacob: I sat in a room early on at Chef with a bunch of video game developers that were running the U.S. operations for one of the biggest video games of all time. And their boss sat across the table from them, and to my face, in front of them, said, "My guys aren't smart enough to learn Ruby." If you just interviewed system administrators from that era, 100% of them have that story.

Jesse Robbins, founder and executive chairman of Orion Labs, former co-founder and CEO of Chef: In operations, we always missed the launch party, because we were too busy in the data center or locked in an office looking at green screens trying to support a launch. We were never there for the fun part. We were always the ones that were giving up our nights and our weekends, and we're powerless to actually improve things.

When emergencies are a day job

Andrus: The on-call training I received at every company amounted to: "Here's your pager, good luck. You're smart, you'll figure it out."

Harvey: I remember a conversation I had with Ron Vidal, who is a firefighter in the San Francisco area. And one of the things he said to me was: "A firefighter has never, in their life at work, responded to an emergency. If your house is on fire, that's an emergency for you, but for the firefighters, that's their job."

Robbins: I'm a firefighter by training, and when I joined Amazon in 2001, "master of disaster" was my title. I realized that the way that we were running operations at Amazon was fundamentally not going to scale and that we needed a process and almost a cultural overhaul.

I began turning Amazon into a fire department. I literally took the sort of incident management principles that we used in the fire service and turned that into what we call GameDays and Scale Days, using essentially the incident command system in order to support people through the various ways of thinking when the red light is on.

Davis: A lot of what operations is like encourages this heroism: You have to do everything to keep it running and just throw yourself into it. It's not sustainable work. It's not great, it's terrible, and you're celebrated when you save the day but the reality is, it's terrible. It harms your relationships, and it harms your health and just frames how you work with other people.

Nora Jones, founder and CEO, Jeli: We're shifting towards a kind of a time where people see issues and incidents as a symptom rather than a cause of something, and trying to understand the bigger system that is playing out in those organizations.

Robbins: I owned availability at Amazon, and when I say owned it, I was sort of a tyrant, and ran it very aggressively. There was this big outage that we had [in the early 2000s], and there was a person early in their career who was literally shaking when I walked into the room because they were so afraid of what was going to happen.

I realized, "I've got to change the way that I approach this entirely and make it safe to experiment, safe to do these other things, to not have this punitive model and approach." It was seeing that person's face where I'm like, "Oh, I'm not the fire department, I'm like a bad guy. I'm being a villain."

Davis: If we reduce the heroism, we can reduce burnout.

Robbins: There is an ethos that came from all of that early work that recognizes how it is important to be kind to each other. And part of what I did early on at Amazon was create a culture of safety. You only get to do really big, great things when you're able to take great risks safely.

A meeting of like minds

John Allspaw, founder and principal, Adaptive Capacity Labs: These topics deserved an entire conference. I guess it was less that it deserved an entire conference, but more that a few folks convinced Tim O'Reilly to actually do it.

O'Reilly: They said, "Look, we need a gathering place for our tribe." We had done that before, for these various open-source communities. A lot of these things are rooted in communities, and so if you can figure out what community you want to bring together, you start by bringing them together.

Allspaw: What [the Velocity Conference] did was important, because it was a signal that operating software and understanding how things are running and anticipating things that can go wrong could be considered distinct from software development.

Artur Bergman, co-founder and chief architect, Fastly: What we were doing was just as critical as writing the code. If you can't run the code, it has no value.

Vogels: The time to develop software is actually quite small [compared] to the time that you have to operate it. So even though you may be building something complex, it may take a year or two years [to build], you may have to operate it for many, many more years to come.

Jacob: Velocity was like the first time that there was a non-academic place where everybody who is doing that work could get together. And it was like, well-funded and pretty. It wasn't like we were meeting up in the American Legion hall or whatever. It was a fucking conference.

Allspaw: We were finding this pretty significant common ground. For many, many years, they didn't have a place to put these ideas, or even labels or terms or vocabulary to talk about the dread, or actually sort of outright terror, that can come with, "shit's broken, and we have no idea."

So there's this lived experience of, "OK, you're with your colleagues and shit's broken and you don't have 100% clarity, but you've got a couple of good ideas that look sort of fruitful. And okay, so it seems like we should connect this thing to this thing and restart this other thing? We should do it in that order. What do you think about that?" You'd see this in IRC, we didn't have Slack back then.

This conference exists because we've got this shared experience with incidents and the general challenge is not just responding to incidents, but trying to work out how to prevent the ones in the future. And it's difficult work.

Time for a hug

Jacob: I'm a very huggy person. And so I hugged all of those people [at Velocity], all the time. Because it was happening to this group of people who ... their work environment was not a place where you got a fucking hug.

Davis: We're building complex systems that include the people. And so how do we handle the unpredictable stress of complex systems? When you think about hugs, hugs are used to reduce pain. They're used to show that you care and they're used to reduce fear.

Jacob: So Artur Bergman was, is?, a particularly salty dude. He swears as much as I do, maybe more, and he's Swedish, so like when he swears, it's better.

Artur is not a person who was huggy. Artur would maybe suffer a hug from me, or suffer a hug from John [Allspaw]. At some point, John made a T-shirt that is the earliest I remember of the #hugops-y thing, and on the back of it it basically says, "Hug Artur Bergman."

Bergman: [During one Velocity] I gave a keynote and then [Adam] gave a keynote where he told people to hug me, and I was not aware that he had said that. During the day around the conference, random people started coming up and hugging me, which was, you know, quite uncomfortable, especially because I had no idea why. And so I ended up hiding for the rest of the day until I finally found out at the end of the day why this was happening.

Artur Bergman, who is not the naturally huggy type, at the Velocity Conference in 2010. Photo: James Duncan Davidson/O'Reilly Conferences

Jacob: It was a very special moment in time where there was this very high degree of camaraderie, there was this really high degree of familiarity.

Allspaw: Capturing this real dread, these pretty scary, pressure-filled situations, sort of fueled that you're part of this tribe. I don't know who you are, but you're here and you're talking and, so having that common ground is what I think genuinely got people [to be] like, "Can I give you a hug?"

Jacob: We knew people at all of those [big tech companies], right? And so as everybody starts to know each other, when like, Facebook would have an outage, you'd use the #hugops hashtag and you were like literally talking to your people.

Robbins: It's not a surprise that what began with a sarcastic joke to troll one of my best friends became an idea that a lot of people have rallied around because it reflects the world that they're building continuously, that they're continuously improving.

Davis: It's just a message of caring. It's a shorthand to show that I have empathy for where you're at, because I'm going to be there at some point. And I hope you show me that empathy too, but also, you know what? You are not alone.

The future according to #hugops

Jones: What we're really seeing right now is a shift in the software industry and us buttoning up and understanding that our software is quite critical. But the pressures that people are under to write this software is a lot.

Take Slack. During that outage, they had all just come back, it was the Monday that everyone came back from New Year. I can't imagine being in that office, because you're just getting used to writing code again, you're just getting used to deploying things again, and then all of a sudden, all the world is signing on to Slack at the exact same time. It makes total sense that they had an incident that day.

I think part of what we're seeing from the "learning from incidents" community is just a shift in thinking and software to say, "OK, they didn't do something wrong. Something happened that made sense for them to do what they did," and kind of allowing for that conversation to happen.

Robbins: That shift happened because we made it happen, in part because we simply made it so clear that large businesses, large organizations cannot succeed with this kind of outdated enterprise software legacy mindset. To be always on, to be always available, you're always improving, and that means dealing with failures and enabling rapid change.

I think we're in the second chapter now of a movement that has new leaders emerging and evolving. It's not a part of the MBA curriculum yet, but it soon will be.

Andrus: Inertia within an organization is hard. You can get a team of 10 to pivot quickly. You're a startup, you've got 100 people, you can change your process. You've got 10,000 engineers, it's a lot harder to get everyone to change how they've done things the last decade or two.

Harvey: The #hugops movement and the ideas behind it really speak about, "How do we build more empathy for the other humans that we interact with every day?" In my mind, it certainly goes beyond technology.

As a society, we could take some real lessons from this: How do we just have better empathy for and respect for the work and the way that people show up in the work that they do, and the fact that you know, everyone is out there doing absolutely the best that they can with what they have? I think that's really, really important.

Davis: Every time I hear "NoOps" or "NoDev," I'm like, "Nooooo...." Because when people are saying that the robots and automation are gonna take over, that doesn't think through all of these complexities that humans are really great at.

Yes, reducing the toil is so great. And we can have these conversations about how to balance out what availability is, and how much I'm going to spend on resolving things, and have those kinds of conversations separate from like, "We're gonna just eliminate all the humans because humans make mistakes." Humans make mistakes building the stuff that then we're relying on; we need humans as the safety checks.

Bergman: If you have a long outage, you need to care about your people and their sleep schedules, and the fact that they have to eat. And by day four or five, if you didn't do that, you're just gonna have a bunch of really tired and grumpy people who are going to make more mistakes.

Andrus: I did enjoy at Amazon and at Netflix the approach of, "You should know how your software behaves." If you've written software and deployed it and then you're turning a blind eye to it, that's just not good engineering.

Davis: What is so fascinating is that the next generation isn't putting up with this negative stuff. They're setting the expectations and they're very vocal about what they want their work environments to be like and how they want to work.

John Allspaw values the shared experience of the Velocity Conference. Photo: pinar@pinarozger.com

Jones: We need to be asking different questions and we need to give more people seats at the table. I've been at way too many organizations where the incident was just the [site reliability engineers] in the room. It should have had marketing in the room, it should have had PR in the room, it should have had customer service in the room, it should have had leadership in the room. But it's thought of as kind of an SRE issue, like SREs have to prepare for any type of situation that gets thrown their way.

I was at one organization a while back where we launched a Super Bowl commercial. And we had some bumps when we launched the commercial, but the SRE team didn't get a ton of notice that the commercial was happening, I think it was either same-day notice or a couple days beforehand, and that was not really mentioned in the post-incident review.

Andrus: The flip side of #hugops is I do think there is responsibility that should be held to the leadership of those companies. We're empathetic to the engineers that are dealing with the situation they have, but in part that's because leadership isn't prioritizing their actions, or resilience and reliability in the same way that they prioritize some of their product efforts.

Allspaw: As my colleague Dr. Richard Cook has said, we shouldn't be surprised that these systems go down. We should be more surprised that they stay up as often as they do.

Bergman: We took a job that was critical to running the world's largest websites and the internet, that was kind of under-appreciated, and turned it into a movement, modernized it with DevOps, and gave those individuals career paths.

Jacob: Who gets credit when you see a beautiful car? You don't give credit to the mechanics. You're like, "Man, those guys at Porsche really make beautiful cars." You might know, like, one legendary mechanic in the history of great mechanics.

But that's why it's so persistent: because the mechanics know the mechanics.

Originally: Protocol, Article, 2021.

A Developer's View Into Blockchain Network Architecture

Konstantin Richter — Mon, 27 Aug 2018 00:00:00 GMT

Konstantin Richter at Blockdaemon moderated this panel at Heavybit, and I sat with Brian Behlendorf (Executive Director, Hyperledger), Jed McCaleb (co-founder, Stellar), and Jake Craige (Lead Developer, Coinbase). I brought the developer-tools-investor lens to the conversation: how does this perform in production, how do you monitor it, what does the surrounding tooling need to mature.

We covered the real tradeoffs of decentralization, node management, metrics and monitoring requirements, standardization of distributed systems, and the regulatory picture. The questions I kept returning to were practical ones for the developers building on blockchain infrastructure, the same operational and reliability concerns I had been applying since Amazon and through the DevOps movement.

Originally: Heavybit, Panel, 2018. Read the original.

Incident Management for Operations (foreword by Jesse Robbins)

Rob Schnepp, Ron Vidal, Chris Hawley, Jesse Robbins — Sat, 01 Jul 2017 00:00:00 GMT

This groundbreaking book is the foundation to building an effective operations culture for organizations of any size, with systems of any complexity, and failures of any severity.
Jesse Robbins, from the foreword

My foreword to the book

This book originated from an argument during my first year as Amazon's "Master of Disaster," as I began applying the incident management and operations practices I learned as a firefighter to improve Amazon's overall reliability and resiliency. I vividly remember facing a room full of scowling engineers and managers who were saying, "I get that these ideas work for firefighters, but do you really think they can work at internet speed?"

The answer, of course, was yes. The systems and best practices developed over decades of managing complex emergency incidents, where seconds count and lives are on the line, work just as well for managing complex incidents for technology organizations. Over the next few years, my team and I used these techniques and systems to help transform the culture and technology of what is now one of the greatest engineering and operations organizations in the world.

When I left Amazon, it was clear to me that as the world was becoming increasingly connected and distributed, people would come to depend on the new technology we build and systems we run as part of their daily lives. It was also clear to me and a group of passionate peers that there were too few people with the knowledge and experience to build and run these systems at scale. My friend, Artur Bergman helped me found the O'Reilly Velocity Performance & Operations conference to organize, develop, and spread our emerging and critical professional discipline.

As Velocity grew, I started sharing my work with friends and mentors in the Fire Service. I am fortunate to have worked with and been trained by some of the most experienced and respected incident management experts in the world, and I asked them to help build and expand on what I had started.

I convened the first "Web Ops/Fire Ops" summit on a beautiful day at Artur's loft in San Francisco. Attending from "the internet" were Artur Bergman (Fastly/Wikia), John Adams (Twitter), Johnathan Heiliger (Facebook), Pedro Canahuati (Facebook), Simon Wistow (Fastly), and Christopher Brown (Amazon/Chef/Microsoft). Attending from the "Fire Ops" side were the authors of this book: Chris Hawley, Rob Schnepp, and Ron Vidal.

After a few hours of sharing backgrounds, "war stories," and a lot of laughter, it became clear to everyone that there was both the need and opportunity for a tech-oriented incident management training program. Chris, Rob, and Ron formed Blackrock Partners and began consulting with large companies on how to improve their operations. Since then they have worked with dozens of tech companies, trained thousands of new responders, and reviewed hundreds of incidents as they help companies "work like a fire department at internet speed."

This book is the first publicly released product of their exceptional work, and is the essential foundation for building technology and organizations that people can depend on. I hope you use it.

As we say in the fire department, "See you at the big one!"

— Jesse Robbins Founder and CEO, Orion Labs, Inc.

Originally: O'Reilly Media, Other, 2017. Read the original.

'Star Trek Communicator Startup' Sets Out to Build a World Powered by Voice

Wired — Wed, 21 Jan 2015 00:00:00 GMT

Wired covered our rebrand from OnBeep to Orion Labs and the idea behind it: a world powered by voice. The Star Trek communicator comparison followed Onyx from the day we launched it, and we stopped fighting it. Tap the badge on your chest, talk to your team, anywhere.

The product was a wearable, but the company was about voice as an interface for people who work with their hands and eyes busy. Firefighters, nurses, field crews, warehouse teams. That conviction came straight from my time in emergency services, and it is why I cofounded the company.

Originally: Wired, Article, 2015. Read the original.

This Startup Thinks Your Workplace Needs Wearable Walkie-Talkies

Davey Alba — Wed, 05 Nov 2014 00:00:00 GMT

Wired covered the launch of Onyx, the wearable walkie-talkie we built at OnBeep. Onyx was a $99 device that clipped to your clothing, paired with your phone, and gave a team instant push-to-talk voice over Wi-Fi or cellular, at any distance.

The idea came from my years as a volunteer firefighter. On an incident, voice is the coordination layer: one button, instant connection to the whole team, eyes up the entire time. Most workplaces ran on radios that had not changed in decades, or on phones that bury urgent communication under apps. We wanted to give frontline teams the immediacy of a radio with the reach of the internet.

OnBeep became Orion Labs a few months later, and the work continued from there.

Originally: Wired, Article, 2014. Read the original.

Building Companies that Devs & DevOps Teams Love, And Avoiding Expensive Mistakes

Jesse Robbins — Tue, 25 Jun 2013 00:00:00 GMT

This talk covers the expensive mistakes I made founding Chef and the patterns I keep seeing developer-tools founders repeat. The gap between building something technically impressive and building something teams adopt at scale is where most companies die. Great technology does not sell itself.

The lessons cover product positioning, developer experience, pricing, and the go-to-market traps that catch technical founders. I made most of these mistakes firsthand building Chef from an open-source project into an enterprise infrastructure company. Learn a dollar of lesson for every one you spend in failure.

Originally: Heavybit, Article, 2013. Read the original.

Tim O'Reilly on Why We Started the Velocity Conference

Tim O'Reilly — Mon, 17 Jun 2013 00:00:00 GMT

Back in 2006, Debra Chrapaty, then VP of Operations for Windows Live (later CIO at Zynga, and now CEO of Nirvanix) made a prescient comment to me: "In the future, being a developer on someone's platform will mean being hosted on their infrastructure." As it often turns out, things don't work out quite as planned. A few months later, Amazon announced EC2, and it was Amazon, not Microsoft, that became the platform whose infrastructure startups chose to host their applications on. But Debra certainly nailed the big idea!

I wrote a blog post about that conversation, entitled Operations: The New Secret Sauce, which included the statement "Operations used to be thought of as boring. It's now ground zero in the computing wars." Jesse Robbins, then "Master of Disaster" at Amazon and later co-founder and CEO of Opscode, told me that everyone in operations at Amazon printed out that blog post and posted it in their cubicles. Operations had been a relatively low-status job. Jesse told me that was the first time anyone had made a strong public statement about how important it was becoming.

As a result of that post, Jesse, Steve Souders, and a group of others came to me the following year and said "We need a gathering place for our tribe." That gathering place became the Velocity Conference, now in its sixth year. We chose to include not just web operations, but also web performance and the emerging field of "DevOps," the development model for applications hosted in the cloud.

This seems to be part of the secret sauce of some of our most successful events: the recognition that it's not just about technology but the people who put it into practice. At the heart of conferences like Velocity and Strata are new job descriptions, new skills, and new opportunities to grow careers and companies. That's also why we increasingly think of these events not as conferences but as gathering places for communities. Technology matters. The people who put it into practice matter more.

Originally: O'Reilly Radar, Article, 2013.

Jesse Robbins on the Rise of DevOps (InfoQ Interview)

Harry Brumleve — Thu, 17 Jan 2013 00:00:00 GMT

Harry Brumleve interviewed me for InfoQ in January 2013, the point where DevOps was crossing from a niche conversation into a category the broader industry was paying attention to. We covered the transition from traditional operations into DevOps, what infrastructure as code actually meant in practice, and how the operating-model work I had done at Amazon could be made replicable for everyone else.

DevOps is a reorientation

DevOps is a reorientation of how organizations build and run software. The walls between development teams writing code and operations teams running it are failure modes: points where accountability breaks down and blame accumulates instead of improvement. At Amazon, we treated operations as competitive advantage instead of cost center. The question I kept getting in 2013 was how to make that culture replicable beyond the hyperscalers.

Infrastructure as code

By 2013 Chef was the clearest example of what infrastructure as code meant in practice. Servers are instances of a declared state that code creates, modifies, and destroys. The same version control, testing, and review workflows software teams use for applications now apply to infrastructure. A change to a server config becomes a pull request, audited and reproducible, instead of an undocumented action by a sysadmin at 2am. Teams deploy infrastructure the way they deploy code, with confidence the outcome will match the spec.

Velocity and the practitioners

Velocity, which I cofounded with Steve Souders at O'Reilly in 2007, was the room where fierce competitors, Google, Amazon, Microsoft, Facebook, shared what they knew about running complex systems reliably. The practitioners there developed shared vocabulary, shared standards, and shared intuitions about what good operations looked like. By 2013 that work had a name.

Originally: InfoQ, Article, 2013. Read the original.

Q&A: Ex-Amazon 'Master of Disaster' Jesse Robbins on the Power of 'Relentless Optimism' in Startups

Jeff Dickey — Sat, 27 Oct 2012 00:00:00 GMT

When you're trying to change the way big organizations work, a lot of people say no a lot. Rather than try to fight them, you've got to find a way to make them say yes. Being a force for awesome in the world is finding ways to say yes.
Jesse Robbins

Jeff Dickey caught me in 2012, mid-stride at Opscode. I had been a technology builder since high school, stepped away from tech to train as a firefighter and EMT, and walked back in through the door at Amazon on August 20, 2001. The Q&A traces that path and lands on the operating principle I had been using inside Amazon and was now using to build Opscode.

On 9/11, I woke up in a hospital after emergency surgery and watched the day unfold on a television. That was when I understood that the operational skills I had been training in the fire service translated directly to a technology organization that thousands of people depended on. As I told Jeff: "I decided I'm going to figure out a way to mix these two worlds together." That decision became Master of Disaster, GameDay, and eventually Chef.

On driving change in large organizations: "When you're trying to change the way big organizations work, a lot of people say no a lot. Rather than try to fight them, you've got to find a way to make them say yes. Being a force for awesome in the world is finding ways to say yes."

On startup life, my advice in 2012 was the same advice I give founders now: "If you're struggling, recognize it's going to be this way forever." The work does not get easier; you get better at it.

Originally: GeekWire, Article, 2012. Read the original.

Resilience Engineering: Learning to Embrace Failure

Tom Limoncelli — Wed, 12 Sep 2012 00:00:00 GMT

You can't choose whether or not you're going to have failures — they are going to happen no matter what — but you can choose in many cases when you're going to learn the lessons.
Jesse Robbins

Three teams. Three companies. Same answer arrived at independently. That is the part of this article that still matters.

Tom Limoncelli moderated the discussion and wrote it. Tom Limoncelli is extraordinarily accomplished. His books shaped our profession as it evolved. He pulled three of us into the same room: Kripa Krishnan from Google, John Allspaw from Etsy, and me from Amazon. None of us had read each other first. We had built versions of the same discipline because the problem was the same.

Kripa Krishnan ran Google's program, which they called DiRT. By 2012 she had been doing it for about six years. Her exercises ran 72 to 96 hours, hundreds of engineers around the clock, war rooms staffed by about fifty rotating volunteers. The details I have never forgotten are the failures that surfaced. Google brought down a network in São Paulo and watched the links die in Mexico, because nobody knew the dependency was there. A data center where the machines refused to come back online because they had run out of DHCP leases. Kripa had Ben Treynor as her executive sponsor. Ben went on to create a similar program at Google called Site Reliability Engineering (SRE). Without that air cover, none of this happens.

John Allspaw was running technical operations at Etsy after stops at Salon.com, Friendster, and Flickr, where he was engineering manager. John brought the academic frame to the conversation. He was the one who put Erik Hollnagel's four cornerstones of resilience on the table: anticipation, monitoring, response, learning. He was the one who said the thing the industry took years to absorb. He had announced publicly that he would not fire an engineer for taking down a site he was responsible for. He described the substitution test Etsy used in postmortems. Pull in an uninvolved engineer, give them the same context, ask what they would have done. Almost every time, the answer is the same command. The problem is not the person. The Brooklyn Bridge line is his too. You do not shut down the whole bridge because one lane is out.

What I described was GameDay at Amazon, which I had started in 2003 and 2004, when horizontal scalability across unreliable hardware was not yet a settled idea. We were feeling our way. The exercises were not simulated. We powered facilities off without notice and let the systems fail naturally. In one of them I used my fire service training to script a simulated fire down to the minute, with operators posing as facilities staff calling operations with updates. I had executive support I never took for granted. Werner Vogels was my exec sponsor. Jeff Bezos thought the idea was interesting. The Amazon ops team was the village that actually did the work. Tim O'Reilly later gave this community a place to find each other in public, the Velocity Conference. I cofounded Velocity with Steve Souders after I left Amazon, and later passed the torch to John Allspaw.

The discipline framework did not come from tech. I trained as a volunteer firefighter with the Seattle Fire Department before I stepped away from tech. Fire service teaches you that you do not get good at incident response by having an opinion about it. You get good by drilling, and then drilling more, and then drilling again under conditions you do not control. Build the muscle continuously, because the only way to find the failures hiding inside a complex system is to trigger them on purpose, on your terms, before they trigger themselves on theirs.

That is the line I gave Tom that has traveled the furthest. You do not get to choose whether you have failures. You get to choose, in many cases, when you learn the lessons.

The convergence is what still holds up. Three organizations, no shared playbook, all arriving at the same answer. Stop trying to prevent failure. Build the muscle to absorb it. Never blame the human at the end of the chain for the system around them. What happened after this article ran is the next part of the story.

Originally: ACM Queue, Article, 2012. Read the original.

Jesse Robbins on DevOps as Business Alignment

Jez Humble — Fri, 17 Aug 2012 00:00:00 GMT

The role of operations is the role of enabling as much awesome as you can.
Jesse Robbins

Jez Humble interviewed me for Thoughtworks in 2012 as part of his continuous delivery video series. The conversation lays out how I was framing DevOps then: business alignment first, collaboration second.

DevOps as business alignment

The starting point is the incentive problem. Developers create value when code reaches production. Operations teams were traditionally rewarded for keeping production unchanged. Most companies treated both groups as support functions instead of seeing delivery and operations as part of the same value stream. DevOps is the shift where operations becomes a way to help the business move faster without giving up reliability.

"Code that is written and not deployed is worthless."

Operations as enablement

I walked Jez through my own pivot from gatekeeper to platform builder. At Amazon, I blocked a launch because I was worried about availability. Neil Roseman overrode me. The site went down for a couple of days. The stock and order rates went up. That was when I realized the better role for operations was to make launches safer and more repeatable, not harder to do.

"The role of operations is the role of enabling as much awesome as you can."

Chef and infrastructure as code

Chef sits inside the larger move toward programmable infrastructure. EC2, Rackspace, OpenStack, VMware, and private-cloud APIs made it possible to provision infrastructure through software. Chef was the glue that let teams describe the desired state of infrastructure and applications together. Infrastructure as code is the moment infrastructure becomes part of the application: configurable, repeatable, testable, close enough to the product that teams can move without waiting on manual provisioning.

EC2 and the cloud operating model

I told Jez the story of EC2's origins inside Amazon, including my own first reaction: I tried to block it. Chris Pinkham and Christopher Brown built the project away from Seattle, in Cape Town, and I gave them a hard time about exposing what I considered my operational perimeter to the public internet. EC2 changed who could ask for infrastructure, how fast they could get it, and what operations teams had to become.

Simple services, fast feedback

The cloud architecture advice from the back half still reads cleanly. Build small services. Keep APIs simple. Push complexity up the stack. Avoid monoliths that force every scaling problem into one place. Prefer rough consensus with running code over elaborate first designs. If the system is split into clear services, the teams can own, operate, and improve those services. Architecture is a way of making responsibility visible.

Originally: Thoughtworks, Video, 2012. Read the original.

Changing Culture & Being a Force for Awesome

Jesse Robbins — Thu, 28 Jun 2012 00:00:00 GMT

Don't fight stupid. Focus on where you can make more awesome.
Jesse Robbins

This is my Velocity 2012 talk. I had been giving versions of this to smaller rooms since 2011 and earlier at Amazon. Velocity was the room where I tried to say it cleanly to the people who were going to take it back to their own organizations.

The framework

Five steps, each building on the last.

Start small. Pick the smallest project with receptive people. Call it an experiment. Do not trigger the organizational immune system.
Create champions. Get your boss on board first. Then spread credit as widely as you can. Let other people feel ownership of the change.
Use metrics to build confidence. Find one number that supports your change, time from commit to deploy, cost of an outage, and use it ruthlessly to build the case.
Celebrate successes. Tell the story with data. Be positive about people. Leave room for resistors to come around without losing face.
Exploit compelling events. When the site goes down or a compliance mandate lands, use the moment to push for the change you have been building toward.

The Katrina lesson on permission

I tell a story from my deployment as a task force leader during Hurricane Katrina. A volunteer kitchen staffed by anarchists was feeding thousands of people a day, and FEMA kept trying to shut them down because no one would say who was in charge. The fix was simple. Make every volunteer a "site director." When FEMA asked who was in charge, someone would answer "I'm a site director," and FEMA would deliver supplies.

The lesson stuck with me: most of the time when people are saying no, what they really mean is they do not know how to say yes. I used the same move at Amazon by typing "Master of Disaster" into a form as my job title. It stuck.

The rule

The talk keeps returning to one line: don't fight stupid. Focus on where you can make more awesome.

Originally: O'Reilly Velocity Conference, Video, 2012. Read the original.

Jesse Robbins on the State of Infrastructure Automation

Timothy M. O'Brien — Fri, 11 May 2012 00:00:00 GMT

Tim O'Brien interviewed me for O'Reilly Radar in May 2012, while I was Chief Community Officer at Opscode. We talked about Chef's evolution from a tool the early adopters loved into something enterprise customers were buying at scale, and about the broader shift toward treating infrastructure as code.

Originally: O'Reilly Radar, Article, 2012.

5 Pivotal Documents in the Evolution of the DevOps Movement

Klint Finley — Tue, 03 Apr 2012 00:00:00 GMT

Operations is a Competitive Advantage (Secret Sauce for Startups!)
Jesse Robbins (cited document)

Originally: SiliconANGLE, Article, 2012. Read the original.

The Convergence of DevOps

John Willis — Fri, 30 Mar 2012 00:00:00 GMT

John traces three threads that ran in parallel in the late 2000s and converged into the DevOps movement: Agile Infrastructure, Velocity, and Lean Startup. Patrick Debois, Andrew Clay Shafer, and the agile system administration community on one strand. The Velocity Conference at O'Reilly on another. Eric Ries and the Lean Startup community on a third.

The Velocity strand at O'Reilly grew out of a 2007 Radar post I wrote applying the technical-debt frame to operations and a Tim O'Reilly follow-up titled "Operations: The New Secret Sauce." DevOps Days Mountain View 2010, organized by Patrick Debois, John Willis, Andrew Clay Shafer, and Damon Edwards, was the first US DevOps Days. I was honored to be there.

GameDay: Creating Resiliency Through Destruction

Jesse Robbins — Tue, 20 Dec 2011 00:00:00 GMT

You don't choose the moment, the moment chooses you. You only choose how prepared you are when it does.
Jesse Robbins

This is my USENIX LISA 2011 talk on GameDay. Boston, December 2011. I had been running these exercises at Amazon since 2003 and talking about the methodology at Velocity, at the early DevOps Days events, and in dozens of smaller rooms before this one. The USENIX talk is the cleanest single-stage recording of the methodology end to end.

I opened with the story my fire chief told me on my first day of firefighter academy. He said when you go home tonight and tell your neighbor you are becoming a firefighter, something changes. At 2:00 in the morning when their kid starts choking, they will not dial 911. They will pound on your door with a slumped over kid, looking to you to do something. "Welcome to the other side of 911." And then he said the thing that set the arc of my entire career: "You don't choose the moment. The moment chooses you. You only choose how prepared you are when it does."

That is operations. We are the ones they call. We are the ones people look to when things are broken.

The core argument of the talk is simple. Resilience is a property of your entire system, and the system includes people, culture, processes, applications, infrastructure, and hardware. People and culture are the most important part. That is weird for a room full of us who tend to shy away from human interaction, but it is the truth. GameDay is about changing people to be resilient.

I created GameDay at Amazon after joining in 2001 while testing for the Seattle Fire Department. My title was "Master of Disaster." I owned website availability for every property that bore the Amazon name. I realized the way we were running operations was not going to scale, and I began adapting fire service incident management processes, training, drilling, and fire prevention concepts to make Amazon operate like a fire department. I substituted the word "management" for "command" and created an incident management system that was a word-for-word adaptation of the fire department's incident command system. Werner Vogels was my internal executive sponsor.

The methodology has three stages. First, preparation. You identify and mitigate risks. You walk through the system and find the stupid stuff. The oil barrels next to the core database servers. The single points of failure nobody talks about. This alone reduces failure frequency and improves recovery time.

Second, you run the drill. This is where most organizations fail. They do a lightweight tabletop exercise, declare victory, and never go further. If you never go all the way to a full-scale live failure exercise, you do not get the confidence that comes from responding to a stressful situation at full speed. As a firefighter, I have been through countless live fire drills. The fire will kill you. People die in training. I can only be an effective firefighter having fought fire. You also discover that systems and processes you thought would work do not. Cell phone systems have single points of failure. Nobody has a printout of everyone's phone number. The most basic things only surface under real stress.

Third, you expose latent defects. These are the impossible failures. The ones that cannot happen until they do. They sit underneath the waterline of your systems, and you cannot discover them any other way. They are immediately recognizable in hindsight. "Oh, we totally should have known there was a dependency on that developer's desktop." You do not get to choose whether you have latent defects. You do get to choose when you discover some of them. Run the exercise off-peak rather than finding out on your most important traffic day.

The progression matters. You start small. You work with the smallest group of developers who are receptive, and you break something that is a little scary but not too scary. Think of the kid with the fire hose. These hoses produce 80 to 100 pounds of back pressure. You do not hand a recruit a high-pressure fire line on day one. You give them a garden hose, a small pan fire, and you let them succeed. Then they tell everyone how awesome it was. You build on those successes.

Then you move up to the full-scale live fire exercise. You pick the worst survivable scenario. I recommend a full data center power-down. It will terrify everyone. Everyone will hate you during the planning phase. It is going to be good for them. You give them a couple months notice. You tell them the date, the time, the facility. They have months to remediate. And then the week comes, and people ask if you are really going to power it down. Yes. You are really going to power it down. You can slip a date, but never cancel. Otherwise no one will ever believe you again. I always power it down.

The first time will be a disaster. You will learn more from that one exercise than from years of tabletop reviews. Probably you will learn that your database masters do not come back up after an EPO reset. You will be glad you chose when to learn that.

The reason all of this works is the OODA loop, the observe-orient-decide-act cycle that John Boyd described from fighter pilot training. In any crisis, you follow a predictable response. You observe what is going on. You orient yourself based on your training, your experience, whether you have been exposed to situations like this before. If you have never been through a full-scale outage, you will lock up. There are predictable failure modes. I can tell you what you will do. The only reason I am different is I have been through it a lot.

GameDay became an internal competitive advantage at Amazon. One team would say, "Go ahead, rip it out. We can power stuff off all day." The other teams wanted to get there. That is the confidence-driven currency for change John Allspaw talks about, where people say "We are better because this happened," instead of the crisis-driven kind where a bad outage opens a short window for sweeping fixes.

Every large-scale web operation has since learned some version of this or perished. Google adopted it. Engineers who had been at Amazon brought the practice to Netflix and built Chaos Monkey, which randomly terminated instances in production. They expanded it into the Simian Army and later Chaos Kong for regional failover testing. Facebook, Yahoo, and dozens of others built their own programs.

AWS turned GameDay into a core operational practice and eventually a customer-facing program. The Well-Architected Framework now defines "game day" as a formal reliability concept. AWS Fault Injection Service is a managed chaos engineering service that automates the kind of fault injection I was doing by hand. The FIS team runs their own game days using the service they built. That is the right kind of recursion.

The practices I described in this talk, controlled fault injection, pre-announced failure exercises, progressive escalation, and blameless post-incident review, became core patterns in what the industry later formalized as chaos engineering.

Originally: USENIX, Talk, 2011. Read the original.

Meet 2011 TR35 Winner Jesse Robbins

MIT Technology Review — Fri, 02 Dec 2011 00:00:00 GMT

From the MIT Technology Review TR35 honoree page

Category: Internet & web Year Honored: 2011 Organization: Opscode Region: Global Focus: Fault-tolerant online infrastructure

Biography

Jesse Robbins applied for two jobs in 2001: a Seattle bus driver position and a backup systems engineer role at Amazon.com. Amazon's offer came first, beginning a decade of work on how web companies operate complex server and software networks at scale.

Drawing from his background as a volunteer firefighter, Robbins brought crisis management principles to infrastructure design. He recognized that massive global operations inevitably experience failures and built systems to withstand them safely. Rather than preventing failures, he made Amazon resilient to them through architectural fault tolerance and live operational drills that tested teams by temporarily taking entire data centers offline, without affecting customer experience.

After leaving Amazon in 2006, Robbins shared his methodologies through blogging. In 2007, he cofounded Velocity, now an annual conference where major competitors openly discuss infrastructure management.

Robbins cofounded Opscode in 2008. The company's flagship product, Chef, is an open-source framework for cloud-based infrastructure automation. One notable application involved scientists using Chef to deploy a 10,000-processor supercomputing cluster in 45 minutes on Amazon's cloud, completing complex protein-binding research in eight hours, then shutting down operations, all at a fraction of traditional supercomputing costs.

Originally: MIT Technology Review, Video, 2011. Read the original.

The Chef, the Puppet, and the Sexy IT Admin

Cade Metz — Wed, 26 Oct 2011 00:00:00 GMT

Cade Metz wrote about the rivalry between Chef and Puppet for Wired Enterprise in 2011. Mainstream tech press covering configuration management was new. For years the only people who cared about infrastructure automation were the ones carrying pagers.

Adam Jacob, Barry Steinglass, Nathan Haneysmith, and I cofounded Chef to bring the automation Google and Amazon kept as closely guarded secrets to everyone else. Chef let engineering teams define infrastructure as code, writing recipes to configure entire fleets of servers instead of managing them by hand. Puppet was working the same problem with a different philosophy, and the comparison became a running storyline as the category grew.

This piece marks the moment infrastructure as code stopped being an insider practice and became an industry. When Wired compares your configuration tool to your rival's, the category has arrived.

Originally: Wired, Article, 2011. Read the original.

DevOps Cafe Episode 19: Jesse Robbins

Damon Edwards, John Willis — Tue, 20 Sep 2011 00:00:00 GMT

Damon Edwards and John Willis hosted me on DevOps Cafe for a long conversation: teenage ISP work, the Seattle Fire Department, joining Amazon, building GameDay, cofounding Velocity, and starting Chef. A full transcript ran on the Chef blog at the time.

Originally: DevOps Cafe Podcast, Podcast, 2011. Read the original.

Puppet, Chef Ease Transition to Cloud Computing

Olga Kharif and Ashlee Vance — Thu, 01 Sep 2011 00:00:00 GMT

The custom tools built by Google, Amazon, and some other guys were such closely guarded secrets. Our founding thesis was to open up these tools to everyone else.
Jesse Robbins

A note from Jesse

Olga Kharif and Ashlee Vance wrote this for BusinessWeek on September 1, 2011. The original BusinessWeek URL is gone. The full piece is preserved here from archive.org because it captures the moment infrastructure automation crossed into the enterprise business press, with Opscode's founding thesis on the record.

Puppet, Chef Ease Transition to Cloud Computing By Olga Kharif and Ashlee Vance — BusinessWeek, September 1, 2011

Organizations as diverse as Northrop Grumman, Harvard University, Zynga, and the New York Stock Exchange have filled job websites with requests for talented puppeteers and master chefs. A quick dig into the job listings reveals that these positions have nothing to do with office entertainment or gourmet meals. Instead, the companies want people who have mastered Puppet or Chef, competing software tools that sit at the heart of the cloud computing revolution.

In essence, Puppet and Chef are levers used to control data center computers in a more automated fashion. The software has helped companies tap vast stores of computing power in new ways, accelerating research in fields such as financial modeling and genetics.

"This really changes the way science gets done," says Jason Stowe, the chief executive officer of Cycle Computing, a startup that uses Chef to configure thousands of computers at a time so that clients can perform calculations at supercomputer speeds. Before adopting Chef, doing such configurations took hours or even days. "We're down to single-digit minutes now," Stowe says.

The need for such tools originated with Google, Amazon.com, and their peers, who have long had to deal with the burden of managing tens or even hundreds of thousands of servers to support vast Web operations. Over the years these companies developed custom tools that can quickly turn, say, a thousand new servers into machines capable of displaying Web pages or handling sales. These programs allow the companies to run enormous, $500 million computing centers with about three dozen people at each one.

As more and more businesses move their software applications to the cloud, a handful of startups have developed mainstream versions on such data-center software. Puppet and Chef are the two with the highest profile.

"The custom tools built by Google, Amazon, and some other guys were such closely guarded secrets," says Jesse Robbins, co-founder of Opscode, the 20-person, Seattle-area startup behind Chef. The company has raised $13.5 million in venture capital. "Our founding thesis was to open up these tools to everyone else."

Opscode's Chef and its competitor, built by Puppet Labs, are both open source: Anyone is free to use and adapt the software. The companies make money by selling polished versions of the core technology and additional features, and by charging for advice on how to implement and best use it.

Luke Kanies came up with the idea for Puppet in 2003 after getting fed up with existing server-management software in his career as a systems administrator. In 2005 he quit his job at BladeLogic and spent the next 10 months writing code to automate the dozens of steps required to set up a server with the right software, storage space, and network configurations. He formed Puppet Labs to begin consulting for some of the thousands of companies using the software, the list includes Google, Zynga, and Twitter, and earlier this year he released the first commercial version.

Stanford University used to rely on a hodgepodge of tools to manage its hundreds of servers. They've since replaced that unorganized toolbox with Puppet. "We were a ragtag team, and now we are a cohesive unit, and our servers require a lot less attention," says Digant Kasundra, an infrastructure systems software developer at the university.

Palo Alto-based Jive Software has used Puppet to double the number of servers a single engineer can handle. "It's a huge impact for us," says Matt Tucker, chief technology officer and co-founder of the company.

Rivalry between Chef and Puppet is fierce. Puppet Labs argues that its software requires less training and collects more data about what's happening on the network. Chef claims a bigger developer base.

Michael Dell, the founder and CEO of Dell, follows Kanies on Twitter. Traditional data-center heavyweights such as Hewlett-Packard and IBM have shown interest in this type of software and could emerge as potential acquirers.

The bottom line: Investors have bet $20.5 million that Puppet and Chef, competing server-management tools, will be at the forefront of cloud computing.

Originally: BusinessWeek, Article, 2011.

DevOps Culture Hacks: Infecting your Boss & your Business with Awesome

Jesse Robbins — Tue, 08 Mar 2011 00:00:00 GMT

Don't fight stupid, make more awesome.
Jesse Robbins

This is the first time I presented the full culture hacks framework to a room. DevOpsDays Boston, March 2011. No slides. No video. Just me talking through the pattern I had figured out the hard way at Amazon.

We did not call it DevOps at the time. We did not have a word for it. I had been the stereotypical evil, nasty, mean ops guy. I took every outage personally. I had a record of 167 pages for heightened severity incidents in a single 24-hour period. I had multiple stretches of working 72 hours or more recovering from outages. People were afraid of me. I was proud of that, which tells you something about where my head was.

The story that made the room understand the problem was about Neil Roseman. I called Neil the VP of Awesome. Every cool project at Amazon seemed to be under Neil. Kindle. Search Inside the Book. A bunch of others. We had a battle over a deploy that I knew would take the site down. I did what every good ops person would do. I said absolutely not. Neil overrode me. He said, "The website may go down, but the stock price will go up." The site went live, and a few seconds later it crashed. Two days of chaos. And the stock and order rates went up.

At year end, I was penalized for the outage and for getting in the way of development. The dev teams were rewarded for shipping. That is the fundamental disconnect. Penalized for something out of my control. Rewarded for deploying and creating value. That misalignment of incentives is the root cause of most of the stupid things organizations do.

I became so famous for saying "No" that I would sign the Amazon launch posters with a big "No" and a scribble. It was the only value I could create. That was not progress.

So I figured out the formula. Five steps, each building on the last.

Start small. Find the smallest group of people who are already excited. Call it an experiment. Do not trigger the organizational immune system. I learned this in the fire service. I always tell people I will take 100 percent of the blame for whatever goes wrong as long as we make space to try.

Create champions. Get your boss on board first. Give everyone else the credit. You can accomplish anything you want so long as you do not require credit or compensation. At Amazon, I created the Call Leader Program to train senior people to run high-severity incidents. It became a high-status thing. Managers wanted in.

Use metrics to build confidence. Find a number that supports your change and use it ruthlessly. Tell the story with data. Have your champions evangelize on your behalf.

Celebrate successes. Create moments in time where people recognize that a change has occurred and that change is good.

Exploit compelling events. Big outages create cultural permission to make sweeping changes. And sometimes you create the compelling event yourself. That is what GameDay was. I got executive sponsorship, created a program where we broke critical parts of the infrastructure, and suddenly everyone needed something from me. Compelling event, manufactured.

I refined this talk at Velocity in 2012, but this was the first time I said it all out loud to a room of practitioners. The pattern has not changed. Don't fight stupid. Make more awesome.

MIT Technology Review TR35: Innovators Under 35

MIT Technology Review — Sat, 01 Jan 2011 00:00:00 GMT

MIT Technology Review named me to TR35 in 2011, citing the GameDay program at Amazon and the founding work at Opscode. The shorter, video version of the same recognition lives at Meet 2011 TR35 Winner Jesse Robbins.

Originally: MIT Technology Review, Article, 2011. Read the original.

Web Operations: Keeping the Data on Time

John Allspaw, Jesse Robbins — Mon, 28 Jun 2010 00:00:00 GMT

The Web is changing the way we live and touches every person alive. As more and more people depend on the Web, they depend on us. Web Operations is work that matters.
Jesse Robbins, from the foreword

John Allspaw and I co-edited Web Operations: Keeping the Data on Time and O'Reilly published it in June 2010. It is a book of essays from people who were actually doing this work at the time, written for the people who would do it next.

The book grew out of the Velocity Conference community I cofounded at O'Reilly in 2008. Velocity gave the people running the largest sites on the web a place to compare notes in public for the first time. Web Operations is what those notes looked like in book form.

The contributors are the people who built the discipline. John Allspaw on capacity planning and the relationship between development and operations. Theo Schlossnagle on dealing with unexpected traffic. Baron Schwartz on databases under load. Eric Ries on continuous deployment. Andrew Clay Shafer and Patrick Debois on the cultural shift that would soon get the name DevOps. Adam Jacob on infrastructure as code. Alistair Croll, Heather Champ, Paul Hammond, Richard Cook, Mike Christian, Eric Florenzano, Justin Huff, Jake Loomis, Matt Massie, Brian Moon, Anoop Nagwani, and Sean Power on the rest of what it took to keep a real site running. Every contributor was practicing what they wrote about.

The thesis was that operating large systems is its own engineering discipline, not a chore tacked onto development. That position was contested at the time. It is now the consensus, and the lineage from this book runs through DevOps, the SRE books from Google, and the platform engineering work the CNCF formalized a decade later.

I am proud of how it came together and prouder of who I got to do it with.

My foreword to the book

It's been over a decade since the first websites reached real scale. We were there then, in those early days, watching our sites growing faster than anyone had seen before or knew how to manage. It was up to us to figure out how to keep everything running, to make things happen, to get things done.

While everyone else was at the launch party, we were deep in the bowels of the datacenter racking and stacking the last servers. Then we sat at our desks late into the night, our faces lit with the glow of logfiles and graphs streaming by.

Our experiences were universal. Our software crashed or couldn't scale. The databases crashed and data was corrupted, while every server, disk, and switch failed in ways the manufacturer absolutely, positively said it wouldn't. Hackers attacked, first for fun and then for profit. And just when we got things working again, a new feature would be pushed out, traffic would spike, and everything would break all over again.

In the early days, we used what we could find because we had no budget. Then we grew from mismatched, scavenged machines hidden in closets to megawatt-scale datacenters spanning the globe filled with the cheapest machines we could find.

As we got to scale, we had to deal with the real world and its many dangers. Our datacenters caught fire, flooded, or were ripped apart by hurricanes. Our power failed. Generators didn't kick in, or started and then ran out of fuel, or were taken down when someone hit the Emergency Power Off. Cooling failed. Sprinklers leaked. Fiber was cut by backhoes and squirrels and strange creatures crawling along the seafloor. Man, machine, and Mother Nature challenged us in every way imaginable and then surprised us in ways we never expected.

We worked from the instant our pagers woke us up or when a friend innocently inquired, "is the site down?" or when the CEO called scared and furious. We were always the first ones to know it was down and the last to leave when it was back up again.

Always.

Every day we got a little smarter, a little wiser, and learned a few more tricks. The scripts we wrote a decade ago have matured into tools and languages of their own, and whole industries have emerged around what we do. The knowledge, experiences, tools, and processes are growing into an art we call Web Operations.

We say that Web Operations is an art, not a science, for a reason. There are no standards, certifications, or formal schooling (at least not yet). What we do takes a long time to learn and longer to master, and everyone at every skill level must find his or her own style. There's no "right way," only what works (for now) and a commitment to doing it even better next time.

The web is changing the way we live and touches every person alive. As more and more people depend on the web, they depend on us.

Web Operations is work that matters.

— Jesse Robbins

The chapters

From John Allspaw's preface, "How This Book Is Organized":

Chapter 1, Web Operations: The Career by Theo Schlossnagle. What this field actually encompasses, and why the skills needed are gained by experience more than by formal education.
Chapter 2, How Picnik Uses Cloud Computing: Lessons Learned by Justin Huff. How Picnik.com deployed and sustained its infrastructure on a mix of on-premise hardware and cloud services.
Chapter 3, Infrastructure and Application Metrics by Matt Massie and John Allspaw. The importance of gathering metrics from both your application and your infrastructure, and considerations on how to gather them.
Chapter 4, Continuous Deployment by Eric Ries. The advantages of deploying code to production in small batches, frequently.
Chapter 5, Infrastructure as Code by Adam Jacob. An overview of the theory and approaches for configuration and deployment management.
Chapter 6, Monitoring by Patrick Debois. The various considerations when designing a monitoring system.
Chapter 7, How Complex Systems Fail by Dr. Richard Cook. His whitepaper on systems failure and the nature of complexity often found in web architectures, with web operations-specific notes added to the original.
Chapter 8, Community Management and Web Operations. John Allspaw's interview with Heather Champ on how outages and degradations should be handled on the human side of things.
Chapter 9, Dealing with Unexpected Traffic Spikes by Brian Moon. Experiences with huge traffic deluges at Dealnews.com and what they did to mitigate disaster.
Chapter 10, Dev and Ops Collaboration and Cooperation by Paul Hammond. Places where development and operations can come together to enable the business, both technically and culturally.
Chapter 11, How Your Visitors Feel: User-Facing Metrics by Alistair Croll and Sean Power. Metrics that can be used to illustrate what the real experience of your site is.
Chapter 12, Relational Database Strategy and Tactics for the Web by Baron Schwartz. Common approaches to database architectures and some pitfalls that come with increasing scale.
Chapter 13, How to Make Failure Beautiful: The Art and Science of Postmortems by Jake Loomis. What makes or breaks a good postmortem and root cause analysis process.
Chapter 14, Storage by Anoop Nagwani. The gamut of approaches and considerations when designing and maintaining storage for a growing web application.
Chapter 15, Nonrelational Databases by Eric Florenzano. Considerations and advantages of using a growing number of "nonrelational" database technologies.
Chapter 16, Agile Infrastructure by Andrew Clay Shafer. The human and process sides of operations, and how agile philosophy and methods map (or not) to the operational space.
Chapter 17, Things That Go Bump in the Night (and How to Sleep Through Them) by Mike Christian. The various levels of availability and Business Continuity Planning (BCP) approaches and dangers.

Originally: O'Reilly Media, Other, 2010.

Ex-Amazon 'Master of Disaster' Animates Server Chef

Cade Metz — Tue, 22 Jun 2010 00:00:00 GMT

The Register profiled my move from Amazon to Opscode under the headline "Ex-Amazon 'Master of Disaster' Animates Server Chef." The article introduced the "Master of Disaster" framing to a global technology audience and connected it to Chef, the open-source infrastructure automation framework we had just launched.

At Amazon, my title was Master of Disaster. The Register put the job plainly: I was "responsible for website availability for every property bearing the Amazon brand." The practice we built for that work was GameDay: schedule failure on purpose, run the drill, expose the latent defects, do it again. That same orientation shaped Chef. Instead of configuring servers one at a time, engineers wrote "recipes" that programmatically configured and managed fleets of servers across cloud providers like Amazon EC2 and in private data centers. The Register called it "object-oriented programming for system administrators."

Adam Jacob and I co-founded Opscode with Nathan Haneysmith and Barry Steinglass, betting that infrastructure could be versioned, tested, and deployed like application code. Chef grew to serve Apple, Facebook, Google, and IBM, among many others, before Progress Software acquired the company in 2020.

Originally: The Register, Article, 2010. Read the original.

The Origins of Amazon's Cloud Computing

Stacey Higginbotham — Fri, 18 Jun 2010 00:00:00 GMT

I was horrified at the thought of the dirty, public Internet touching MY beautiful operations.
Jesse Robbins

Stacey Higginbotham at GigaOM reconstructed the actual origin of Amazon's cloud computing platform. Chris Pinkham, an Amazon engineer, wanted to return home to South Africa, and Amazon agreed to let him keep working from Cape Town. Pinkham and Christopher Brown built the small remote team that designed and shipped the first virtualized server platform inside Amazon, the system that became EC2.

I was running availability at Amazon at the time, and I told Stacey I had initially resisted the project. "I was horrified at the thought of the dirty, public Internet touching MY beautiful operations." Pinkham and Brown built their platform in a separate data center, outside my operational perimeter. Werner Vogels was the executive sponsor who created the space for them to do it.

The piece ends with a line from Carl Brooks that the operations posture at Amazon helped create the conditions for the platform that would invert it, an early instance of the pattern I had named three years earlier on O'Reilly Radar: you become what you disrupt.

Originally: GigaOM, Article, 2010. Read the original.

Velocity: The Art of Web Operations

Tim O'Reilly — Mon, 22 Jun 2009 00:00:00 GMT

A note from Jesse

This is what Tim wrote the week Velocity 2009 opened. The original is gone from O'Reilly Radar, so I am preserving it here. He tells the story of Steve Souders, Andy Oram, and me walking into his office and saying "we need a separate conference for our community." Tim's 2013 retrospective covers similar ground with more hindsight. This one captures the energy of the moment. Over 700 people showed up that year. The community had found its gathering place.

Two years ago, at the 2007 O'Reilly Open Source Convention, a group of web operations professionals, led by Jesse Robbins and Steve Souders along with O'Reilly editor Andy Oram, asked for a meeting with me. Their message: "We need a separate conference for our community." That community: the web operations professionals who keep sites up and running.

They knew I was receptive. A year earlier, I'd published a blog post entitled Operations: The New Secret Sauce. I had been pushing for years to get books on web operations into our publishing list (and in fact, Steve's book, High Performance Websites was in production at that time, and Andy had a number of other titles in the works.)

But nonetheless, the meeting felt like an intervention. It was absurdly exciting. I had been thinking in the abstract about the fact that as we move to a software as a service world, one of the big changes was that applications had people "inside" of them, managing them, tuning them, and helping them respond to constantly changing conditions. The skills and tools used by these people would need to be spread to a wider audience. But here were a group of these people, a big group, saying "We need an identity as a profession, and we need a gathering place for our tribe. We want your help."

How could I say no? We agreed to start with a "Summit" meeting to bring together the community and brainstorm ideas. Gina Blaber, our VP of Conferences, organized a meeting of 30 or 40 of the "big dogs", and the excitement was palpable. She moved quickly on from there to launch the Velocity conference. It was a success in its first year out, and the second annual conference, starting today in San Jose, promises to be even better.

What's more, the fact that attendance has surpassed last year, in an economy that has depressed attendance at many industry conferences by 30-50%, says something about the growing importance of this new field.

Back when I first began thinking about this topic for our publishing program, five or six years ago, observing that we needed books on what went on inside of Google and companies like it, the tools and processes they use to deliver such astounding performance and scale, the pushback was that "there are only a few companies operating at that scale."

But of course, if you've heard me speak, you've probably heard me quote William Gibson: "The future is already here. It's just not evenly distributed yet." Now, there are hundreds of companies (at least) operating at the scale that Google was operating at when I first made those statements. Over 700 of the people who keep them running are converging on San Jose today.

Originally: O'Reilly Radar, Article, 2009.

Jesse Robbins

What the Apple Neural Engine and Google's TPU tell us about the next decade of inference

How to run vector search on a website in the browser with no server (like I do)

Good Fences make Good Agents: sandvault + /sv skill (part 1 of n)

Sunil Dhaliwal, Founder of Amplify Partners, on Lessons from 27 Years in VC

The Seed 100: The Best Early-Stage Investors of 2026

Call a savepoint

Musical AI Raises $4.5M: Jesse Robbins Joins Board

What I look for when I invest

Leverage and Focus

The Big Lie About Investment Stages

Market Size Is the First Filter

Co-Founder Dynamics

Founder Grit

How to Get on VCs Radar

The Investment Process

This is an Adventure!

Musical AI Bags $4.5M to Scale AI Attribution Tech: Jesse Robbins Joins Board

Investing in Vibrant Labs: AI Agent Simulation Infrastructure

What Investors Look For in AI Startups: Builders with Taste

Test and Taste

Still Automating Toil

The Next Abstraction Layer

Agents Are Just Another Developer

The Future of Dev Tools Is Autonomous: Engineers Will Become Fleet Generals

Designing for agents

Open source reignited my joy

Collaboration over fighting

Experimentation, Causal Inference, and AI: Sean Taylor of OpenAI and VC Jesse Robbins at Data Council 2025

Next in Tech Ep. 197: Data Pipelines for AI

The DevOps moment for data

Start small, iterate locally

Pipeline quality as competitive advantage

The Data Pipeline is the New Secret Sauce

AI Investor Jesse Robbins on NYSE Floor Talk

Full Interview Transcript

Jesse Robbins Named One of the 30 Most Successful Early-Stage Startup Investors

Heavybit Welcomes New Member: Continue

Cloud Native StartupFest 2023

Generative AI in DevOps and Incident Response: What the Experts Actually Think

Key Themes

Where AI actually helps: summarization

Why Hallucination Disqualifies Autonomous Remediation

The Junior Developer Analogy

The AI SRE Career Question

The Human Judgment Floor

Further Reading

DevOps is dead? Nope, it is maturing ft. Jesse Robbins

What to Know About the Modern Incident Response Lifecycle

Fireside Chat with Jesse Robbins and Kolton Andrus • Failover Conf 2021

The Seed 100: The Best Early-Stage Investors of 2021

An oral history of #hugops: How tech's first responders built a culture of empathy

Life of a sysadmin

When emergencies are a day job

A meeting of like minds

Time for a hug

The future according to #hugops

A Developer's View Into Blockchain Network Architecture

Incident Management for Operations (foreword by Jesse Robbins)

My foreword to the book

'Star Trek Communicator Startup' Sets Out to Build a World Powered by Voice

This Startup Thinks Your Workplace Needs Wearable Walkie-Talkies

Building Companies that Devs & DevOps Teams Love, And Avoiding Expensive Mistakes

Tim O'Reilly on Why We Started the Velocity Conference

Jesse Robbins on the Rise of DevOps (InfoQ Interview)

DevOps is a reorientation

Infrastructure as code

Velocity and the practitioners

Q&A: Ex-Amazon 'Master of Disaster' Jesse Robbins on the Power of 'Relentless Optimism' in Startups

Resilience Engineering: Learning to Embrace Failure

Jesse Robbins on DevOps as Business Alignment

DevOps as business alignment

Operations as enablement

Chef and infrastructure as code

EC2 and the cloud operating model

Simple services, fast feedback

Changing Culture & Being a Force for Awesome

The framework

The Katrina lesson on permission

The rule