What you don’t know about working with AWS

August 27, 2021, was my last day at Amazon Web Services (AWS). I spent two years there, most of it running the company’s open source marketing and strategy team. While ostensibly helping the world better understand the open source work AWS does, we actually spent most of our time inside AWS, helping product teams understand why and how to contribute to the relevant open source upstreams upon which their services might depend. The rest was spent outside the company, working with open source companies such as Confluent and Databricks to improve AWS partnerships with those companies.

Oh, and along the way, I helped put out dumpster fires that erupted when AWS was perceived to be doing “bad things” to open source companies and communities.

In my experience, much of the ire directed at AWS over open source is misplaced. No, it’s not that AWS is perfect, though the company remains one of the world’s biggest contributors to open source projects. (Whether you measure by number of active contributors or code, you can see the data for yourself by running this code.) Rather, it’s mostly an error in thinking of AWS as a monolithic entity with a common approach to open source. This is one of the primary myths about AWS that leads to misunderstanding, but there are more, as I’ll try to tackle here. Nothing I’ll write here is secret, though it’s almost as if Amazon hides it all in plain sight.

Two-pizza teams

 Amazon Founder and former CEO Jeff Bezos instituted the “two-pizza rule” early in the company’s history: “We try to create teams that are no larger than can be fed by two pizzas.” This is a bit of an exaggeration, but the principle is religiously adhered to across AWS. Teams tend to be relatively small and, just as important, they are almost wholly autonomous.

What does this mean? Well, it means that you might be right that service team X isn’t currently contributing back to an open source project, but that doesn’t mean the same is true of the other service teams (more than 200 of them). The ElastiCache team, for example, employs one of the five maintainers of Redis. Other teams make significant contributions to Rust, Apache Lucene, Kubernetes, OpenTelemetry, etcd, Apache Iceberg, OpenJDK, GraphQL, and more.

Are there service teams who don’t yet work with open source upstreams? Of course, just as there are at Microsoft, Google, Alibaba, etc. But while I worked for AWS, I saw this shifting. It’s a slow process precisely because Amazon almost never works by top-down fiat. If you want Amazon to contribute more, you need to focus on individual teams and, just as importantly, you need to speak Amazonian.

Yes, those LPs are real

By “speak Amazonian” I’m referring to the language and thought process embedded in the company’s 16 Leadership Principles (LPs). Before joining AWS I thought the LPs must be cheesy sloganeering at best, jingoism at worst, but it turns out they provide a common framework for more than one million Amazon employees to talk to each other and work together. They permeate virtually every discussion at AWS, including the famous “6-pagers” that are used to raise ideas and decide operational plans.

When I joined AWS, I initially avoided using the LPs in discussions. It didn’t go well. I’d say, “We should contribute to project X because it’s the right thing to do!” Blank stares. “We need to give engineers more time to contribute to project Y so we can influence the project’s direction.” Raised eyebrows. I was getting nowhere.

But then I tried framing my arguments with the LPs and things got much better. I started saying things like, “It’s hard to ‘Obsess over Customers’ and deliver innovation with ‘Frugality’ if we don’t ‘Earn Trust’ with the communities we depend on by making code contributions. This also allows us to ‘Insist on the Highest Standards’ because our contributions put us in a better position to ‘Deliver Results’ and support customers.”

Suddenly, people understood what I meant. It’s not that they were dense before; rather, I needed to speak the language that calls out the principles that govern everything that is done at AWS (and, indeed, all of Amazon). If you want to change behavior at AWS, you must frame the desired outcome using the LPs. My team became increasingly adept at doing this, and it’s paying off in ever-greater service team involvement in the projects upon which they depend. This is not to say there isn’t room for improvement.

The spirit is willing

In my time at AWS, I never heard a single person disparage the importance of open source. Quite the contrary. I know it’s fun to caricature AWS as a bunch of evil henchmen intent on strip-mining open source, but I never encountered anyone that fit that description. Rather, when I’d have disagreements with service team general managers or others about the relative importance of open source to their business, our disagreements invariably came down to which LPs we weighed heaviest in a particular situation.

For example, “Customer Obsession” comes first for all Amazonians, but it can be read in different ways. I might view open source contributions as critical to obsessing over customers in the medium to long-term, but a service team general manager also must consider the near term, which might mean creating a private branch of code to ensure a company could rapidly fix bugs or deliver features customers were demanding. It’s also the case that although customers all seem to like and want open source, many value the operationalization of that code even more (something that Tim Bray pointed out years ago).

This means that ensuring a seamless customer experience in the short term can sometimes consume the engineers who might otherwise be contributing to the longer-term success of a given project. I’ve seen this changing for the better at AWS but, again, there is no one-size-fits-all approach to open source in a company filled with two-pizza teams.

It also means that LPs such as “Ownership,” “Invent and Simplify,” “Insist on the Highest Standards,” and “Deliver Results” can seemingly conflict with the desire to partner well with commercial stewards of open source projects. If a customer wants Apache Kafka made easier for them, the immediate response is to build a service that you can manage on their behalf, with as few moving parts or opportunities for failure as possible. Another response is to partner to ensure that seamless customer experience. Though AWS has perhaps historically found the first option easier to deliver, I’m encouraged by all the success I saw with Confluent (in the Kafka case), as well as other open source companies.

In this area and others, AWS still has a ways to go—but then we all do. One of the things I’ve loved most for more than 20 years in open source is just how much more we, as an industry (and as individual people and companies), have to learn despite years of trial and error. No one has cracked the code on the perfect way to build and run an open source project or business. We’re all still learning.

So let’s be patient with each other, and seek to understand why a person or company operates as it does, as I’ve tried to do here for AWS.