You should not be using AWS. Probably.
Why choosing a popular cloud service, like AWS or Google Cloud Platform, may be the wrong choice for you and what to do instead.
If you're an indie hacker, a boostrapper, a startup, an agency or a consultancy, or just a small team building a product, chances are you are not going to need the cost and complexity that comes with modern cloud platforms.
It costs how much?!
Unless you hire a good (read: expensive) devops person to look after your infrastructure, clickops-ing your way through the AWS console will likely leave behind a pile of unused instances and components that will eat into your budget.
On top of the financial cost, there is productivity and opportunity cost from having to understand, deal with and work around the idiosyncracies of complex cloud platforms. This effort is much better spent on delivering value to users.
Let's face it, choosing AWS is the cloud computing version of "nobody ever got fired for buying IBM". There is perceived safety in choosing a popular offering — it's what everybody else does. Many big players that product teams look up to, be it Netflix or Twitter or some other household name, have built all or some of their infrastructure on top of public cloud services. This creates scale-envy and blinds teams to the actual requirements that exist here and now.
The world is not driven by greed, it's driven by envy.
— Charlie Munger, vice chairman of Berkshire Hathaway
Just because Netflix, which accounts for about 15% of the global internet traffic, has built a complex architecture of custom software and microservices on top of a popular public cloud provider does not mean you absolutely, positively have to replicate their setup.
The all-you-can-eat buffet problem
Finally, there is the ecosystem. Having a great ecosystem is a huge benefit. It's like an all-you-can-eat buffet. And what happens in an all-you-can-eat buffet? You tend to overeat.
Public cloud platforms such as AWS have hundreds of services to choose from. It can be easy to convince yourself that you need to go serverless or run on ECS or set up your own Kubernetes cluster instead of using something like
Heroku or Render, when the former are at "arm's reach". And often it is so alluring that actual requirements and costs fly out the window. Need to scale... infintely? Just use Lambda Functions. Easy. A globally distributed datastore. OMG yes, we need it.
The problem is that every choice you make is not just the choice on its own, there are always second and third order implications down the line.
In a vacuum, each exotic technology only has benefits. It is like the old quote about LISP programmers: "(LISP) programmers know the value of everything and the cost of nothing". A specific technology product never exists in a vacuum — it has to communicate and co-exist with other components in the system. There are costs associated with every choice, often hidden costs.
A wealth of information creates a poverty of attention.
— Herbert Simon, political scientist
Herbert Simon was right. The abundance of options pulls focus away from what really matters — your users and customers.
Culture of simplicity eats strategy of complexity for breakfast
Using the shiniest new technologies is rarely the cause for success, it's usually the result. Resist the urge to buy an excavator when a shovel will do. Cultivate a culture of ruthlessly fighting complexity and focusing on what your users want.
1. Optimise for operational simplicity
We all want to learn and use new technologies, play around with new languages and frameworks, and add the latest tool to our resume. Cloud services give you plenty of rope to hang yourself with and Docker enables practically every team, regardless of size, to use their programming language or framework of choice.
This sort of complexity is extremely hard to operationalise. Google has roughly 28,000 engineers and only 7 official programming languages. Ask yourself, does your team of ten really need three different tech stacks?
It is much easier to go from simple to complex than the reverse. Instead of building directly on top of public cloud services like AWS or GCP, go with
Heroku, Render or Fly. Start with a simple data model and architecture, and any one of those three will take you very far. Choose simple, "boring" technologies.
2. Optimise for internal simplicity
Pretty much every modern software project that touches the web is a complex system. You cannot build a correctly functioning complex system from scratch. This is why big-bang releases and sweeping re-writes hardly ever go well.
Invariably, all complex systems that work evolved from simpler systems that worked.
— John Gall, complex systems theorist
Although the marketing copy on public cloud websites can make it seem like choosing
$TECHNOLOGY will enable your app to scale infinitely, the harsh truth is that neither Lambda Functions, nor Kubernetes, nor Kafka on their own will magically make your app work correctly, be performant and deliver value.
Most of this stuff is learned the hard way, earned as experience in the trenches. It comes down to good software engineering skills. It comes down to understanding databases, caching and queues. It means developing deep expertise in one or two programming languages to build the product with. It means thinking through what data access patterns will look like in production and how to design for read-heavy vs write-heavy services and endpoints.
None of this is easy. If it was, we wouldn't have any re-writes, over time and over budget projects, and products that collapse under the weight of their own technical debt.
3. Focus on delivering value to your users
A useful lens to look things through is "is what we're doing providing the most amount of value to our users, relative to what we could be doing?". It starts with technology, the data model and architecture choices, and ends with new features, products and services. I would argue that choosing to eliminate technical debt can be the highest value work if it improves developer workflow significantly and enables providing value to customers even more quickly/robustly/cheaply as a result. The best teams think like investors.
There are many examples of folks relentlessly focusing on delivering value. And there numerous examples of dead startups that burned through an astronomical amount of cash without ever making a dollar.
I am not saying you should be writing cowboy code and having no processes — quite the opposite. I am saying, though, that spending tens or hundreds of thousands on infrastructure does not automatically make you useful to customers. It just means you've over-engineered your product.
Calibrate your choice of programming languages, external software components, infrastructure and other technology against the litmus question:
"Why do we think this choice will provide the most value for users compared to the alternatives?"
And try to answer honestly.
: I cannot recommend Heroku after tweets like this https://twitter.com/dannypostmaa/status/1625098298465087488