Microservices are simple. Why are they so hard?
This is the first part and an introduction for what I plan to be a series of articles on modern distributed software development practices.
Microservices are the most intuitive thing in the world. It should be clear that you shouldn’t have large development teams for anyone who ever worked in one. It’s also obvious that you should decouple different parts of your application as much as you can. After all, there’s a reason this is a basic principle in software design. There are also good reasons to dislike monolithic databases. Since “every understanding is misunderstanding” and since each team will want to look at the data using slightly different meanings, we easily end up with an incoherent mess instead of the “perfect approximation”.
Microservices are a good idea. For the development team, the engineer, the manager, it all makes sense. So why is it notoriously hard to stabilise microservice-based projects? The development effort is there to serve the end-product, so more freedom and flexibility for each team should clearly yield a better, more stable product with much more manageable dev cycles, shouldn’t it? The promise of microservices is that we can finally start concentrating on developing working code instead of dealing with the technical “friction” between distant functions within the company. The promise is that we should only have to deal with some data issues when they arise, which (in the grand scheme of things) is a miniscule amount of effort when compared with the alternative.
My experience
I understand that I’m just one person and not a representative sample, but bear with me for a moment. Having worked in enterprise (mostly finance) IT for 20+ years, I’ve never seen anyone fired or demoted for incompetence. Let’s dwell on this fact for a minute because (to me) this is mind-boggling. How is it possible that in one of the most intellectually challenging, abstract profession with so many moving parts, I’ve never even heard of anyone who encountered problems in their work due to their lack of technical abilities? Is everyone who gets in to banking IT a genius? Is software development just a well-rehearsed set of drills that anyone can learn? Wouldn’t this mean that the “not enough developers” trope should have gone away at least a decade ago, another decade after IT development became so well paid?
There’s a clear gap between the fact that developers are so scarce and that competence isn’t the benchmark for success young IT folks like to think it is.
What could explain this? My theory is that this is exactly because the profession is so opaque. It’s next to impossible to know everything we’re doing even in the most vanilla enterprise environments. I routinely had no idea about the complexities my closest co-workers were facing in their work (yes, we were agile, and yes, we did share the code) and they didn’t know about mines. What I would have guessed for the others to take a few days could have easily been weeks and the other way round. We both inherit and introduce so many assumptions into our models that you have to be a specialist to be able to do any meaningful work, so we can’t know whether the other is being slow because they don’t know what they’re doing or because I don’t know what I’m thinking.
Foreword to the series
In other words, we don’t know what we can’t know. An obvious example most people have encountered is SQL. Learning the basics of it takes maybe an afternoon. It’s clear and intuitive that you can SELECT for data, which you can UPDATE or DELETE and that you can INSERT new information. On the other hand, I can count on one hand the number of developers I’ve worked with who were happy to think about isolation guarantees or their underlying implementation and how those impacted their software.
The dark side of designing our systems and tools to be intuitive is that we’re giving false confidence to its users. A lot of people will adapt the tools because of their ergonomics but few dive deep enough to grasp the finer details of what they’re actually doing, which decides things like stability, scalability and extensiveness, which comes back to bite them once their software does what it was intended to do, run in production.
This series will be about how we, as an industry do microservices now and how we can improve on that. It’s about how the technology could be tamed to serve the developers and how its promises could come true.
In the second article in the series, we make an attempt at implementing a very simple microservice-based project.
https://andrasgerlits.medium.com/microservices-and-the-myth-of-loose-coupling-9bbca007ac1a