Emergent Architectures and Beating the Monolith

Richard Bown: 00:00:00

Hello and welcome to Loving Legacy.

Richard Bown: 00:00:01

This is episode 28.

Richard Bown: 00:00:03

I'm Richard Bown your host, and this time I'm talking about emergent

Richard Bown: 00:00:07

architectures and beating the monolith.

Richard Bown: 00:00:10

It's been a while since I recorded myself just talking about

Richard Bown: 00:00:12

things and to tell the truth.

Richard Bown: 00:00:13

I've been thinking long and hard about this next step.

Richard Bown: 00:00:17

I've been working on a course which will teach technical leaders why all of their

Richard Bown: 00:00:20

effort to gain clarity and control of software development and delivery is

Richard Bown: 00:00:25

essentially wasted unless some effort is spent to acknowledge how software

Richard Bown: 00:00:29

gets built outside of all the frameworks and techniques that we are bombarded.

Richard Bown: 00:00:34

This means consuming a lot of those frameworks, ideas and techniques

Richard Bown: 00:00:37

myself, and then putting some words together to form a common understanding

Richard Bown: 00:00:42

of what we're trying to achieve.

Richard Bown: 00:00:44

Ironically, for some, you might recognize this as a fundamental

Richard Bown: 00:00:47

principle of domain-driven design, defining a domain language, and then

Richard Bown: 00:00:50

modeling a solution using that language.

Richard Bown: 00:00:53

Okay, so let's dive in.

Richard Bown: 00:00:56

In Sam Newman's building Microservices second edit.

Richard Bown: 00:00:59

It's all about building distributed systems.

Richard Bown: 00:01:02

Pretty much any system is a distributed system right now goes the argument.

Richard Bown: 00:01:06

So should we all be setting out explicitly to build

Richard Bown: 00:01:08

microservices based architectures?

Richard Bown: 00:01:12

Many seem to think so and have invested heavily both in the technology

Richard Bown: 00:01:15

platforms to support this as well as the skills and learnings that

Richard Bown: 00:01:18

allow successful implementation.

Richard Bown: 00:01:20

However, is this a one size fits all?

Richard Bown: 00:01:25

emergent architecture is a smart way to say we built something and once we had

Richard Bown: 00:01:29

a plan, but now we have to change it.

Richard Bown: 00:01:32

This is therefore the definition of pretty much any software

Richard Bown: 00:01:35

ever a monolith can appear.

Richard Bown: 00:01:37

It's just a block of code, and we will find that difficult

Richard Bown: 00:01:40

to break down any further.

Richard Bown: 00:01:41

Usually due to design decisions made before or during its assembly,

Richard Bown: 00:01:46

a monolith could even appear nominally in a microservice, although

Richard Bown: 00:01:49

that would probably mean that it wasn't a microservice anymore.

Richard Bown: 00:01:53

Documentation is of use in order for software archeologists to discover

Richard Bown: 00:01:57

the business motivations for why we're built that way in the first place.

Richard Bown: 00:02:01

They're a great starting point to understand how your software

Richard Bown: 00:02:03

developed to the point it is now.

Richard Bown: 00:02:07

The history of software development is littered with the bones of great

Richard Bown: 00:02:10

technological frameworks and ideas that have briefly set the world a

Richard Bown: 00:02:13

light only to disappear Agonizingly slowly , as the zealots of the

Richard Bown: 00:02:18

ideology continue to clinging on to the investment they have made.

Richard Bown: 00:02:21

This is the very definition of the sunk cost fallacy.

Richard Bown: 00:02:25

Those that have invested so much financially, emotionally, and

Richard Bown: 00:02:28

reputationally, that they must double down on that approach until

Richard Bown: 00:02:31

it's either universally accepted or until it's dead in the water.

Richard Bown: 00:02:37

This approach, I call the "Abyss of Obsession", and it's a social construct.

Richard Bown: 00:02:42

having been there myself with ideas, designs, and architectures, I feel I

Richard Bown: 00:02:46

must talk them through with peers until we reach either an understanding that

Richard Bown: 00:02:50

could work or a feeling that it couldn't.

Richard Bown: 00:02:54

Any ideas that persist after this peer group round housing will eventually

Richard Bown: 00:02:58

start to take seed in our minds.

Richard Bown: 00:03:00

This will be our famous thing we think, and we get ahead of ourselves

Richard Bown: 00:03:03

in believing that we've solved the world's software problems.

Richard Bown: 00:03:07

If you're not familiar without feeling yet, then I think you probably need

Richard Bown: 00:03:10

to find a better class of software developers to hang around with and

Richard Bown: 00:03:14

just dream a little bit bigger.

Richard Bown: 00:03:16

Usually this kind of discussion happens in the pub after a few ales.

Richard Bown: 00:03:19

But it's important to dream big.

Richard Bown: 00:03:22

We're all capable of it, and software is the perfect scaffolding for these dreams.

Richard Bown: 00:03:28

Okay, so saying we implement something using this technique and it works.

Richard Bown: 00:03:31

Then we start to enter this abyss of obsession and we are most

Richard Bown: 00:03:35

likely not to return for a while.

Richard Bown: 00:03:37

We are a submarine disappearing to the Mariana's trench of ambition.

Richard Bown: 00:03:41

This applies as much to architecture frameworks as it does process frameworks

Richard Bown: 00:03:45

like Agile, as it does technologies.

Richard Bown: 00:03:48

We have a hammer and everything looks like a.

Richard Bown: 00:03:52

. Okay, so taking obsession around frameworks, technologies, and processes.

Richard Bown: 00:03:57

As read, we should be careful when accepting new ideas and new words

Richard Bown: 00:04:01

and concepts into our vocabulary.

Richard Bown: 00:04:04

The monolith doesn't actually exist as in as much a microservice

Richard Bown: 00:04:08

doesn't actually exist.

Richard Bown: 00:04:09

These are abstract ideas, or we can build our imperfect models

Richard Bown: 00:04:13

of the world and attempt to implement them and learn as we go.

Richard Bown: 00:04:16

What we end up with in software design, development and

Richard Bown: 00:04:19

delivery is a complex system

Richard Bown: 00:04:20

According to Dave Snowden's Canne Framework, a complex

Richard Bown: 00:04:23

system is the unknown.

Richard Bown: 00:04:25

Unknowns cause and effect can only be deduced.

Richard Bown: 00:04:28

In retrospect, there are no right answers.

Richard Bown: 00:04:31

Instructive patterns can emerge if the leader conducts

Richard Bown: 00:04:34

experiments that are safe to fail.

Richard Bown: 00:04:36

Can Evan cause this process probe sense?

Richard Bown: 00:04:40

If you've ever tried to reverse engineer a system, then you will know that a complex

Richard Bown: 00:04:44

system is exactly that, often intractable.

Richard Bown: 00:04:47

They are the result of forgotten rules.

Richard Bown: 00:04:49

One can design a system as well as you like, but if you've forgotten

Richard Bown: 00:04:51

a crucial rule until later in the process, then often the amount of

Richard Bown: 00:04:55

work to completely redesign the system to accommodate the fundamental

Richard Bown: 00:04:58

thing you've forgotten is too cost.

Richard Bown: 00:05:03

I've tried to graph this effect and I will link an article which shares these graphs.

Richard Bown: 00:05:07

We map cost of implementation versus cost of design, and we call design

Richard Bown: 00:05:12

anything in this case that isn't coding, testing, deploying, or supporting the

Richard Bown: 00:05:15

code directly via incidents, et cetera.

Richard Bown: 00:05:20

The cost of design therefore could encompass anything that is seen as

Richard Bown: 00:05:23

thinking about the system, refinement sessions about architecture.

Richard Bown: 00:05:28

Workshops, presenting facts or findings, learning new techniques.

Richard Bown: 00:05:31

Anything which falls into the category of talking about rather than doing

Richard Bown: 00:05:39

what you see from typical software projects is that either

Richard Bown: 00:05:42

zero sum or significant upfront planning or design is made.

Richard Bown: 00:05:47

If no upfront planning is made, then the project starts amidst a

Richard Bown: 00:05:50

flurry of activity, coding, design, experiments, trial and error, depending

Richard Bown: 00:05:55

on the ambition of the project.

Richard Bown: 00:05:57

This might be the right way to go.

Richard Bown: 00:05:59

No, upfront cost means jumping in, iterating quickly, and

Richard Bown: 00:06:02

learning as it moves forward.

Richard Bown: 00:06:05

Some design means that decisions perhaps around technology platforms

Richard Bown: 00:06:09

architecture have been made, and we perhaps know an outline of our

Richard Bown: 00:06:13

system design before we start coding.

Richard Bown: 00:06:16

Significant design means mapping out exactly how our system is going to look.

Richard Bown: 00:06:20

and proceeding on that basis.

Richard Bown: 00:06:22

This means that building is optimized, but we might end up building the

Richard Bown: 00:06:26

wrong thing for any approach.

Richard Bown: 00:06:29

The only time we can validate our assumptions is when we get

Richard Bown: 00:06:32

our system in front of the people it's supposed to be serving.

Richard Bown: 00:06:34

When we receive that initial feedback, we can iterate our design and depend on

Richard Bown: 00:06:39

how flexible our initial architecture is.

Richard Bown: 00:06:41

This will come with a variable cost.

Richard Bown: 00:06:45

I call this cost to first "gotcha".

Richard Bown: 00:06:47

The moment where we think, oh no, we forgot this fundamental thing,

Richard Bown: 00:06:50

and often software systems lurch from gotcha to gotcha until perhaps

Richard Bown: 00:06:54

they either fulfill their initial brief or requirements and become

Richard Bown: 00:06:57

useful to the customer or they fail.

Richard Bown: 00:07:00

But no system is ever completely free of gotcha's because users can always

Richard Bown: 00:07:04

think of things that we can't as system designers or builders and software being.

Richard Bown: 00:07:09

Being flexible and software engineers also being prone to

Richard Bown: 00:07:12

fall into the abyss of obsession.

Richard Bown: 00:07:14

No change is too great for our system.

Richard Bown: 00:07:17

No matter what approach we take initially, we will always be faced with a dilemma.

Richard Bown: 00:07:20

Should we extend this legacy system to include new functionality

Richard Bown: 00:07:24

or should we just write a new.

Richard Bown: 00:07:28

Therefore legacy is a social trap even more than it is a technological trap.

Richard Bown: 00:07:33

This is the very basis of emergent architecture and why we'll never

Richard Bown: 00:07:37

actually beat the monolith.

Richard Bown: 00:07:39

It's not something to be beaten.

Richard Bown: 00:07:40

It's something to be used when appropriate.

Richard Bown: 00:07:42

It's something to be used as any other tool in your toolkit.

Richard Bown: 00:07:46

often as programmers, we don't set out with anything in mind

Richard Bown: 00:07:50

other than a purpose for our work.

Richard Bown: 00:07:52

The rest of the discussion, the design architecture technologies and the

Richard Bown: 00:07:55

things that we use to build our ideas upon are often only abstract concepts

Richard Bown: 00:08:00

which we use to share our ideas.

Richard Bown: 00:08:02

Once they are made flesh in whatever tech we choose, they inevitably change.

Richard Bown: 00:08:07

Do not let our ideas become code.

Richard Bown: 00:08:10

This is why I have a somewhat ambivalent attitude to diagramming

Richard Bown: 00:08:13

tools and ways to turn ideas into code.

Richard Bown: 00:08:16

Things like the C4 framework are good at a high level, but these are

Richard Bown: 00:08:19

merely conceptual boxes of things.

Richard Bown: 00:08:21

Any detailed design driven from flittering ideas are usually

Richard Bown: 00:08:25

not proven to be helpful.

Richard Bown: 00:08:26

We will, at some point, have to compromise on these ideas, and usually at that

Richard Bown: 00:08:30

point we'll work too hard to modify our original design to accommodate change.

Richard Bown: 00:08:35

Then perhaps it's not even helpful.

Richard Bown: 00:08:37

The same happens when we consider any design paradigm, purely idiomatic.

Richard Bown: 00:08:42

If you read the Blue Book by Eric Evans on domain-Driven Design,

Richard Bown: 00:08:45

you'll find the premise of it is one.

Richard Bown: 00:08:48

For most software projects, the primary focus should be on the domain and

Richard Bown: 00:08:51

the domain logic, two complex domain designs should be based on a model.

Richard Bown: 00:08:57

For me, there is nothing to argue with here.

Richard Bown: 00:08:59

Understand what you're trying to achieve and then deliver a model of it.

Richard Bown: 00:09:02

Also talking the same language as the users, but it's surprising how often we

Richard Bown: 00:09:06

are drawn to focus on techniques or sub techniques rather than the bigger picture.

Richard Bown: 00:09:11

That's understandable because to form thoughts like this takes

Richard Bown: 00:09:14

a lot of hard work and time, and it's only conceptual work.

Richard Bown: 00:09:17

It's not a useful day to day for an engineer or engineering leader.

Richard Bown: 00:09:20

Even practical techniques are the things that we fall back on

Richard Bown: 00:09:23

and utilize, remembering to use.

Richard Bown: 00:09:26

And remembering why we're using them is good to always keep in mind.

Richard Bown: 00:09:32

Emergent architecture means that things change.

Richard Bown: 00:09:34

We can be resilient to change, not just with how we build our

Richard Bown: 00:09:37

applications physically, but also by expecting things to need to change.

Richard Bown: 00:09:41

To be aware that circumstances outside our control may intercede

Richard Bown: 00:09:45

to make us have to change.

Richard Bown: 00:09:48

One of the conclusions we must be able to draw now from the Elon

Richard Bown: 00:09:50

Musk evisceration of engineering at Twitter is that the system was

Richard Bown: 00:09:54

certainly well enough engineered to survive significant tampering.

Richard Bown: 00:09:57

It's still running.

Richard Bown: 00:09:57

Despite a few blips and outages, things are changing.

Richard Bown: 00:10:01

The architecture is adapting and growing.

Richard Bown: 00:10:03

The rest is supposition but all we can do is base an understanding on what

Richard Bown: 00:10:07

we feel and experience as customers.

Richard Bown: 00:10:11

Whatever we choose as a starting point.

Richard Bown: 00:10:13

Know some or lots of design, our decisions are always going to be

Richard Bown: 00:10:16

called into question, but new people challenging our ideas of technology,

Richard Bown: 00:10:20

fitness, for purpose and business direction are always going to appear.

Richard Bown: 00:10:23

Therefore, our starting point doesn't really matter, and by

Richard Bown: 00:10:26

extension, where we are now with our systems doesn't really matter.

Richard Bown: 00:10:30

This is why a legacy system is really every system we ever build.

Richard Bown: 00:10:34

If we truly value it and we want it to be better for our customers, then we must

Richard Bown: 00:10:38

treat every system in whatever state, age, or technology with the same reverence.

Richard Bown: 00:10:44

It's only dependent upon relevance in the market and relevance to the customers.

Richard Bown: 00:10:49

I propose a framework of the best practices that allow us to stay grounded

Richard Bown: 00:10:53

to our customers' needs as closely as possible, while reusing all the terms

Richard Bown: 00:10:56

that any good engineering organiz.

Richard Bown: 00:10:59

are already familiar with.

Richard Bown: 00:11:02

I believe that just like a good domain model, we already have

Richard Bown: 00:11:05

enough or perhaps too many words to describe the patterns that we see in

Richard Bown: 00:11:09

enterprise systems and architecture.

Richard Bown: 00:11:11

What is missing is a good Rosetta Stone that enables us to speak

Richard Bown: 00:11:14

the same language across ideas.

Richard Bown: 00:11:16

If we can reuse rather than reinvent terms and agree on a language, then perhaps we

Richard Bown: 00:11:21

edge closer to a common understanding of what it means to build great software.

Richard Bown: 00:11:25

This is where novels like the Phoenix Project or the

Richard Bown: 00:11:28

Unicom Project, are so strong.

Richard Bown: 00:11:30

They place these facts in a story as techniques to try and help us

Richard Bown: 00:11:33

understand where we want to go.

Richard Bown: 00:11:35

And this is something that we can all do.

Richard Bown: 00:11:36

Storytelling with practical examples from your business, working out how the

Richard Bown: 00:11:40

stories of your systems need to change in order for you to deliver the right

Richard Bown: 00:11:44

value to your customer at the right time.

Richard Bown: 00:11:47

Fundamentally, this is all we can hope to achieve from software engineering

Richard Bown: 00:11:51

. Engineering anything is about

Richard Bown: 00:11:53

Something that a customer will pay money for or utilize.

Richard Bown: 00:11:57

Therefore, we define software engineering as building the

Richard Bown: 00:11:59

right software for a customer.

Richard Bown: 00:12:02

Emergent architecture is used for understanding and extending complex

Richard Bown: 00:12:05

systems, which have arrived at their state through an unknown path.

Richard Bown: 00:12:10

Try to learn how you got where you are now through analysis,

Richard Bown: 00:12:13

reading, talking to engineers and customers, finding out the history.

Richard Bown: 00:12:17

Only then will you be able to plot a path to where you need to be next.

Richard Bown: 00:12:21

Your frameworks, technology choices, et cetera, either constrain

Richard Bown: 00:12:25

or enable its path and should also inform you of your future.

Richard Bown: 00:12:30

I hope I've given you something to think about in this episode.

Richard Bown: 00:12:32

I'll be covering more around this in the future, so, If you enjoy this, then

Richard Bown: 00:12:36

please like and subscribe to the podcast.

Richard Bown: 00:12:38

And, feel free to share it with your friends and colleagues, and also feel

Richard Bown: 00:12:41

free to reach out to me if there's anything you'd like to talk about

Richard Bown: 00:12:43

legacy systems engineering, or the future of emergent architecture.

Richard Bown: 00:12:48

Until next time, this is Richard Bown wishing you goodbye and good luck.