Observability Engineering and Customer Needs with Stephen Townsend

Richard Bown: 00:00:00

Hi, this is Richard Bown.

Richard Bown: 00:00:02

Welcome to loving legacy.

Richard Bown: 00:00:04

Now at the beginning of this episode, there's a slight bit of swearing, but I

Richard Bown: 00:00:07

hope you'll agree to add some context.

Richard Bown: 00:00:09

Let's get on with the show.

Stephen Townsend: 00:00:11

The first panel's like this is how to draw an owl, and

Stephen Townsend: 00:00:13

it's like a little bit of a squiggle.

Stephen Townsend: 00:00:15

It's like the back of the wing and then a little bit more

Stephen Townsend: 00:00:17

maybe the, the tip of the thing.

Stephen Townsend: 00:00:18

And then it's the whole drawn picture of owl and it's like, step

Stephen Townsend: 00:00:21

one, do this, step two, do this.

Stephen Townsend: 00:00:22

Step three, draw the fucking owl, you know, That's what SLOs are like.

Stephen Townsend: 00:00:26

And that's what everyone focuses on.

Stephen Townsend: 00:00:28

But the other side of it was, it seems to me we need a way in the digital

Stephen Townsend: 00:00:32

era to say we wanna achieve something.

Stephen Townsend: 00:00:35

We wanna create something for, for customers and get some outcome,

Stephen Townsend: 00:00:37

a business outcome, a customer outcome, whatever it is, and we go

Stephen Townsend: 00:00:41

and build this stuff and then we never check whether we achieve that.

Richard Bown: 00:00:43

hello and welcome to the Loving Legacy Podcast.

Richard Bown: 00:00:45

I'm joined this week by Stephen Townsend and we're gonna talk about observability.

Richard Bown: 00:00:49

Stephen, thanks for joining me.

Stephen Townsend: 00:00:51

No problem.

Stephen Townsend: 00:00:52

It's a pleasure to be here.

Richard Bown: 00:00:53

So perhaps before we dive into the subject, which is

Richard Bown: 00:00:56

fantastic, I think, um, maybe you could tell everybody a little bit

Richard Bown: 00:01:00

about who you are and what you do.

Stephen Townsend: 00:01:02

Sure.

Stephen Townsend: 00:01:03

So I am currently a developer advocate.

Stephen Townsend: 00:01:07

Just going back to the rest of my career, which has been about 15

Stephen Townsend: 00:01:11

years now, I guess maybe a bit more.

Stephen Townsend: 00:01:13

I studied computer science, uh, and then I.

Stephen Townsend: 00:01:16

Decided during that time that I couldn't possibly work in technology.

Stephen Townsend: 00:01:21

So I auditioned for, uh, , which is the New Zealand Drama School.

Stephen Townsend: 00:01:25

And, and, and so I trained for three years to be a professional actor.

Stephen Townsend: 00:01:28

I thought, this is it.

Stephen Townsend: 00:01:30

This is the dream.

Stephen Townsend: 00:01:31

I'm gonna, this is gonna be my life.

Stephen Townsend: 00:01:33

And it was like a pretty awesome experience.

Stephen Townsend: 00:01:34

But I'm from New Zealand, small country, there's not a lot of acting

Stephen Townsend: 00:01:37

work . So , I started looking for technology work after three and a half.

Stephen Townsend: 00:01:41

Having finished my degree, uh, no one would hire me, uh, and as a

Stephen Townsend: 00:01:46

developer, but I managed to somehow land this job as a junior performance

Stephen Townsend: 00:01:50

tester, uh, for a consulting company.

Stephen Townsend: 00:01:52

And

Richard Bown: 00:01:53

Yeah.

Stephen Townsend: 00:01:54

then I did, uh, performance testing and

Stephen Townsend: 00:01:56

performance engineering for 13 years

Stephen Townsend: 00:01:59

. And then about a year and a half ago I

Stephen Townsend: 00:02:05

site reliability, engineering, which or service reliability engineering,

Stephen Townsend: 00:02:08

depending on, what your context is

Stephen Townsend: 00:02:11

and then about three months ago, I joined a company called Square Up who do

Stephen Townsend: 00:02:17

unified dashboarding and visualization, um, as a developer advocate.

Stephen Townsend: 00:02:20

That's been pretty fantastic.

Richard Bown: 00:02:22

Sounds good.

Richard Bown: 00:02:23

You have to deal with developers a lot, presumably as well, cuz

Richard Bown: 00:02:25

you're an advocate for them.

Stephen Townsend: 00:02:27

It's true.

Stephen Townsend: 00:02:28

Yeah.

Stephen Townsend: 00:02:28

So that's, that would be, um, the community, um, our customers.

Stephen Townsend: 00:02:33

But I'm also sort of working internally, almost like a consultant, uh, to sort of

Stephen Townsend: 00:02:38

uplift their own observability and ways of working and sort of eat our own dog food.

Stephen Townsend: 00:02:44

We say all these fancy concierges and let's try, let's

Stephen Townsend: 00:02:46

try them internally and see.

Richard Bown: 00:02:48

Fantastic.

Richard Bown: 00:02:48

Cause I, I, I've found, so my background is development and I've kind of come

Richard Bown: 00:02:52

through that path and done various bits and bobs and, but a large part

Richard Bown: 00:02:55

of that, and the bits that I've always found the most enjoying are

Richard Bown: 00:02:57

the customer touchpoints, basically.

Richard Bown: 00:02:59

How do you find that juxtaposition of requirements?

Richard Bown: 00:03:02

Because there's.

Richard Bown: 00:03:03

Push for features.

Richard Bown: 00:03:05

This is the whole point of it.

Richard Bown: 00:03:06

I suppose there's a push for features, but observability and the monitoring

Richard Bown: 00:03:10

and understanding of where we are actually are with our products.

Richard Bown: 00:03:13

Sometimes it gets overlooked, doesn't it?

Stephen Townsend: 00:03:16

So in terms of like developers or any, anyone

Stephen Townsend: 00:03:20

working in technology even.

Stephen Townsend: 00:03:23

Operations in the operations world.

Stephen Townsend: 00:03:25

Uh, I've just came from a massive enterprise before and there was a massive

Stephen Townsend: 00:03:29

disconnection between the day-to-day work or what's, what's on the minds of the

Stephen Townsend: 00:03:34

people in those engineering teams versus the actual customers and the business.

Stephen Townsend: 00:03:40

You know, that this is all for the massive disconnect because there's so

Stephen Townsend: 00:03:43

many layers of bureaucracy, uh, that the, the idea of product management

Stephen Townsend: 00:03:49

is supposed to be this conduit.

Stephen Townsend: 00:03:50

Or product ownership that sort of brings it together.

Stephen Townsend: 00:03:53

But it, it's in practice in very big organizations, it's,

Stephen Townsend: 00:03:56

it seems very hard to do.

Stephen Townsend: 00:03:59

It's, it's a bit different now working for a small product company where it's so

Stephen Townsend: 00:04:04

small that we are all together and it's more about Trying to figure out, you know,

Stephen Townsend: 00:04:07

how things are landing with customers.

Stephen Townsend: 00:04:09

You can interview them of course, but then how do you get like bigger

Stephen Townsend: 00:04:12

picture data about that that's actually meaningful and actually accurate that

Stephen Townsend: 00:04:15

helps you direct your where to go next.

Stephen Townsend: 00:04:19

So yeah, I think it's a, it's a pretty challenging thing, but, and what you're

Stephen Townsend: 00:04:24

saying about, you know, tying it back to observability, um, observability

Stephen Townsend: 00:04:29

is the buzzword obviously, and I'm sure we can go into lots of detail.

Stephen Townsend: 00:04:33

What it is and what it isn't.

Stephen Townsend: 00:04:35

Recently, I, I've always had this idea in my head, I didn't look up

Stephen Townsend: 00:04:39

a definition and no one's sort of told me, this is observability,

Stephen Townsend: 00:04:42

you know, this is what it is.

Stephen Townsend: 00:04:43

So I just thought, okay, what does it mean?

Stephen Townsend: 00:04:45

It must mean how observable something is.

Stephen Townsend: 00:04:46

Okay, what does that mean?

Stephen Townsend: 00:04:47

And it sort of, it seemed to me it was two things, but most people, when

Stephen Townsend: 00:04:51

you think of one of them, and that is, it's the ability to sort of underst.

Stephen Townsend: 00:04:55

The internal state of a complex system in order to better understand

Stephen Townsend: 00:04:59

to understand behavior remediate issues, um, look for opportunities

Stephen Townsend: 00:05:03

to optimize and tune and improve it.

Stephen Townsend: 00:05:05

And that's what everyone focuses on.

Stephen Townsend: 00:05:07

You know, massive, massive community and industry around there.

Stephen Townsend: 00:05:11

But the other side of it was, it seems to me we need a.

Stephen Townsend: 00:05:14

way in the digital era to say, we want to, we wanna achieve something.

Stephen Townsend: 00:05:19

We wanna create something for, for customers and get some outcome,

Stephen Townsend: 00:05:22

a business outcome, a customer outcome, whatever it is, and we go

Stephen Townsend: 00:05:25

and build this stuff and then we never check whether we achieve that.

Stephen Townsend: 00:05:28

There's this total missing element of we built this thing, did we actually

Stephen Townsend: 00:05:32

achieve what we set out to do?

Stephen Townsend: 00:05:34

And that seems like the most important thing in the entire world.

Stephen Townsend: 00:05:37

to me when it comes to software and.

Stephen Townsend: 00:05:41

I mean, there's this book I read recently, sooner, safer, happier by

Stephen Townsend: 00:05:44

Jonathan Smart, and it talks about the digital era we work in and how

Stephen Townsend: 00:05:49

we work with unknowable unknowns.

Stephen Townsend: 00:05:51

So it's basically chaos.

Stephen Townsend: 00:05:53

There is no way that anyone can go, here's an idea that's implemented

Stephen Townsend: 00:05:57

and this is what will happen.

Stephen Townsend: 00:05:58

It's literally impossible because of the complexity of the technology

Stephen Townsend: 00:06:01

and business and indu industry.

Stephen Townsend: 00:06:03

The only way we can actually carve a path forward is try something experi,

Stephen Townsend: 00:06:09

you know, do a little bit of, so, Put it in front of real customers and business.

Stephen Townsend: 00:06:14

See what happens and adjust that is literally the only

Stephen Townsend: 00:06:16

way that you can succeed.

Richard Bown: 00:06:18

Yeah, it's kind of like, sorry to interrupt, interrupt, but

Richard Bown: 00:06:21

you sparked something with me there.

Richard Bown: 00:06:22

It's almost like the Heisenberg uncertainty principle, isn't it?

Richard Bown: 00:06:24

It's kind of like the act of observation changes what you're

Richard Bown: 00:06:26

trying to do in the first place.

Richard Bown: 00:06:27

You know, sometimes, because you kind of get to a point where, I mean even with

Richard Bown: 00:06:32

a throw away feature implementation or something like that, you can still lose

Richard Bown: 00:06:37

sight of what you're trying to achieve by the time it's gone into production,

Richard Bown: 00:06:39

let alone be observed, you know?

Stephen Townsend: 00:06:41

Mm.

Stephen Townsend: 00:06:43

I think good observability.

Stephen Townsend: 00:06:45

What you said, uh, change, you know, the, the act of observing

Stephen Townsend: 00:06:49

it should change something.

Stephen Townsend: 00:06:50

Otherwise, what's the point if we just observe it passively and go, Hey, look

Stephen Townsend: 00:06:53

at that, it's crashing and burning, or, Hey, no one's using this thing and

Stephen Townsend: 00:06:56

they're not doing anything about it.

Richard Bown: 00:06:58

Yeah, well, exactly.

Richard Bown: 00:06:59

Even get as far as a metric on a dashboard somewhere.

Richard Bown: 00:07:02

It just gets ignored.

Richard Bown: 00:07:03

So how do you close that loop, I suppose, is the thing.

Richard Bown: 00:07:05

So that is, is that the, the kind of, the, the bit that's interests you the

Richard Bown: 00:07:08

most, the kind of closing that loop?

Richard Bown: 00:07:10

And, and if so, how would you go about that

Richard Bown: 00:07:14

. Stephen Townsend: I think

Richard Bown: 00:07:16

Yeah.

Richard Bown: 00:07:17

And of, of course, just having observability, just answering those

Richard Bown: 00:07:20

questions isn't the, the answer in itself.

Richard Bown: 00:07:23

It's, it's kind of about that relationship between different

Richard Bown: 00:07:28

stakeholders that make up.

Richard Bown: 00:07:29

An organization or you know, they're putting it all together.

Richard Bown: 00:07:32

So having that clear line of communication between the leadership

Richard Bown: 00:07:37

and the executives who have these business goals they want to achieve

Richard Bown: 00:07:41

between, you know, the products owners or managers that, those teams who want

Richard Bown: 00:07:46

particular outcome for customers maybe.

Richard Bown: 00:07:48

And then the engineers who want things to run smoothly and uh, and

Richard Bown: 00:07:53

also want to do, to be able to.

Richard Bown: 00:07:56

Quickly and reliably as well as operated effectively, uh, and combining all

Richard Bown: 00:08:00

these things in a meaningful way.

Richard Bown: 00:08:01

And I kind of say as I, as I say that, it sounds like this very

Richard Bown: 00:08:06

challenging thing because each group of stakeholders speaks a different

Richard Bown: 00:08:08

language and, but there has to be some common ground, some sort of sweet spot

Richard Bown: 00:08:16

. Richard Bown: I mean, and I see,

Richard Bown: 00:08:20

across industry completely.

Richard Bown: 00:08:21

So how do you find, yeah, do you find that it does change according

Richard Bown: 00:08:26

to the scale of the company?

Richard Bown: 00:08:27

Because I've seen this happen even in quite small companies as well,

Richard Bown: 00:08:29

the kind of, it's kind of like there's an industry standard that

Richard Bown: 00:08:32

we need to do things a certain way.

Richard Bown: 00:08:33

So for example, looking at the Dora metrics, which are, I think

Richard Bown: 00:08:37

everyone kind of agrees are a great baseline for understanding.

Richard Bown: 00:08:42

How well, yeah.

Richard Bown: 00:08:43

How can we define this in a nice way, , because it's not

Richard Bown: 00:08:47

about delivering customer value.

Richard Bown: 00:08:48

It's kind of, it's, it's, it's really about understanding.

Richard Bown: 00:08:52

We're not breaking stuff.

Richard Bown: 00:08:53

That's the way I see it in some ways, that we're not breaking things as,

Richard Bown: 00:08:55

as an experience and we're doing things at a reasonably fast pace.

Richard Bown: 00:08:58

It's not really about value, I would say.

Richard Bown: 00:09:01

What do you think?

Stephen Townsend: 00:09:03

Kind of to me says that there's like a machine which produces.

Stephen Townsend: 00:09:08

Outcomes.

Stephen Townsend: 00:09:09

It's supposedly creating outcomes for business or customer, and it tells

Stephen Townsend: 00:09:13

you how well all that machine is.

Stephen Townsend: 00:09:15

It doesn't actually necessarily tell you whether the outcomes are

Stephen Townsend: 00:09:17

happening, you know what I mean?

Stephen Townsend: 00:09:18

It's just that the machine's humming along nicely, which, you know, to

Stephen Townsend: 00:09:22

be fair, is something that, that knowledge is missing from most

Stephen Townsend: 00:09:27

organizations that I've ever worked with.

Stephen Townsend: 00:09:29

So it, it's a good starting point

Richard Bown: 00:09:31

definitely.

Richard Bown: 00:09:33

Yeah.

Richard Bown: 00:09:33

And have you been involved in the implementation of DORA

Richard Bown: 00:09:35

metrics at all or parts of them?

Stephen Townsend: 00:09:38

So we are looking at them at square up internally.

Stephen Townsend: 00:09:42

It's really interesting.

Stephen Townsend: 00:09:43

There's actually someone else leading it and I'm just sort of

Stephen Townsend: 00:09:45

assisting, but it's, I love it.

Stephen Townsend: 00:09:47

It's really exciting to me to act, actually do it and, and

Stephen Townsend: 00:09:50

have these things like, here's a metric, what does that mean?

Stephen Townsend: 00:09:53

What does lead time mean to us?

Stephen Townsend: 00:09:54

Like exactly from what point to what point.

Stephen Townsend: 00:09:57

And then even if we measure that, what does it actually tell us?

Stephen Townsend: 00:10:00

You know, how does it help us as an organization?

Stephen Townsend: 00:10:03

You know what I think is interesting about trying to implement DORA metrics

Stephen Townsend: 00:10:07

is you'll do a thing and you'll get a number and then you'll start

Stephen Townsend: 00:10:10

questioning it and it just opens up really interesting questions.

Stephen Townsend: 00:10:13

For example, you know,

Richard Bown: 00:10:14

you know,

Stephen Townsend: 00:10:15

we were talking about, um, meantime to recovery, I think it

Stephen Townsend: 00:10:20

was today, and we had this, I dunno, we had this some figure with like five

Stephen Townsend: 00:10:24

days and then we started saying, what does that, what does that include?

Stephen Townsend: 00:10:27

Like what.

Stephen Townsend: 00:10:28

Uh, bugs or incidences and does that include, and, and

Stephen Townsend: 00:10:31

do we need to classify that?

Stephen Townsend: 00:10:32

And, and then, oh, it actually turns out that there's two different kinds

Stephen Townsend: 00:10:35

of things and you know, it, it just starts really interesting conversations

Stephen Townsend: 00:10:38

about what does it mean and do we need to do something about it.

Stephen Townsend: 00:10:43

Hmm.

Richard Bown: 00:10:44

Yeah, for sure.

Richard Bown: 00:10:44

And.

Richard Bown: 00:10:46

I think it's great that, well, I find relief in it because I've been involved in

Richard Bown: 00:10:49

meetings previously in previous jobs where no one knows what to define to begin with.

Richard Bown: 00:10:54

So like what is useful lead time is kind of, is generally

Richard Bown: 00:10:57

agreed, is a, is a good one.

Richard Bown: 00:10:58

Uh, so that's the time from when a, um, either an incident or a feature

Richard Bown: 00:11:02

is created in, like, for example, JIRA to the time that it actually arrives

Richard Bown: 00:11:05

in production in front of a customer.

Richard Bown: 00:11:07

As, as my understanding of it.

Richard Bown: 00:11:08

There's, there's, there's debate over.

Richard Bown: 00:11:10

Whether or not it's behind a feature flag.

Richard Bown: 00:11:12

So is it in production?

Richard Bown: 00:11:13

Is it actually released to a customer?

Richard Bown: 00:11:14

That kind of stuff.

Richard Bown: 00:11:15

And I think that's, that's fair enough.

Richard Bown: 00:11:17

However, I found there's been so much debate over what the metrics are

Richard Bown: 00:11:21

in the first place, that sometimes the implementation gets completely

Richard Bown: 00:11:24

way further down the line, so you don't even get to that part.

Richard Bown: 00:11:26

But yeah, as you say it's great to have those conversations, but when do those

Richard Bown: 00:11:31

conversations start to be actually useful to the customer, I suppose?

Stephen Townsend: 00:11:36

Mm.

Stephen Townsend: 00:11:38

That is a very good question.

Stephen Townsend: 00:11:41

Maybe that, maybe like, I, like I was saying, they, they probably, maybe they

Stephen Townsend: 00:11:46

don't, they, what they do is they, they get rid of, All of this other mess,

Stephen Townsend: 00:11:50

which is going to get in the way of technology teams being able to deliver

Stephen Townsend: 00:11:55

outcomes to let them focus on that.

Stephen Townsend: 00:11:58

But then there's some other, there's another world around, you know, stuff I

Stephen Townsend: 00:12:02

don't know, maybe not that much about, around how do we either understand what.

Stephen Townsend: 00:12:09

A business or customer group wants or needs, and then so we

Stephen Townsend: 00:12:13

can articulate and deliver that.

Stephen Townsend: 00:12:15

Or if it's a new product, maybe without a clear set of customers already, you

Stephen Townsend: 00:12:19

know, to understand what the market needs or what, what the need is or

Stephen Townsend: 00:12:21

opportunity, what's gonna excite people, delight people, or whatever.

Stephen Townsend: 00:12:26

Yeah, that's kind of beyond my knowledge at the moment, how to do that.

Stephen Townsend: 00:12:30

I don't know.

Stephen Townsend: 00:12:30

I don't have the

Richard Bown: 00:12:31

Yeah, I think, well, I think if you can answer that, then you,

Richard Bown: 00:12:32

just to be honest, we've solved a lot of.

Richard Bown: 00:12:35

The question of why we create software, something, which kind of

Richard Bown: 00:12:38

bothers me, , because the name of this podcast is Loving Legacy and I'd

Richard Bown: 00:12:41

like to give Legacy a second chance.

Richard Bown: 00:12:44

Really like, kind of understand why we are so keen to rebuild stuff.

Richard Bown: 00:12:47

And is that just a, uh, an itch that either a founder or an owner or a senior

Richard Bown: 00:12:51

person coming into a certain role wants, they're kind of throw their weight around

Richard Bown: 00:12:55

a little bit and create something new?

Richard Bown: 00:12:57

Or is it just the case of the, the thing isn't fit for purpose anymore?

Richard Bown: 00:13:00

The, the legacy system or the way that we're working stuff.

Richard Bown: 00:13:03

Various other reasons, for example, that is laden with tech desk

Richard Bown: 00:13:06

or the technology stack's too old or, or something like that.

Richard Bown: 00:13:10

However, yeah, I, I'm really interested to, to dive down a little bit more into

Richard Bown: 00:13:16

what observability actually is and how it's implemented like these, these days

Richard Bown: 00:13:20

in inverted coms, because I'm from the old school where it's like diving into logs.

Richard Bown: 00:13:23

So you would have a server log somewhere and you'd be SSHing to a machine and then

Richard Bown: 00:13:28

grepping it and all that kind of stuff.

Richard Bown: 00:13:30

What does observability mean for you these days and how that,

Richard Bown: 00:13:33

how is that kind of approached?

Stephen Townsend: 00:13:37

Okay.

Stephen Townsend: 00:13:38

I think, well, I, it's definitely worth starting with the sort of

Stephen Townsend: 00:13:41

classical, not the classical definition.

Stephen Townsend: 00:13:43

There's no, it's not classical.

Stephen Townsend: 00:13:44

The, the software observability definitions.

Stephen Townsend: 00:13:48

I'm actually reading, I'm only a few chapters into observability

Stephen Townsend: 00:13:52

engineering at the moment.

Stephen Townsend: 00:13:54

Um, by Charity Majors, Liz Form Jones and George Miranda.

Stephen Townsend: 00:13:58

Yeah, I think I, I think I wrote down how to describe it.

Stephen Townsend: 00:14:01

It's a measure of how well you can understand and explain any state

Stephen Townsend: 00:14:05

that your system can get into no matter how novel or bizarre.

Stephen Townsend: 00:14:09

So that, that's the, that's the summarized, um, sort of

Stephen Townsend: 00:14:14

definition from the book.

Stephen Townsend: 00:14:15

And I tend to agree with it.

Stephen Townsend: 00:14:17

And I think it's about, you know, I think there's been a lot of use of

Stephen Townsend: 00:14:21

logs and, and metrics in, in the past.

Stephen Townsend: 00:14:24

My entire career has been filled with it.

Stephen Townsend: 00:14:26

I guess what's kind of new is distributed tracing because for, for better or

Stephen Townsend: 00:14:32

worse, you know, maybe legacy was good in a way cause it was simpler.

Stephen Townsend: 00:14:36

now we're serverless and containers and all these ephemeral objects and well,

Stephen Townsend: 00:14:39

they're being scattered everywhere.

Stephen Townsend: 00:14:40

It's just hard to know what's happening.

Stephen Townsend: 00:14:43

You know, a user click to button.

Stephen Townsend: 00:14:45

Where did it go and, and, and, and what was the path like?

Stephen Townsend: 00:14:48

Did it just go to one thing or did it call something a hundred

Stephen Townsend: 00:14:51

thousand times before it moved on?

Stephen Townsend: 00:14:52

Like, how, what's going on?

Stephen Townsend: 00:14:54

So, distributed tracing, I think is a good example of a technique which is

Stephen Townsend: 00:14:59

trying to solve, you know, answer that, that question like what's happening.

Stephen Townsend: 00:15:04

So that when something goes wrong and the customers, not getting

Stephen Townsend: 00:15:07

what getting outcomes they want, then you can answer those questions.

Stephen Townsend: 00:15:11

And I think that is by far, the biggest focus of what people are talking

Stephen Townsend: 00:15:15

about in the world of observability.

Stephen Townsend: 00:15:16

And don't get me wrong, I think that's hugely valuable.

Stephen Townsend: 00:15:19

It's very valuable.

Stephen Townsend: 00:15:20

I remember, you know, not even, not that long ago, like three or four years ago,

Stephen Townsend: 00:15:24

trying to do exactly the same thing that distributed tracing does, but without

Stephen Townsend: 00:15:27

distributed traces, just by getting logs from all these different servers all over.

Stephen Townsend: 00:15:31

using, you know, uh, trace studies and then writing my own, like Python

Stephen Townsend: 00:15:35

or power show codes, like pull it all together into data and then plotted in

Stephen Townsend: 00:15:38

a visualization tool to see the, the, the flow chart of how, what's happening.

Stephen Townsend: 00:15:42

You know, I, it's like, it's not new, but it's just becoming, it,

Stephen Townsend: 00:15:47

it's becoming not a manual task That took me, uh, used to take me

Stephen Townsend: 00:15:51

a week maybe to get one flow chart.

Stephen Townsend: 00:15:52

Now you should theoretically be able to get it whenever you want it, but it's

Stephen Townsend: 00:15:57

that it's the sitting out, which is hard.

Stephen Townsend: 00:15:59

So that's,

Richard Bown: 00:16:00

Yeah.

Richard Bown: 00:16:01

So sorry to jump in there as well.

Richard Bown: 00:16:02

The.

Richard Bown: 00:16:03

And one thing that I noticed a few years ago was there was a great pop, uh, things

Richard Bown: 00:16:06

like Splunk, for example, and Datadog like, um, proprietary solutions where

Richard Bown: 00:16:09

you can just point stuff at a, a lake of data as they call it, and, uh, just

Richard Bown: 00:16:14

ingest it all and then no one looks at it or you, or you write the wrong report.

Richard Bown: 00:16:18

So it's kind of a, there's a.

Richard Bown: 00:16:20

And then of course the bills start coming in and people go, oh my God, I

Richard Bown: 00:16:23

can't afford to pay for this anymore.

Richard Bown: 00:16:24

It's ridiculous.

Richard Bown: 00:16:24

So, so would have you seen Yeah.

Richard Bown: 00:16:27

Any kind of migration from the kind of one-stop shops, uh, solutions like that

Richard Bown: 00:16:31

towards more kind of homegrown, we need to get back to the actual data and then Yeah.

Richard Bown: 00:16:37

Seeing it as a kind of its own, um, how would you say it may

Richard Bown: 00:16:40

be almost department within an organization to focus on observability

Richard Bown: 00:16:46

alongside SRE or as part of sre?

Stephen Townsend: 00:16:49

I do definitely see two things.

Stephen Townsend: 00:16:52

In my last role, we, we were using Splunk.

Stephen Townsend: 00:16:54

Um, there's a lot of Splunk and also New Relic, uh, sort of from more on

Stephen Townsend: 00:16:58

application performance monitoring perspective, but even the little.

Stephen Townsend: 00:17:03

You know, we're doing, you know, a fair amount of work with those things.

Stephen Townsend: 00:17:06

They, but the bills were so high already that we were talking about, well,

Stephen Townsend: 00:17:09

maybe we should pull some of the stuff OnPrem and use an open source tool.

Stephen Townsend: 00:17:12

And our SR SRE team was building some of that stuff, you

Stephen Townsend: 00:17:15

know, standing up on-premise.

Stephen Townsend: 00:17:18

This is free for you to use Prometheus.

Stephen Townsend: 00:17:20

We'll help you set it up and configure it.

Stephen Townsend: 00:17:21

And you can use this Grafana instance to, you know, it's, it's different than what

Stephen Townsend: 00:17:26

you had in Splunk, but it's free and, and we can make it scalable, you know.

Richard Bown: 00:17:29

Observability is product, which is the the buzzword.

Richard Bown: 00:17:32

Of course, everything has to be a product these days.

Stephen Townsend: 00:17:34

Yeah, yeah.

Stephen Townsend: 00:17:36

The platform.

Stephen Townsend: 00:17:37

Oh, goodness.

Stephen Townsend: 00:17:40

I guess it's a bit of an anti-pattern, but I, I mean, I think it is happening.

Stephen Townsend: 00:17:43

Yeah.

Stephen Townsend: 00:17:44

Well, we've always had sort of IT operations teams who seem to be

Stephen Townsend: 00:17:47

the, the only people who cared.

Stephen Townsend: 00:17:49

Did anything in the word of monitoring,

Richard Bown: 00:17:52

That's it.

Richard Bown: 00:17:53

That's it.

Richard Bown: 00:17:53

You know, this whole DevOps nonsense, I would say nonsense, but it is,

Richard Bown: 00:17:57

um, comes about because people want to know what's going on.

Richard Bown: 00:18:01

You know, it's like a developer or an operations, but, or whoever

Richard Bown: 00:18:04

in between whatever color of their skin or sex or whatever, you know.

Richard Bown: 00:18:09

Some people are interested in this stuff and some people just want to turn up to

Richard Bown: 00:18:12

a day job and do a day job kind of thing, and that's nothing wrong with that.

Richard Bown: 00:18:16

It's just the investment in the product itself.

Richard Bown: 00:18:18

I've used the P word again, investment in the, in the, um,

Richard Bown: 00:18:22

in the customer experience.

Richard Bown: 00:18:24

I suppose you'd say.

Richard Bown: 00:18:24

The, the end customer experience for some is greater than for others, and

Richard Bown: 00:18:28

that drives us or them to create these or look into creation of these tools.

Richard Bown: 00:18:33

and that's why these amazing platforms like, um, Splunk

Richard Bown: 00:18:37

do turn up because Cuz yeah.

Richard Bown: 00:18:39

When it, when something like that does arrive and you can, oh, hang

Richard Bown: 00:18:41

on a sec, this makes things a lot easier for me to, to investigate

Richard Bown: 00:18:44

what was going on previously.

Richard Bown: 00:18:46

Brilliant.

Richard Bown: 00:18:47

Um, but I, yeah, I would say that the, the, the, the questioning thing has

Richard Bown: 00:18:50

always been there in, in certain people.

Richard Bown: 00:18:53

How do we make that, how do we bring that out?

Richard Bown: 00:18:55

I dunno.

Richard Bown: 00:18:55

Give 'em a difficult problem.

Stephen Townsend: 00:18:58

I was talking to someone else recently, uh,

Stephen Townsend: 00:19:00

who's an observability person.

Stephen Townsend: 00:19:02

I think there's, there are observability people, I guess now I'm one of them.

Stephen Townsend: 00:19:05

Um, and I asked the question how, you know, how do you get.

Stephen Townsend: 00:19:10

You know, engineers just who aren't their job isn't to observe things.

Stephen Townsend: 00:19:15

Uh, understanding the value of this and interested in it.

Stephen Townsend: 00:19:17

And one of the answers is, well, we need to enforce you build it, you run it.

Stephen Townsend: 00:19:22

What do you think of that, ? I'm curious.

Stephen Townsend: 00:19:25

What do you think of You build it, you run it as a concept.

Richard Bown: 00:19:28

Uh, yeah, I think as a concept it's okay.

Richard Bown: 00:19:32

Although I just have so many battle scars from early on in my career as well.

Richard Bown: 00:19:35

A long time ago when I was.

Richard Bown: 00:19:37

Doing that, building it and running it or, and actually being on

Richard Bown: 00:19:41

real page duty with a real pager.

Richard Bown: 00:19:43

And I think that's how old it was.

Richard Bown: 00:19:44

Um, and just being burnt out from that.

Richard Bown: 00:19:46

So it's fine as long as you manage that, I think in the right way.

Richard Bown: 00:19:50

So yeah, you can, you can do this.

Richard Bown: 00:19:52

It depends on a lot of, yeah, like so many answers.

Richard Bown: 00:19:54

It depends, you know, depends on a lot of things, but I like

Richard Bown: 00:19:56

the concept of it for sure.

Richard Bown: 00:19:57

And also as a big fan of, of.

Richard Bown: 00:19:59

Books like, um, the Phoenix Project Unicorn Project and, and that kind of,

Richard Bown: 00:20:03

that storytelling vibe is strong in, in the way that I feel that you mo you, yeah.

Richard Bown: 00:20:09

You most connect with a bit of software that you're working on.

Richard Bown: 00:20:12

You, you do have to put a little piece of yourself in there, you know, put a bit

Richard Bown: 00:20:16

of skin in the game, as they say, um, to, to, to make it a reality, a good reality.

Richard Bown: 00:20:21

And I don't think there's anything wrong with that, but you have

Richard Bown: 00:20:23

to approach it and manage it.

Stephen Townsend: 00:20:28

you asked me earlier what my def, what my definition

Stephen Townsend: 00:20:29

of observability was, and I talked about the software side of it.

Stephen Townsend: 00:20:32

I guess, you know, I'm thinking right now about a lot about this because I think

Stephen Townsend: 00:20:36

the other side of it for me is I guess what you traditionally would call BI.

Stephen Townsend: 00:20:41

Um, but when I think of bi , I think of like quite large teams

Stephen Townsend: 00:20:47

of people doing a bunch of stuff.

Stephen Townsend: 00:20:49

Huge amounts of data.

Stephen Townsend: 00:20:51

Data, which.

Stephen Townsend: 00:20:53

And like monthly and quarterly reports, you know what I mean?

Stephen Townsend: 00:20:56

As opposed to, I think that there's something in that, in that world

Stephen Townsend: 00:20:59

where we want to understand, you know, we've got a business objective.

Stephen Townsend: 00:21:02

We want to gain 500,000 customers this year.

Stephen Townsend: 00:21:05

Are we on track?

Stephen Townsend: 00:21:06

How are we tracking?

Stephen Townsend: 00:21:07

And I wanna see that in real time constantly.

Stephen Townsend: 00:21:09

Like, just like we would monitor anything else.

Stephen Townsend: 00:21:11

So that's the other side of it for me, is, is, is monitoring.

Stephen Townsend: 00:21:15

Monitoring as if it's like a technology system business and.

Richard Bown: 00:21:19

Yeah.

Stephen Townsend: 00:21:20

Like I said, so, so, so once you're doing that,

Stephen Townsend: 00:21:22

then you can say, are we on track?

Stephen Townsend: 00:21:24

Do we need to pivot or adjust, or, you know, what we're

Stephen Townsend: 00:21:26

doing to get their feedback?

Stephen Townsend: 00:21:27

And I, I, I don't know.

Stephen Townsend: 00:21:29

That's what I'm, I'm experimenting with right now at

Stephen Townsend: 00:21:31

Squared Up as well internally.

Richard Bown: 00:21:33

And I think, um, from, yeah, what I've seen that does, that is

Richard Bown: 00:21:37

done typically by sales teams as well.

Richard Bown: 00:21:38

So if, if sales teams have a, have a target, of course they

Richard Bown: 00:21:40

will say, well, I'll, we on track.

Richard Bown: 00:21:42

They want to know, you know, and how they, they're less concerned

Richard Bown: 00:21:44

about what they're trying to sell.

Richard Bown: 00:21:46

I would say a lot typically, I, I wanna be unfair to, to sales people

Richard Bown: 00:21:50

generally, they're less concerned about what they're trying to sell.

Richard Bown: 00:21:52

They're trying to achieve their targets.

Richard Bown: 00:21:55

Through some method, and they will track this stuff because they have

Richard Bown: 00:21:58

vested interest in making sure that those, those targets are met for

Richard Bown: 00:22:01

the, obviously for their own benefit and for the company's benefit.

Richard Bown: 00:22:04

So even in quite small companies, I've seen even scale-ups, for example,

Richard Bown: 00:22:09

maybe 200 people, uh, I've seen.

Richard Bown: 00:22:12

And get to a point where there's such a disconnect between the engineering

Richard Bown: 00:22:15

output and what's going on in the, in the, in the product and the, the sales or

Richard Bown: 00:22:19

the business side, as you might see it.

Richard Bown: 00:22:21

That, that, yeah, it'll all be held in Excel.

Richard Bown: 00:22:24

So typically the, and that's pretty much the answer for a lot of

Richard Bown: 00:22:27

business pro problems, accounting and finance, all in Excel.

Richard Bown: 00:22:31

Um, and sales or in Excel or something else, you know, or there, maybe there'll

Richard Bown: 00:22:36

be Salesforce or some something that will help a CRM perhaps, but typically

Richard Bown: 00:22:40

it'll be close, as close to the person as possible under their control.

Richard Bown: 00:22:43

Anytime you put a system in between or, or a build process between you

Richard Bown: 00:22:48

yourself and having that knowledge, it, it just, it kind of, uh,

Richard Bown: 00:22:51

disappears down into a excel hole over.

Stephen Townsend: 00:22:55

I, I guess the third group there is the sort of

Stephen Townsend: 00:22:58

product ownership management team who might be look actually looking

Stephen Townsend: 00:23:01

at information around the customer.

Stephen Townsend: 00:23:02

Like say there's a customer journey that they want people to go, like what's

Stephen Townsend: 00:23:05

the drop off rate on each step of it?

Stephen Townsend: 00:23:07

And you know, how long are they spending that, that kind of stuff, which is,

Stephen Townsend: 00:23:10

it can be really useful knowledge, but, you know, but I think the real

Stephen Townsend: 00:23:13

lost, the lost opportunity is, hey, the, we are making technology changes.

Stephen Townsend: 00:23:17

How does it change these other things?

Stephen Townsend: 00:23:18

Like these, these metrics we're following which matter to us.

Stephen Townsend: 00:23:21

, we made this big change.

Stephen Townsend: 00:23:22

We expected it to change this thing.

Stephen Townsend: 00:23:24

Did it like connecting up those three domains into one

Stephen Townsend: 00:23:29

? Richard Bown: So maybe we could

Stephen Townsend: 00:23:31

what does the company squared up?

Stephen Townsend: 00:23:33

What's your, what does your company do?

Stephen Townsend: 00:23:35

Yeah, it, it's basically the idea that, um, I'm sure from my

Stephen Townsend: 00:23:39

experience anyway, every organization that I've worked with has these monitoring

Stephen Townsend: 00:23:45

tools and usually quite a lot of them.

Stephen Townsend: 00:23:47

So one team over here has this tool, that team has another tool and it's

Stephen Townsend: 00:23:50

never unified and there's sort of these silos of knowledge and they don't

Stephen Townsend: 00:23:54

necessarily share or make it open.

Stephen Townsend: 00:23:57

So rather, I think the way, what Squared up does is that you have

Stephen Townsend: 00:24:01

this one place that you can.

Stephen Townsend: 00:24:04

Almost tells stories by dashboards and visuals, uh, and connect via in real time

Stephen Townsend: 00:24:09

using APIs to all these different things and pulling all the data in real time.

Stephen Townsend: 00:24:12

It doesn't store the data, it just displays it all and

Stephen Townsend: 00:24:15

shows it all in real time.

Stephen Townsend: 00:24:17

There's a bit more to it than that.

Stephen Townsend: 00:24:18

There does store some metadata so that you can sort of see connections between

Stephen Townsend: 00:24:21

things coming from different places.

Stephen Townsend: 00:24:22

But yeah, that, that's sort of the general.

Richard Bown: 00:24:26

Okay, so I'm, I've got, my first thought is like, so

Richard Bown: 00:24:28

do you do historical data as well?

Richard Bown: 00:24:29

So if you don't store it, do you handle history in some way.

Stephen Townsend: 00:24:34

No.

Stephen Townsend: 00:24:34

So you, no.

Stephen Townsend: 00:24:35

So, so you'd need to store wherever it's coming from.

Stephen Townsend: 00:24:38

It needs to be stored in order to have historic data there.

Stephen Townsend: 00:24:40

Yeah.

Richard Bown: 00:24:41

So bringing that knowledge into a place where it can be Yeah.

Richard Bown: 00:24:46

Acted upon.

Richard Bown: 00:24:47

So do you have any examples of when, of, of, of that in use, I

Richard Bown: 00:24:51

suppose, or a typical use case?

Stephen Townsend: 00:24:53

Yeah, well, it's a, it is a new product,

Stephen Townsend: 00:24:55

but we are using it right now.

Stephen Townsend: 00:24:57

So we have just taken a new direction, um, with the product,

Stephen Townsend: 00:25:00

really focused for the year.

Stephen Townsend: 00:25:02

Our CEO is stepping in as, uh, chief Product Officer.

Stephen Townsend: 00:25:06

And he's really driving the product.

Stephen Townsend: 00:25:08

He wants to know how is the product going in the wider context.

Stephen Townsend: 00:25:11

And so I've been running these workshops with a few, few people, including him,

Stephen Townsend: 00:25:17

sort of figure out, okay, in terms of business and customer and technology, both

Stephen Townsend: 00:25:24

the technology itself and how operating and delivering it, what, you know, what is

Stephen Townsend: 00:25:29

actually important for us to know, like.

Stephen Townsend: 00:25:31

I keep using the word indicators, so you could say KPIs, maybe.

Stephen Townsend: 00:25:34

What are the things which, which, if we look at this, actually tell us if

Stephen Townsend: 00:25:38

we're heading in the right direction.

Stephen Townsend: 00:25:39

So we've been having these really interesting workshops and uh, and,

Stephen Townsend: 00:25:42

and it's very weird to come from an engineering background and be talking

Stephen Townsend: 00:25:45

about all these business objectives and things, but it's, it's great.

Stephen Townsend: 00:25:48

It's fantastic.

Stephen Townsend: 00:25:49

So we have these workshops and then my job is to, once we figure out what

Stephen Townsend: 00:25:52

these indicators or KPIs are to go and pull them together, build a view

Stephen Townsend: 00:25:56

that shows all these things together and how they interact with each other.

Stephen Townsend: 00:26:01

The first one I'm looking at, we had this, we figured out our North

Stephen Townsend: 00:26:03

Star metric is monthly active users and we had to define what that meant

Stephen Townsend: 00:26:07

cuz it could mean a lot of things.

Stephen Townsend: 00:26:09

And then I've been using, um, going, you know, where do I get the starter from?

Stephen Townsend: 00:26:14

Uh, there's lots of different places.

Stephen Townsend: 00:26:15

Um, but I ended up going to Google BigQuery cause we seem to dump a

Stephen Townsend: 00:26:18

bunch of analytics data on there.

Stephen Townsend: 00:26:20

You, which is maybe something a traditional observability

Stephen Townsend: 00:26:23

engineer wouldn't do.

Stephen Townsend: 00:26:24

But it's fantastic, you know, it's the same process.

Stephen Townsend: 00:26:28

but you're pulling in information about customers and

Stephen Townsend: 00:26:30

business and it's, it's great.

Stephen Townsend: 00:26:32

So, um, that's what I've been doing, um, and it's early days, but it's

Stephen Townsend: 00:26:35

the first thing I've been working on

Richard Bown: 00:26:37

Awesome.

Richard Bown: 00:26:38

Yeah, cuz that's, yeah.

Richard Bown: 00:26:38

Going back to what we, we talked about earlier, that connection

Richard Bown: 00:26:41

between engineering and.

Richard Bown: 00:26:43

Is, if you can close that gap a little bit and get excited about it, then surely

Richard Bown: 00:26:47

that's kind of done deal really, isn't it?

Richard Bown: 00:26:48

Because you understand where things are going.

Richard Bown: 00:26:50

And I, I wonder, I suppose, and this is maybe a question for someone who is

Richard Bown: 00:26:54

kind of running the company or running a company who, who would like, how

Richard Bown: 00:26:59

it can also become a little bit of a following indicator maybe sometimes.

Richard Bown: 00:27:04

And you kind of like you are attempted to do, keep on doing more of the same thing.

Richard Bown: 00:27:07

So how you interpret that data is gonna be key, I suppose,

Richard Bown: 00:27:10

depending on what you're trying to.

Stephen Townsend: 00:27:14

Ah, this is terrible.

Stephen Townsend: 00:27:16

I always get, whenever you say, like leading and lagging indicators,

Stephen Townsend: 00:27:19

like I know what they are and then immediately when I think

Stephen Townsend: 00:27:21

about it, I get confused again.

Stephen Townsend: 00:27:24

, I dunno what it's about them.

Richard Bown: 00:27:25

anyway, so yeah, , so I called it the wrong thing.

Richard Bown: 00:27:27

But, yeah.

Richard Bown: 00:27:28

Well, in that, in that, in those terms, um, you are, you're measuring what you've

Richard Bown: 00:27:32

decided to measure, and then you are, and then you are making your, your decisions

Richard Bown: 00:27:36

based on what you decided to measure.

Richard Bown: 00:27:38

Maybe not looking at stuff, which.

Richard Bown: 00:27:40

You should be measuring.

Richard Bown: 00:27:41

You see what I mean?

Stephen Townsend: 00:27:42

Yeah, but that's, that's why I think all of this, I

Stephen Townsend: 00:27:46

think that anything in observability and, and, and the widest things that

Stephen Townsend: 00:27:51

I'm talking about, or even if you're doing software observability, it

Stephen Townsend: 00:27:54

needs to be thoughtful and continual.

Stephen Townsend: 00:27:56

Like it can't be.

Stephen Townsend: 00:27:57

We've set it up, we are done.

Stephen Townsend: 00:27:59

You know, we don't have to think about it ever again.

Richard Bown: 00:28:02

Yeah, and something sparked me there around a couple of things.

Richard Bown: 00:28:06

Firstly, when do you know that a metric isn't working anymore

Richard Bown: 00:28:10

and what do you do about it?

Richard Bown: 00:28:11

So kind of like is it a case of you have to kind of prune occasionally as

Richard Bown: 00:28:15

well and not show stuff, which is maybe not helpful to, to where you're going?

Richard Bown: 00:28:19

And secondly, , what's the visibility on the metrics typically?

Richard Bown: 00:28:24

I mean, yeah, it's obviously up to you or one as a business to decide, but would you

Richard Bown: 00:28:29

advocate for sometimes it being completely open and global and everyone looking at

Richard Bown: 00:28:33

being able to see it on a dashboard or, or on a Bayer or something like that?

Richard Bown: 00:28:36

Or would you say no, sometimes we need to kind of bring, just

Richard Bown: 00:28:39

keep this to ourselves, you know?

Richard Bown: 00:28:41

And when I say public, maybe public internally rather than

Richard Bown: 00:28:43

public to to, to customers.

Stephen Townsend: 00:28:45

Yeah.

Stephen Townsend: 00:28:46

Uh, I generally of the opinion, I like the, I've, I've liked the term

Stephen Townsend: 00:28:50

democratizing data, so making it a free and available for anyone to

Stephen Townsend: 00:28:54

consume should be no secrets to hide.

Stephen Townsend: 00:28:56

Um, So that's generally my perspective.

Stephen Townsend: 00:29:00

But I've been in situations going back earlier in my career where I know too

Stephen Townsend: 00:29:06

much about the finances and the health of a company and the pressures that

Stephen Townsend: 00:29:10

executives are going through, and that just trickled down to me, and then I

Stephen Townsend: 00:29:12

felt an incredible anxiety about it.

Stephen Townsend: 00:29:14

So I think there are, you know, everyone doesn't need to know everything.

Stephen Townsend: 00:29:18

I think sometimes it's, it's good to not push the, all the stress

Stephen Townsend: 00:29:22

and burden and anxiety down the company until everyone's feeling it.

Stephen Townsend: 00:29:26

Yeah.

Stephen Townsend: 00:29:27

There's a lot of that sort of short term thinking when , when the direction of

Stephen Townsend: 00:29:31

a company is just constantly changing based on who, who's in leadership at

Stephen Townsend: 00:29:35

that particular time, or what someone thinks they need to do to get a bonus.

Richard Bown: 00:29:39

So it's not even about the product, it's more about the process.

Richard Bown: 00:29:42

It's easy to get distracted and easy to kind of like get all these dashboards.

Richard Bown: 00:29:47

Sorry, gone down the black hole there.

Stephen Townsend: 00:29:49

No, no, no.

Stephen Townsend: 00:29:50

I, I know what you mean.

Stephen Townsend: 00:29:51

We change things too often and it's a huge, it's, it's, it's, it's a real shame.

Stephen Townsend: 00:29:56

The, one of the things I used to love about performance testing is that when

Stephen Townsend: 00:30:00

I used to do workload modeling, uh, which is you, you want to understand.

Stephen Townsend: 00:30:05

, you know, what sort of load is a system gonna be under in terms of,

Stephen Townsend: 00:30:08

you know, different user groups and what they're doing and the key, you

Stephen Townsend: 00:30:11

know, services or transactions they're completing and, and how that fluctuates

Stephen Townsend: 00:30:15

and changes over the year and month and the week and the day and the hour.

Stephen Townsend: 00:30:19

That's, I love that.

Stephen Townsend: 00:30:20

That was a really interesting part of that job.

Stephen Townsend: 00:30:22

And some clients I'd go to and they would've like seven years of data

Stephen Townsend: 00:30:26

and being able to go back and say, see their whole business and all the

Stephen Townsend: 00:30:30

customers and maybe the orders for a retail site that have gone through for,

Stephen Townsend: 00:30:33

for seven years and it showed them.

Stephen Townsend: 00:30:35

It's just amazing, you know?

Stephen Townsend: 00:30:37

And being able to see very clearly, yeah.

Stephen Townsend: 00:30:39

Oh, year in year, 20% growth.

Stephen Townsend: 00:30:41

We've got fair amounts of confidence that's gonna happen again.

Stephen Townsend: 00:30:43

We've gotta be ready for it.

Stephen Townsend: 00:30:44

Are we ready for it?

Stephen Townsend: 00:30:45

You know, I think our historic data's fantastic.

Stephen Townsend: 00:30:48

If you can, if

Richard Bown: 00:30:49

That is, yeah, that is brilliant.

Richard Bown: 00:30:50

So when you can actually put that in front of someone and show them, then

Richard Bown: 00:30:53

it's like light bulb moment for sure.

Richard Bown: 00:30:55

So how typically would that then move on?

Richard Bown: 00:30:57

Would they say, great, fantastic, thank you very much.

Richard Bown: 00:31:00

We can we, we've got it from now?

Richard Bown: 00:31:01

Or would they then implement something which would allow them to have

Richard Bown: 00:31:03

this, this insight on a, on a daily?

Stephen Townsend: 00:31:06

Well, this is going back to the mean, this is one of the

Stephen Townsend: 00:31:09

reasons I wanted to, I, I wanted to move out of performance testing and

Stephen Townsend: 00:31:12

engineering, is that you'd do this thing once and use it as an input into

Stephen Townsend: 00:31:17

how you test the, you know, low test.

Stephen Townsend: 00:31:20

Uh, and you'd put a.

Stephen Townsend: 00:31:22

You know, you'd say, here it is, and you'd put it down and

Stephen Townsend: 00:31:23

everybody's go, that's amazing.

Stephen Townsend: 00:31:24

And then never look at it again.

Stephen Townsend: 00:31:25

You know, there would be no setting or something up to,

Stephen Townsend: 00:31:27

to continue to look at it.

Stephen Townsend: 00:31:29

It would just be a one-off thing.

Stephen Townsend: 00:31:31

And so that's why I, I thought if I get into this field of sre, it sounds

Stephen Townsend: 00:31:36

like, uh, I'm gonna get a chance to, you know, to set things up like that, which

Stephen Townsend: 00:31:41

I think actually far more valuable in.

Stephen Townsend: 00:31:44

Then creating, you know, spending weeks or months creating this elaborate low

Stephen Townsend: 00:31:48

test suite and running it in, in this environment, which is in production.

Stephen Townsend: 00:31:52

You know, it seems like a huge waste opportunity at times.

Richard Bown: 00:31:57

Yeah, and you'd hope with the.

Richard Bown: 00:32:00

Is that we have now in terms of observability, that you can do it

Richard Bown: 00:32:04

in production basically as well.

Richard Bown: 00:32:06

It's kinda like in some ways come full circle.

Richard Bown: 00:32:08

We've tried to kind of do it in dev or do it in test by making all our

Richard Bown: 00:32:15

envi, aligning all our environments.

Richard Bown: 00:32:17

But again, it comes down to costs.

Richard Bown: 00:32:18

A lot of the time you don't wanna be able to stand up a dev or test environment,

Richard Bown: 00:32:21

which is the same size or performance as production because it costs too much,

Richard Bown: 00:32:24

especially cloud-based these days or two.

Richard Bown: 00:32:26

So the only place you're gonna get meaningful data is in.

Richard Bown: 00:32:30

maybe that's, that's the convergence point now, isn't it?

Richard Bown: 00:32:33

Is that, and that's why this is such an, an interesting field because it's got

Richard Bown: 00:32:38

to a point now where we've realized that there's only one place where we can do it.

Richard Bown: 00:32:40

So we might as well get this, uh, super fast.

Richard Bown: 00:32:42

You know, same, same with DevOps.

Richard Bown: 00:32:44

I mean, for many years it was, oh yeah, we need to create poor requests

Richard Bown: 00:32:47

and we need to feature branches to be able to separate all of.

Richard Bown: 00:32:51

Different streams of work, but now it's like, no, we all need to do trunk based

Richard Bown: 00:32:54

development and we wanna get stuff into production as soon as possible, but make

Richard Bown: 00:32:57

sure that it's good before we get there.

Richard Bown: 00:32:58

How can we kind of shave that process off as well?

Richard Bown: 00:33:01

So, or make that process slimmer, I should say.

Richard Bown: 00:33:04

Okay, that's really interesting.

Stephen Townsend: 00:33:06

I have this, um, I, I host a podcast called Slight

Stephen Townsend: 00:33:09

Reliability, and it's about sre, but I got a, cuz I know from the community, a

Stephen Townsend: 00:33:14

performance engineer called Joey Hendrix.

Stephen Townsend: 00:33:16

And there's an interview I did with him.

Stephen Townsend: 00:33:19

He does performance testing in production and he used this,

Stephen Townsend: 00:33:22

um, analogy of Plato's forms.

Stephen Townsend: 00:33:25

So Plato has this idea that there's this perfect form out there.

Stephen Townsend: 00:33:28

It's the perfect of every, of, of a particular thing.

Stephen Townsend: 00:33:31

And it just, and in the world of performance testing, that's

Stephen Townsend: 00:33:34

production and anything else we do is like a imperfect, you know,

Stephen Townsend: 00:33:38

representation of, of production.

Stephen Townsend: 00:33:40

So, and everything we get out of it.

Stephen Townsend: 00:33:42

It's not necessarily reliable.

Stephen Townsend: 00:33:45

Yeah.

Stephen Townsend: 00:33:45

Then that's, of course, you have to be very careful and thoughtful if you're

Stephen Townsend: 00:33:48

gonna perform a test in production about when you do it and what you do.

Stephen Townsend: 00:33:51

You know, you might can't necessarily add new customers and put financial

Stephen Townsend: 00:33:54

transactions through without a lot of, you know, preparation and

Stephen Townsend: 00:33:57

forethought on how you're gonna do that.

Richard Bown: 00:33:59

It's a lot of as covering is all it is.

Richard Bown: 00:34:00

I mean, that's, and that's it's all what it, that's all it comes down to.

Richard Bown: 00:34:04

Someone doesn't wanna lose, lose their job because of the, it was,

Richard Bown: 00:34:08

it was on their watch, essentially.

Richard Bown: 00:34:09

So they said, okay, you can do it without doing, I dunno, several acceptance tests.

Richard Bown: 00:34:13

Acceptance tests runs before you go somewhere, you know?

Richard Bown: 00:34:16

Um, and.

Richard Bown: 00:34:18

Yeah, I wonder if that's changing slowly.

Richard Bown: 00:34:20

I mean, for example, I've worked in the financial sector for quite a while,

Richard Bown: 00:34:24

um, on various large projects which are around, yeah, accounting and finance

Richard Bown: 00:34:28

and the goli are always very slow.

Richard Bown: 00:34:29

The development process was typically quite slow as well.

Richard Bown: 00:34:32

So we'd be lucky if we get a couple of races out a year typically

Richard Bown: 00:34:35

what's the worst that could happen?

Richard Bown: 00:34:37

Yeah.

Richard Bown: 00:34:38

you know what?

Richard Bown: 00:34:39

The most successful companies are doing that right now, and they're

Richard Bown: 00:34:42

doing, they're following the best practices that we read about in

Richard Bown: 00:34:44

all these great books, accelerate, et cetera, and DevOps handbook.

Richard Bown: 00:34:47

They do that already and they understand it fundamentally.

Richard Bown: 00:34:50

So that kind of brings me round, that's, sorry I'm ranting slightly here, but

Richard Bown: 00:34:54

it kind of brings me round to think, should we only spend our time working

Richard Bown: 00:34:57

for companies who really get it?

Stephen Townsend: 00:35:01

Um, I, I think so.

Stephen Townsend: 00:35:02

If you're the kind of person who really cares about.

Stephen Townsend: 00:35:05

You know, having an impact and doing, you know, doing good work

Stephen Townsend: 00:35:08

and, and seeing things improve.

Stephen Townsend: 00:35:10

Yeah.

Stephen Townsend: 00:35:10

Um, I know I've got, I, I go crazy.

Stephen Townsend: 00:35:13

I, I, I get really mentally unwell in a situation where

Stephen Townsend: 00:35:16

there's, there's of stagnation and not nothing's moving forward.

Stephen Townsend: 00:35:20

So yeah, if, if you're that kind of person, I think absolutely.

Stephen Townsend: 00:35:22

Go find a, even, even a team within a company who's doing, I think potentially

Stephen Townsend: 00:35:29

can give you that sense of progression.

Stephen Townsend: 00:35:31

Um,

Richard Bown: 00:35:32

I think this is a subject for a whole separate

Richard Bown: 00:35:34

episode, to be honest, isn't it?

Richard Bown: 00:35:36

? Yeah.

Richard Bown: 00:35:36

Because yeah, if you're committed to a piece of work, you put a lot of

Richard Bown: 00:35:39

effort into it, you wanna make sure that it's, it's the best that you can

Richard Bown: 00:35:42

be and also that the company that you work for is the best that it can be.

Richard Bown: 00:35:45

So that's the, going back to the pride, going back to the

Richard Bown: 00:35:47

commitment that you put into a role.

Richard Bown: 00:35:48

So yeah, let's talk about that maybe in the future.

Richard Bown: 00:35:50

That'd be really cool.

Stephen Townsend: 00:35:51

Okay.

Richard Bown: 00:35:52

And you mentioned also slight reliability, which is

Richard Bown: 00:35:54

how I first came to, first came to talk to you in the first place.

Richard Bown: 00:35:57

I love your podcast.

Richard Bown: 00:35:58

I think it's fantastic.

Richard Bown: 00:35:59

Um, best of luck with that.

Richard Bown: 00:36:01

Steven, I just wanna say thanks for joining me

Richard Bown: 00:36:02

. Stephen Townsend: No, it's

Richard Bown: 00:36:04

And I'd love to come back and talk about, you know, working for

Richard Bown: 00:36:08

a, a company which makes, which aligns with how you wanna feel.

Richard Bown: 00:36:11

Yeah.

Richard Bown: 00:36:11

Yeah, that's a really cool way of putting it.

Richard Bown: 00:36:13

Okay, great.

Richard Bown: 00:36:14

Good talking to you.

Richard Bown: 00:36:16

Many, thanks to Steven for agreeing to come on and thanks

Richard Bown: 00:36:19

to you for joining me to.

Richard Bown: 00:36:20

Join me next time on loving legacy.

Richard Bown: 00:36:22

And if you enjoyed this episode, please feel free to tell all your friends

Richard Bown: 00:36:25

and colleagues about it and give us a rating on your favorite podcast app.

Richard Bown: 00:36:29

Until next time.