So, hello, everyone. Thanks for joining us today to the Postgres Dayroom. Our next speaker
is Chris Trevers, who flew all the way from Indonesia. Thank you. And he's going to talk
about why database teams need human factor training. Thank you.
Thank you very much for coming to this talk. I think it's certainly one of the topics I'm
most excited about when it comes to database-related topics, actually, even though I'm very, very
much into Postgres. This topic really excites me. So, just introducing myself a bit for
those who don't know me. I have a bit over 24 years of experience in Postgres, so almost
25 years. I've built accounting software on it. I have worked as a database administrator
on very large databases, built database teams, managed infrastructure, a lot of that sort
of thing. So, I have a wide range of experience. I have submitted several patches in for Postgres,
so which one has been accepted, and I'll probably be submitting more patches at some point in
the future. So, I absolutely love Postgres for its extensibility. And with that, of course,
comes some complexity, some difficulties in kind of maintaining our mental models about
how things are actually working. And especially if you're working at things in scale, it's
really easy for your mental model not to match what's actually happening, and then sometimes
we make mistakes and things happen. So, this talk is basically going to be about two things.
The first thing is something our industry doesn't do very well. And that is how we look at human
error, and how we can possibly do that better. I kind of want to talk a little bit about
how we can improve, and what the benefits are that we can expect from some of the early
steps that we can take as an industry. So, this is very much a talk about database people.
It's a talk about us. It's much less a talk about like a specific technology. But a lot
of the same technical approaches apply. So, I want to give a few thanks, first of all,
timescale for paying me to come and do this. Wouldn't really be feasible for me to fly
from an issue without them. But I also really want to thank two of my prior employers. I
want to thank Adjust, where we were actually able to bring in aviation training on these
human factors. So, we brought in a company that did training for pilots, as well as doctors.
And a lot of the training was really eye-opening, and it allowed us to do some things that we
couldn't do before. This was really a grand experiment, and it had a number of really
big, tangible benefits. And then, of course, I also want to thank Delivery
Hero, who I worked after that, where I was able to kind of work with people and evaluate
both the successes and the shortcomings of what we had done at Adjust, and further develop
some of these ideas. So, these are important areas, and I would also say that I'm involved
in trying to help implement some of these things also at timescale. So, introduction.
So, just as a, this is a completely rhetorical question. You don't have to raise your hand
if you don't feel comfortable doing so. But how many of us have been on a team where somebody
has been working on a production database while they're drunk?
Yes, I see. I mean, as we go through our career, almost every single one of us will probably
have that experience. And yet, how many times does it cause a major problem? Almost never.
At least, I've never seen it cause a major problem. Now, part of that may be the context
in which it happens, like, you know, the subject matter experts spent out partying was not
expecting to, was not really on call and has now been called in an escalation. Somebody
else may be handling a lot of the sort of wider incident strategy stuff where maybe
alcohol might be a bigger problem. But at least in these contexts, alcohol doesn't seem
to be a big factor in the further disruption of things once stuff's going on. But let me
ask another question. How many people here have seen a case where a major incident or
outage happened because somebody made a mistake because that person was tired? See? So, we
valorize the thing that causes us problems. Well, we demonize something that probably
does cause some problems, no doubt. And maybe the demonization helps prevent more problems,
but we valorize something that causes a lot more problems. Okay? Why isn't we do that?
How is it that we should stop doing that and actually we think our priorities? Now, on
one side, this is a good example of human error, right? We can talk about all the factors
that go into that prioritization. But on the other side, it's also partly because we don't
understand human error in our field, right? When we do a post-mortem, if somebody made
a mistake, we just say, oh, human error, and that's it. I'm going to come back to that
point in a few minutes. So, drunkenness versus fatigue. Now, if one person drinks, say, a
bottle of wine, and another person, one group of people drinks each a bottle of wine, and
the other group of the people has, say, their sleep disrupted, so they're only sleeping
four hours, and then get up, and a few hours later, they're in the other group. Give both
of them complex tasks to perform. Who's going to perform worse? Sleep deprivation causes
heavier cognitive deficiencies. Four hours of sleep, missing sleep, is worse than four
drinks. Now, obviously, there are some tasks where that's not the case, like driving a
car or something, because you also have coordination problems induced by the alcohol. From a peer
information processing standpoint, having four hours of sleep only is worse than drinking
a bottle of wine, and it's going to last at least the next day. So, totally, totally worth
thinking about that. So, now that I've talked about, like, one aspect of human error, one
thing that can induce a lot of human error, I want to talk about a brief history of why
this field became really big in aviation. So, back in the 1950s, 1960s, 80% of the aircraft
accidents or incidents were blamed on pilot error. Notice I didn't say human error, I
said pilot error. I'm going to come back to that distinction in a moment. In fact, I think
the number might have been closer than 90%. Today, our incident and accident rates in
airlines are well over 100 times lower than they were at that point. So, if you think
about it, improvements in the technology in men and of the airplanes could only account
for maybe 10% of that improvement. All of the rest of this is due to much better understanding
of the question of human error. There's been a shift from focusing on pilot error to focusing
on human error. When the aviation industry talks about human error, they don't mean somebody
made a mistake, and that's where they leave it. They have a rich taxonomy of understanding
kinds of human error, causes of each of these particular types of errors, and sort of practices
to try to mitigate them. The way I would actually describe this difference is that if you're
debugging software and it's connecting to the database, and every time, let's say you
have an error in your query or something the database can't fulfill your request, it just
says something went wrong. You're not going to be able to debug that software at all,
and you're probably going to have a lot of trouble. That's kind of what we do currently
when we say human error. We just simply say the person made a mistake, and that's as far
usually as we look. The aircraft industry has actually come up with something with a much
richer understanding of this topic and sort of richer system of almost like error codes
that they use when they talk about these issues. The reason is it's a very unforgiving environment.
If you make a mistake, you might or might not be able to recover, so you have a lot
of training on this, and now the chance of a massive disaster is down probably one error
disaster per billion takeoffs, which is really impressive. We'd love to have that.
So they've made this shift. They've also made a shift that we've already made, and it's
worth pointing this out, and that's that they've made a shift from individual responsibility
to collective responsibility. In our terms, we call that blameless culture. Somebody makes
a mistake. We don't blame them. We don't go, hey, stop making mistakes. We try to find
some way to keep that mistake from happening, but because we don't have a clear understanding
of this topic, we try to solve this in ways that maybe aren't as effective as they could
be. I want to give one really good example of sort of a watershed moment. Actually, before
I talk about that, let me just discuss David Beatty's contribution quickly. Beatty was
a cognitive psychologist and pilot in the U.K., and in 1969, he wrote a seminal book
called The Human Factor in Aircraft Accidents, where he basically looked at the kinds of
mistakes that happen and the kinds of circumstances that lead to those mistakes. There are newer
versions of that book out now. It's actually worth reading, but probably not best to read
if you're a nervous flyer. As a good description of how we break down error, it was the industry
starting point. Ten years after that, there was I think the watershed moment in how this
became really big within aviation, and that was the Tenerife disaster. Tenerife disaster
was the most deadly aircraft accident still in history. It happened on the ground in Tenerife
due to a variety of factors. I'm not sure how much detail I should go into in this talk,
but the end result was basically that 1-747 tried to take off down a runway with limited
visibility without a proper takeoff clearance, and they hit another 747 on the ground. Clear
case of human error, and the Spanish report on this more or less blamed it on pilot error.
This guy tried to take off. He didn't have the clearance. It was his fault. The Dutch
report, which is often criticized in some documentaries that I've seen on this, was
very, very different. What they actually did was they asked, why did he try to take off
without clearance? What was going through, how did that mistake happen? The thing was,
he was an extremely experienced pilot. He was their chief pilot. He actually didn't fly
airplanes that much. He was mostly sitting on simulators. The thing was, at that time,
when you were the senior pilot in the simulator, you were giving the clearances. A stressful
situation, visibility is slow, there's pressure to take off, stressful situation. He goes
back to what he's used to doing, which is assuming he has the clearance because he's
used to giving them to himself. Airlines don't do that anymore in their simulators,
for obvious reasons. The Dutch report actually became the basis for how the aviation industry
has started to look at human error ever since. As a result, what we've seen is we've seen
this massive, massive improvement in safety. Every pilot in every airline gets this sort
of training, and it has made our flights much, much, much, much safer.
The question is, can we benefit from the same thing? The answer is yes. We actually can get
a lot more benefits from it than just reducing incidents and recovering from them. In fact,
if you look at the standard definition that people give of crew resource management, it's
the use of all available resources to ensure the safe and efficient operation of the aircraft.
If we can use all of our resources to improve both safety and efficiency, that's going
to make our jobs better, we're going to be more productive, we're going to be happier.
This is actually a really, really important field that I think that we need to improve
on. Now I'm going to talk about how we look at
human error in the industry. Human error typically in the DevOps and SRE systems, we have one
answer to human error. What that is, automation. If somebody made a mistake, we're going to
automate that away. We're just going to add more automation. We're going to add more automation.
It seems like a great idea. Computers are infallible, we're fallible, so we're just
going to use the computers to prevent the mistake. The problem with this, the IEEE
has done a bunch of research on something they call the automation paradox. The automation
paradox is that the more reliable the automation is, the less opportunities humans have to
contribute to the overall success of that. I think I'm going to take a little bit of
time here to talk about why that is the case, and then I'll get reinforced in the next section
when we talk about why we make mistakes. But to start with a basic summary, obviously
we need automation because there are certain kinds of tasks that we're actually very bad
at following, and there are certain kinds of requirements where automation can really
save us a lot of safety considerations. So steps that have to be done together really
should be automated so that they happen together. But automation is just done reflexively, at
least according to a lot of the research that's come out of the IEEE as well as many of the
aviation study groups on this, is that simply throwing automation at a problem can actually
make human error more common and can make human error more severe, and then when things
are out of whack, you have no possibility at all of saving, of preventing a major incident.
Part of the reason here is that we process all of what we see through a mental model,
and so when we add more complexity, when we add more automation around a lot of this,
we make it really, really, really, really, really hard for us to keep that mental model
reasonably in sync with reality. And then when something goes wrong, we can spend a
lot of time and effort struggling to understand what's going on, or we may reflexively react
in ways which actually make the problem worse. So automation isn't the answer, it is part
of an answer. And reflexive automation, oh, we had a problem that's automated away, is
not the answer. Now, I mentioned just a moment ago this issue of mental models. We humans,
we operate in a world that's very different from the way computers operate. Computers
are basically systems that mathematically process inputs and produce outputs, right?
And therefore, computing programs basically operate in a closed world. We humans don't
operate in closed worlds, right? We all operate in open worlds. We have the situation where
we know we don't know everything. We know we don't know some things. We know, well,
and then we don't know other things that we don't know we don't know, right? Some cases
we know we don't know what we don't know, right? But in order to function, we have to
maintain these mental models, and those mental models are necessarily a simplification of
reality. And so when something's going wrong, we have to dig into how we think the system
works and we have to kind of go through that. And the more complexity we throw into our
automation, the harder that process becomes, right?
So automation, as I say, is an important tool. I'm going to talk in a few moments about
good automation versus bad automation, but it's something that we can't rely on to solve
the human error problem. So I mentioned that I talked about good automation versus bad
automation. I think this is really, really, really important here. So oftentimes what
I've seen happen is that you end up with large automated systems, whether they're something
like Ansible or Rax or Kubernetes or whatever. And oftentimes there isn't a clear understanding
of how these things work underneath. Now, if people do understand all of that, and they've
built in a lot of these things, then a lot of that's going to be a lot easier, right?
So good automation is basically going to be a deliberate and engineered process, right?
Rather than something that's thrown together in the course of the messy world of operations,
it is a deliberate process which is designed around two factors and three factors, actually.
The first factor is the system, right? The second factor is the people, and we usually
forget that one. And then the last one is that we actually need to be thinking about
the human machine interaction, right? So good automation takes the people into account.
Good automation is something which has built in decision points where the person can actually
sit there and say, hmm, this isn't going right. We're not going to proceed, right? And good
automation is sort of then a well-understood process, right? So the other thing that is
really important as we look at automation is this issue of feedback, right? Because the
more we automate, typically the more we insulate the individual from the feedback of the individual
steps that would be right, right? So it's really super important to sit down and think
about what's the person going to see? What's the human going to see? How's the human going
to be able to interpret this? How much feedback do we want to send? Do we want to send everything
that we got? Do we want to send some summary of it? And those are going to be decisions
that have to be made deliberately based upon the context of what we're doing, as well as
a clear understanding of what the failure cases of the automation are. And then of course
people actually need to be trained on what the automation is actually doing under the
hood so that they understand it, rather than just simply saying, oh, push button, okay,
everything good. So the way I always look at it is a lot of people think automation
basically, a lot of people think checklists are a step towards automation. I think that
automation should be a step towards a checklist, okay? The relationship should actually be
something around on the other side so that you're thinking about how do I want the human
to interact with this? How do I want the human to perform these? Where do I want the human
to be able to say this isn't going while we are stopping? And those are the sorts of questions
and designs that we have to think about when we're dealing with especially these sorts
of critical systems like the databases, where if the database is down, you know, the business
maybe now. So now I want to talk a little bit about why we make mistakes. Now, I'm
mentioned before, computers operate in a closed world, right? They get inputs from us. They
do processing. They give us outputs, right? We live in an open world. We have, we experience
things, we perceive things. What we perceive is not a complete model of, or it's not even
complete aspect of what our mental models are. We make inferences based on incomplete
data, okay? And in order to function in this world, we have had to adapt and develop certain
kinds of cognitive biases, okay? And a lot of times people look at this and they go,
oh, it's not good to be biased. Bias is a bad word. We don't like biases. But the fact
of the matter is that if you could get rid of all of your cognitive biases, you would
be unable to function, okay? Confirmation bias, of course, is one thing that we tend
to be aware of. But here's another one, continuation bias. Continuation bias is the tendency to
continue to follow a plan you've put in motion, even when you're starting to get good indications
that that's not a good idea, okay? If you didn't have continuation bias, you might have
to sit down and rethink your plan continuously over and over and over again, right? That
wouldn't be very helpful. So continuation bias, just like confirmation bias, actually
helps us function in the real world. Problem is, it can also lead us into situations where
we do the wrong thing. And so understanding these biases, understanding their implications
is very clear, is a very important step to being able to notice when they're causing
problems and start to trap those sorts of problems. So rather than trying to eliminate
our biases, which is, I think, a way in which I see people typically trying to do this,
is better to think about what kinds of problems the biases can cause and how we can detect
and trap those problems, right? And there are a large number of these biases, right?
Expectation bias. Expectation bias is also related to confirmation bias. It's the tendency
to filter out perceptions that don't match your expectations, right? This happens today
in a lot of environments. It happens in our industry. It obviously still happens in aviation,
fortunately, usually not with serious problems. The most common problem it causes there is
that the plane comes up to the gate, the pilot says disarmed doors and cross check. Somebody
misses the door that's going to be opened. The other person cross checks and expectation
bias kicks in and they don't notice that the door is still armed. Go to open the door and
guess what happens. Emergency slide deploys. It doesn't harm anybody on the airplane, but
it's going to make a bunch of people unhappy because the next leg on the airplane's flight
is going to get canceled. And that's usually the worst that happens.
But these are important things and we have to recognize that these sorts of biases are
going to happen and that our ability to maintain a situation awareness in the course of these
biases is very much tied to, very much tied to how aware we are of the kinds of problems
that they can cause, right? Because, you know, we form this mental model. We're going to
interpret things according to that mental model. We're going to continue our existing
plans and things like that. And when somebody says, hey, wait, maybe this isn't right, then
that's suddenly an opportunity to go, hey, my bias is maybe leading me astray. Let's
sit down and figure out what's going on and verify.
And human factors training actually tends to include exercises or training specifically
aimed at doing that. So, second major issue is reversion to prior behavior under stress.
Something that happens to all of us when we're under stress, our focus narrows, right? We
start filtering things out and we start resorting to habit.
What this also means is that in a database team, when there's an outage, if we're not
careful, we will resort to the things that we're used to doing, even if we have decided
that they're not maybe the best ways forward. And, you know, I've watched cases where, you
know, incidents happen and, you know, if a company has been really trying to move towards
a more collaborative approach to incidents, that suddenly when the incident happens, people
are getting stressed out and they're going back to this like hyper-individualistic cowboy
incident response. And a lot of that is just simply due to stress. It's a very well-documented
part of the stress response. One thing that we got at a just with the human factors training
was a strong understanding of that problem as well as good understandings of how to measure
the stress so that we could actually kind of keep an eye on it.
Another major point that causes problems, and I've alluded to this before, is fatigue.
How often do we see people who have a rough on call night, who come back in the next day
and start working on stuff? How often are we willing to say to that person, no, go home,
get some rest, I don't want you working on this stuff right now? Right? How often have
we seen people who are on call for an extended time period and a rough shift make mistakes
after several days of continuous sleep interruptions? You know, do we start to think about the question
of maybe when this happens, we should be switching these people out more frequently.
In the airlines, before any flight happens, the flight crew get together and they check
out how each other are doing, right? And there is an expectation that there is a standby
flight crew so that if you're not feeling your best, you can say, hey, I didn't sleep
well last night, I don't want to fly. And that's another thing which has really helped
the increase of the safety, something we should probably think about doing. You're getting
tired from the on call, time to switch you out. Do we? I have never worked anywhere
the day. So a final major point on how and why we make mistakes has to do with a term
in human factors, Lingo, called workload. Now, I don't like this term in this context
because when we say workload in here, everybody is thinking, oh, I have so many things I need
to get done this month. But in the human factor side, workload doesn't mean over the next
month or over the next week, although planning that can be helpful. What it really means
is how many tasks are you having to pay attention to right now? How many people here can actually
listen to and understand two conversations at the same time? Nobody? Maybe it's possible
for some people to train that. But a brain can't, there are certain kinds of things
that our brains can't parallelize very well. Understanding where those boundaries are.
Switching and flipping between tasks. How much can we reduce that workload? That's actually
really important because one of the things I've seen happen is you have your standard
runbook and the way most people write their runbooks is you have step, explanation, discussion
of output, next step. What happens at three in the morning if you've never done this particular
process is step. Okay? Yes, it did what I expected to. Where's the next step? It becomes
really, really, really easy to miss the next step in your checklist or to miss critical
details that are kind of obscured in the fact that now you're having to read through paragraphs
at three in the morning while troubleshooting a broken system. One of the things that I
did while it was at adjust is I started writing some of our, I guess I would call them unusual
procedure checklists. A non-normal procedure checklist. So things that happen when, things
that you do when something goes wrong. Things that you might have to do at three in the
morning without doing them for any of the previous three months. And what I ended up
doing in this case, and it was actually, this is a good opportunity to talk about some of
the main benefits of this sort of training, is that we talked about, we talked about basically
what we did was we did the following format. It's a bullet point. Here's what you can ideally
copy and paste into the terminal. Expected output, warning signs, all in bullet points
and then back, unindented again, the next bullet point. So they're hierarchical, it's
easy to scan, but then your main points are all really, really, really short. And then
all of the major description that would be in those paragraphs would be moved into foot
notes. And those would all be hyperlinked. So you run a test, you know, you run a step.
Something doesn't look quite right, you want to see the longer description, you click that
hyperlink, you come down to the footnote, you read the whole thing, decide if you want
to proceed or not, and then decide. And what this allowed us to do was to take, like the
standard platform team people who are on call and actually have them do error spike maintenance
at three in the morning on, as I say, the super critical high-speed database system.
And before that, every time there was an error spike issue, it was an automatic escalation.
And it was automatic escalation because we didn't trust that they would be able to do
it or make proper decisions around it. But since we formalized it into checklists and
we offered some training on them, and we tried to make sure that people kind of understood
the overall considerations of the processes, then they could do some basic stuff and then
call us if there were questions that weren't obviously answered by the documentation.
Every very, very good tangible benefit meant that instead of several people waking up in
the middle of the night, they could be done by the on-call engineer.
So that's a really good example of the benefits that come out of paying attention to that
workload issue and the sensory overload that happens that's much more serious at three
in the morning than at three in the afternoon. So at this point, it's really important to
recognize that at this point, we're no longer really talking about human error being somebody
made a mistake. Instead, we're talking about the need to be able to debug the person and
why they made the mistake. And this is something which very often times we don't even try to
do in our industry, but we should. This requires that we have a really good taxonomy of types
of mistakes. That we can say, okay, situation awareness laps because of sensory overload
from too many monitoring alerts going off. A very common one that happens in our industry.
It's also something that's caused airplane issues. So if we understand that, we know,
they lost their situation awareness. They couldn't understand where the problem was. This happened
because they had too many alerts they were trying to focus on. Now the question is, are
we actually throwing too many alerts? Do we need to think about maybe prioritizing things
differently? Do we need to rethink how we do alerting? And suddenly, we have a dimension
for looking at these problems that we currently don't have. Instead, currently, what happens
most places I've worked is, okay, something went wrong. We didn't spot it. Therefore,
let's add another alert over this. But when I was at the delivery here, we actually had
a major incident where somebody, again, missed a problem relating to a database, relating
to a Postgres instance, I believe, if I remember right. Despite the fact that it was well
alerted. I was talking to somebody afterwards and he says, do you know what the false positivity
rate of our alerts are? And I'm like, no, it's like 99.8%. How do you expect somebody
to spot the problem when almost all the time our alerts don't mean there's a real problem?
Okay. Now what he meant by false positivity isn't what I would mean by it. I mean, there
were problems that the alerts were alerting about, but they weren't like customer facing
problems, right? So the second thing is we need a really good understanding of our cognitive
biases and the functions that they provide to us and also the problems that they can
lead us into, right? So one of the good examples is, hey, look, you know, I know you're about
to do this. I'm not sure that's what the problem is. Can we think about this first, right?
And as soon as somebody says that, that means that they're saying, my mental model is not
the same as your mental model. One of us is wrong. We should probably figure that out
before we proceed. Figuring out how to do that's really, really important, especially
when we talk about social factors involved, right? It's one thing to do that with your
peer when you're on an incident call and there are two of you there. Something very different
to do when the person typing the words is very senior and you're very junior and there's
somebody C-level popping into the call to ask for an update. I've been there. I've done
that and yes, no, I have not raised the issue and I should have, right? You know, figuring
out how to make these sorts of interventions and how to understand the intervention and
how to respond to it, those are things that we actually need training on, right? We also
need training on the social factors. We need to understand how power distance affects these.
What happens when there's, you know, the C-level person in the call? How does that change your
social interactions? How does that change your interactions in terms of debugging, right?
Those are important things and that's one thing that we can get some really big improvements
on relating to this. Finally, it's really important for us to be able to get to the
point where we can contextualize the person. In other words, since we operate as humans
in a relatively heuristic manner, right? We need to understand what the situation the
human was in when the mistake happened and that's another thing that these sorts of trainings
can help with. So, I've talked a little bit about social factors here. Power distance
is what it sounds like, you know, how big the difference is between the most powerful
person in the interaction and the least powerful person in the interaction, where we want it
to be kind of, you know, not quite equal but much closer instead of like this, maybe more
like this and, you know, figuring out how to structure things so that power distance
doesn't cause a problem. That also means giving people good training on how to intervene when
they see somebody much more senior about to make a mistake, you know, and you want to
intervene in a way which is not threatening and in the event where there's somebody even
higher in the call isn't going to be perceived as humiliating, right? Having good training
on this and how to communicate in those cases is really, really important. And a lot of
this ends up playing out into trying to create a work relationship between the people on
the team which is very heavily mutually supportive and also kind of helps prevent or checks
and traps the kinds of mistakes that each of us can make. So let's talk a little bit
about the ideal role of humans in database operations. Now, we kind of need to understand
this well. Okay, ten? Okay. Who's checking? Five? Okay, perfect. We kind of need to understand
this. Humans need to be in control. We need to be the decision makers. We need to be the
people who can say, this is what I think is going on. Let's go ahead and try this process.
And halfway through that process goes, this is not going well. Let's back off, rethink,
and make another decision, right? Partly because we're also operating heuristically, we can
do things that computers can't, right? We need to maintain really good situation awareness.
This means we need to have transparency in our complex automation. We need the automation
to be built around helping us, not replacing us. And to do this, we need to be well rested.
We need to be a clear peak capability ideally when we're in the middle of an incident. Now,
we may not be able to completely manage that last part, but if we can take steps towards
it and we can try to improve, we can do better, right? So a lot of this training is, at least
what I've gotten out of it, is really important. What I think is really important about this,
I'll just talk quickly about how to go about doing it, is that if we can get the organizational
leverage behind the training, then we can actually turn the promise of the training
into the reality. Sometimes you can't just teach people something and then have the
punishment abandoned them. That doesn't work. So as an industry, we treat human error the
way pilot error was treated in the 1950s. We have a whole lot to learn from aviation.
Those lessons are already being played out in medicine and many other fields today. We
need to do what we can to learn from it also. And it's really important to recognize that
we can get really good improvements in reliability, efficiency, speed of development, all these
sorts of things if we can better work with the human side of things. And I'm not talking
about human managers rating performance. I'm talking about people on the team understanding
performance for themselves and others. I just want to say that the three pieces of this
are trying to get trainers in who have experience. Also, an organizational commitment to make
it happen and then internally building your own programs and your own recurring trainings
and your own training for new people. So that internally you have a big culture around it
and you have experts who can think about it when it comes to be a post-mortem. So that's
what I have. Any questions?
Thank you. That was an amazing talk. Do you have any recommendations for further reading
if you can't bring in experts?
So this is a field which an aviation has a massive textbook industry. I think probably
the best, sort of the most accessible book I would recommend starting with is the more
recent versions of David Beatty's human factors and aircraft accidents. I think the most recent
version is called the naked pilot of human factors and aircraft accidents. It's just
referring to exposing the inner workings of the human piece of the aircraft. But again,
if you're a nervous flyer, probably look for a crew resource management textbook instead
because it may be less nerve wracking, it may be less intimidating, but it will have
information there too.
Do you have any recommendations for testing or drilling your processes like those checklists?
Yes, I do. One thing that I think we really should figure out how to do is an industry
and I completely believe in this. Obviously, like the chaos monkey idea and Netflix could
be exploited to do this if you can also build war games around it. But the thing is it's
really important to have drills, which means oftentimes you've actually got to probably
simulate or create some sort of a potential incident that you have to come together and
resolve. Now, ideally, you need to figure out how to do this without threatening your
customer-oriented services. In some cases, maybe the cloud's a really good opportunity
for that. But having those sorts of drills, maybe once a quarter or twice a year or something,
can really give you an opportunity to spot problems, figure out improvements and actually
go figure out what to do about those.
Just kind of building on that last point is how do you justify the expense in time or
money? Given that if this is successful, then nothing goes wrong. So it can sometimes be
the outcome of success is that you're spending a lot of effort on apparently doing nothing.
I don't believe that, but that's a reasonable thing that gets asked. How do you go about
justifying the time or the money on this after it's successful?
What I've usually done in the past is I make my points about, yes, we're going to improve
our incident response. This will reduce our time recovery. It'll improve our reliability,
et cetera. Maybe it'll improve our throughput organizationally. But then usually, people
don't listen. And then usually, there are more incidents. And then you can come in and
say, you know, these are specific problems that we had here where this training would
help. And I usually find that after two or three of those, people start listening and
go, oh, really? Maybe there is something to this.
Thank you.