Okay, welcome back.
So while you all have been walking in, I've been quickly reading this book, Efficient
Go, it reads very quickly, and now Bartek has made sure that my code is ten times quicker,
so tell us everything about it.
Thank you.
Thank you very much, everybody.
So welcome.
I hope your travels went well.
Mine were, like, canceled, flight canceled, change of route, so I had lots of adventures,
but generally I'm super happy I made it, and we are at the FOSDEM.
So in this talk, I would like to invite you to learn more about efficiency of our Go programs,
and there are already two talks that I have been on who mentioned, you know, optimizations
in its name, and, like, generally how to make software more efficient.
I wonder where this, I don't know, it's not hype, but it's already three talks about
one topic, why it's so popular, is it because everybody's saving me money, that might be
a reason, but I'm super happy we are really uncovering this for Go, because Go alone might
be fast, but that doesn't mean that we cannot, you know, doesn't need to care about, you
know, making it better, and use these resources when we execute it, right?
So let's learn about that, and turns out that, you know, you can save literally millions
of dollars if you, you know, optimize some code, sometimes in production, long term,
so it really matters, right?
But before we start, short introduction, my name is Bartolome Vodka, I'm an engineer
at Google, normally I work at Google Cloud, Google managed Prometheus service, but generally
I'm open source, I love Go, I love distributed systems, observability topics, I maintain TANOS,
which is like open source scalable Prometheus system, I maintain Prometheus as well, and
generally, yeah, lots of things in open source, I mentor a lot, and I suggest you to check,
you know, also try to mentor others, it's super important to bring new generation of
people up to the speed in the open source, and yeah, I'm active in the CNCF.
And recently, as you see, I published a book, and I think, you know, it's kind of unique,
everybody's doing TikToks now, and, you know, YouTube, and I was like, yeah, let's be old
school, because, you know, you need to be unique sometimes in the world, and I really
enjoyed that, I learned a lot during that, and I would love you to learn as well, so
I'm kind of summarizing of some concepts from my book here in the stock, so let's go.
And I would like to start with this story, and, you know, apparently some of the talks,
one of the best talks, have to start with the story, but this is something that kind
of maybe triggered me to write the book, right, so imagine that, I mean, yeah, that was kind
of five years ago, we just started the project called Thanos Open Source, really it doesn't
matter what it does right now, but, you know, what happens is that it has microservices,
it has, you know, I think, six different microservices written in Golang, you put in communities
or any other cloud, and it's just a distributed database, and one part of this database is
compactor, it's like a component, again, doesn't matter much what it does, what it matters
is that it touches object storage, and it processes, you know, sometimes gigabytes or terabytes
daily of metrics, right, of some data, so what happened is that at the very beginning
of implementation, as you can imagine, you know, we implemented, yeah, MVP, it kind of
functionally worked, but of course, you know, the implementation was kind of naive, definitely
not optimized, we didn't even run any benchmark, right, other than just running on production
and just, yeah, kind of works, so, and you're laughing, but this is usually, you know, what
development in a higher velocity looks like, and it was working very well, until, of course,
more people put load into this, and, you know, we have some issues like Ooms, you know, one
user pointed us to some graphs of, you know, incredibly high spike of memory usage on the
heap, on the Golan heap, right, and you can see it's a drop, which means, you know, there
was a restart or someone killed this, and, yeah, and the numbers are not small, like
15 gates, I mean, for large data set, maybe it's fine, but it was kind of problematic,
right, so it was really interesting to see what different feedback and what different
suggestions community were giving us, and I mean, community, everybody, like users,
other developers, maybe product managers, we don't know sometimes who they role are,
but, you know, probably depending on their background, the answers, the proposals were
totally different, right, so I would like you to kind of, you know, check, and like,
check if you had the same situations in your experience, because, you know, this is kind
of like very ongoing problem, and I would like to, yeah, showcase this, so, you know,
first suggestion was that, can you give me a configuration that doesn't womb, and it's
like, what, do you expect me, like, very new project to have, like, flags, not a womb,
or like, useless memory, this is not as simple as that, yet many, many users are asking us
this question, or person's, or person's project, probably you heard this question, okay, what
configuration I should use, so it uses less memory, right, or like, it just, it's more
optimized, how can I optimize using configuration, it's just, you know, it's not as simple as
that, I guess, you know, maybe in Java, in JVM, you have lots of performance flags, you
sometimes tune things, and it's better, but, you know, it's not so simple, it's a goal,
like, kind of low level, you, I mean, yeah, it's, you need to do more than that, right,
another, you know, interesting approach, but very, very good in some way, is it just, okay,
I will just put this process into bigger machine, and it's that, and that's totally valid,
you know, solution, maybe short term, maybe sometimes it's enough, but, you know, in our
case, it was not sustainable, because of course, you couldn't grow vertically more and more,
and also, even if you would maybe find the big enough machine that was working for your
data set, then, you know, obviously, you were overpaying a lot, if the code is naive and
maybe wasting a lot of memory, right, then finally, you know, the most fun approach, okay,
let's split this one microservice into, you know, like a schedule there, and then, you
know, warcares, and then we'll just replicate in my super nice computer, you know, communities
cluster, and, you know, it will just horizontally scale, so I can use many, many hundreds of
small machines, so it will work, yes, but, you know, you are putting on small, kind of,
microservice so much complexity that it will be, like, more expensive, generally, right,
so the network costs, like, distributed systems, you know, injects, you know, things that you
have to replicate data, finally, so you overpay more and more and more, and you are, kind
of, distributing this non-optimized code to different places, that's not always the solution.
Sometimes the code cannot be optimized more, and we can, you know, we should probably horizontally
scale, but not in the very beginning of the project, right, yet, that was the first suggestion
from the community, right, of course, you can just switch from Thanos to something else,
right, that's also solution, and then, if you have this approach, and probably you would
just jump through project, this is not super efficient, but maybe, you know, some parts
of the project are better or some worse, that's an option, some suggestion, of course, paying
for vendor, right, like, they will solve the problems for me, for real money, so, but yeah,
that's not always a good solution, like, that's just giving up, and also, you know, migration
of data, huge cost of learning new tools, and so on, and, you know, all of this work
we're in the code, we have this, and it's like, you know, it's bumping, and super easy
ways that you could be avoided, right, and, yeah, so, you know, of course, that was Maloch,
so in C++, I mean, in Bugo, we don't have Maloch and so on, but, you know, memory overhead,
memory leaks like that, like, are very common in Golan, like, just imagine how many gorotins
sometimes you put, you created, you forgot to close some kind of abstraction, and the
gorotin is leaking, and so you are leaking memory like this Maloch, right, so, and, you
know, what actually, you know, was the solution, was some contributor finally came up, investigated,
what about this efficiency problem on the code level, algorithm and code level, right,
and we wrote, or like, we wrote small part of the, of the compactor to stream data, right,
so instead of building maybe the kind of resulted object that the compactor is doing in memory,
it was as soon as possible streaming that to file system, easy, generally easy, easy,
easy change, yet there was lots of discussions, lots of stress, lots of weird ideas, and I
would just find it like, over time, amusing that this, this story was repeating in many,
many cases, right, so, and you know, that's not only, you know, of course, experience,
so many, so many kind of nice examples where only small character change, two character
change there, and, you know, so much kind of like improvement over like large system,
so sometimes, sometimes there are like, very easy ways that we can just pick it up and
just do it, right, but we need to know how, right, so kind of two learnings from the story,
one is that software efficiency on code level and algorithms, so changing code, you know,
matters, and learning how to do it can be, can be useful, and second learning is that
there is common pitfall, I think, generally in the, in this years, because in the past
we have premature optimizations, everybody was playing with the code and trying to over-optimize
things, I think now we are lazy and we are more like into DevOps, into changing, you
know, configuration, into horizontal scaling because we have this power, we have cloud,
and this is usually, you know, more chosen solution than actually checking the code,
right, and I call it closed box thinking, and I think this is a threat a little bit
in our ecosystem, so we should acknowledge that there are different levels, we can sometimes
scale, we can sometimes put more bigger machine, we can sometimes throw right to rust, if that
makes sense, but you know, that's not the first solution that should come to your mind,
right?
Okay, before we go forward, I will, I have five books to share, and I will start the
link to quiz at the end, and it will be super simple, but pay attention, right, because
maybe there will be some questions around, and you can answer, send me an email, and
I will just, you know, just choose five people, lucky people, to have my book, so, yeah,
pay attention, all right, five steps, five steps, yeah, for efficiency, efficiency progress.
One thing I want to mention, I don't know if you have been in the previous talk, or like
before previous, he kind of explained a lot of optimization ideas, like I think, and I
might say before, like he mentioned, string optimizations with internings, has just mentioned,
I think, something around, you know, allocations, and many kind of like, I think, padding,
strike padding, and generally, you know, all those kind of ideas, this is fine, but it's
optimizing stuff, it's not like looking through dictionary of things I did in the past, it's
kind of more fuzzy, more involved, so what I would like you to focus, it's not all particular
way of how we optimize an example I would show, because it's super simple and trivial,
but how we get there, right, how we found what to optimize, how we found if we should
even optimize, okay, so focus on that.
So first step, first suggestion I would have, and this is from Book, I kind of found, yeah,
I don't know, like I defined this name TFBO, which is essentially a flow for development,
efficiency aware development that worked for me, and generally I see other professionals
doing that a lot as well, so test, fix, benchmark, optimize, so essentially what it is, it's
like a TDD with something else, and TDD you are probably familiar with, test-driven development,
you test first, as you can see, and only then you kind of like implement or fix it until
the test is passing, right, I would like to kind of do the same for optimizations as
well, so we have benchmark-driven optimizations, because as you can see, we benchmark first,
then we optimize, and then we profile, right, and I will tell you later why, but all of
this is a closed loop, right, so after optimizations we have to test as well, okay, so it feels
complex, but we'll make one loop, actually maybe two, during the stock on a simple code,
so let's do it.
So let's introduce a simple function, super simple, super stupid, we are creating millions
of elements, I mean, a slice with millions of elements, and each of those elements are
just a string, a constant string for them, super simple, it's the first, you know, kind
of first iteration of this program we want to write, so what we do regarding TFBO, okay,
so we test, right, I mean, now we have a code, for example, and we want to maybe improve
it, we test, test-driven development, so let's assume I already had the test, right, but
the test could look like this, and then, you know, I'm ensuring, okay, it's passing,
so nothing functionally I have to fix, so what next?
So next is this measurement, it's a benchmark, and again, has this already mentioned how
to make benchmarks, but I have some additions, extensions to that that you might find helpful,
something I want to mention is that, you know, we were talking about micro benchmarks, because
the same level of testing behavior, like for example, like for this small function, like
we have this create, you know, unit test is totally enough, right, this is on micro level,
we are making just unit test, it's fine, but sometimes if you have a bigger system, you
need to do something on macro level, like integration test, end-to-end test, whatever
bigger, right, and the same happens in a benchmark, right, this is micro benchmark, this is kind
of unit benchmark, there are also micro benchmarks I covered in my book, and then you need to
have more sophisticated kind of setup with low testing, with maybe some automation, with
some observability, like, you know, Prometheus, maybe, which measures over time some resources,
but here we can, we have a simple unit create function, we can just make it simple with
micro benchmarks, and, you know, it has already mentioned, but, you know, there is a special
signature in a test file you have to put, and then there are optional helpers, for example,
that I like actually to put almost everywhere, report allocs, which is by default making
sure that this function will measure allocations as well, and the reset timer, which is super
cool because it resets the measurement, so anything before you allocate, you spend time
on, it will be discarded from benchmark result, so benchmark will only focus on what will
happen within this loop iteration, right?
And then this for loop, you cannot change it, don't try to change it, always copy, this
is a boilerplate that has to be there, right?
Because it allows Go to make repeatable, check the repeatability of your test by running
it, you know, hundreds of times.
Okay, so how we execute it, already, again, has this mentioned, but this is, you know,
how I do it to, like, focus to one test, but this is not enough, in my opinion, right?
By default, it runs only one test, one second.
I recommend to actually make sure you explicitly state some parameters, right?
And I have one liner, one liner in bash, for example, that I often use, so what it is essentially,
I'm kind of creating some variables so I can reference this result later on in a short-term
future, V1, for example, so this will create a V1.txt file in my locale, it will run this
benchmark, it will actually run it, you know, sometime, I specify, again, which is super
amazing because it was like, okay, so I have this V1 file, what I was doing with it, and
then you check in your bash history, okay, oh, that was one second, and then that was
something else, right, so it's kind of useful.
And then this is crucial, this is something I don't know why I didn't learn in the beginning,
maybe you learned the count, dash count, right?
So what it is, is that it runs the same test couple of times, six times, actually, and
so one second, six times, and this is super important because then you can use further
tools you will see to check, you know, how reliable are your results, it will essentially
calculate the variance between the, you know, the timings, for example, so if the variance
is too big, then your environment is not stable, right?
And then I pin to one CPU, this is super important to, generally pinning, not to one, right?
Just pick something that works for you, for concurrency, pick something that runs on production
maybe, or similar, but always between tests, don't change that, right?
So that's super important, and also I recommend to change less than numbers of CPU because
your operating system has to run on something, right?
So those things matter, also don't run this on laptop without power connected because
you will be CPU trolled off.
There are lots of kind of small things that you think, oh, it doesn't matter, no, it matters
because then you cannot rely on your results, right?
So try to make this serious a little bit and at least, you know, don't put, don't benchmark
on your lap, you know, in the bed, you know, because they will be overheating.
So yeah, small things, but it matters, right?
I was doing that all the time, by the way, yeah.
So results, you know, result looks like this.
You can see many of them.
But this is not how I use it or how we supposed to use it, apparently.
There is amazing tool called BenchStat, and it just brings in more human-readable way,
and you can see it also aggregates and have some averages over those runs and tells you
within this percentage.
For example, the time, latency, there is a variance of 1%, which is tolerable, for example,
right?
And you can kind of like customize what exactly, how it calculates this variance and so on.
So we can trust it, like it's within 1% of, I guess, free, you could trust it, depend
on what you do, but generally it's not too bad.
Allocations, fortunately, are super stable, right?
So yeah, so we benchmark, we measure it, okay, we know our function has these numbers, like,
I mean, what's next, right?
Everybody was like, yeah, let's make it faster, let's make it faster, but wait, wait, wait,
why, why should we make it faster, maybe, okay, maybe that's a lot, 100 megabytes of
every, you know, create invocation, but maybe that's fine, right?
So this is where I think we are missing a lot of experience, usually.
I mean, you have to set some expectations, right, like, to what point you are optimizing,
and usually we don't have any expectations, like, okay, yeah, I mean, even from product
management here we have maybe functional requirements, but never really concrete performance requirements.
So we don't know what to do, and honestly, if you don't, you just ignore those requirements,
okay, I don't have, I just want to make it faster, then this premature optimization is
always, right, because it's always premature, because it's a random, a random goal you don't
really understand, right, so maybe, maybe just make it fast, right, that's also like
very fuzzy, obviously, and that's not very helpful.
So what is helpful?
What I will, and I know it's super hard, I know it's kind of uncomfortable, but I suggest
doing some kind of efficiency requirements, spec, super simple, as simple as possible,
I call it rare, so there are efficiency requirements, and what it means is essentially try to find
out some kind of function, right, some kind of, you know, complexity, but not as if it's
very complex, it's just more concrete estimation of the complexity based on inputs, right,
and for simple functions, like for example, our function, we can estimate, you know, what
in our minds we think should happen, roughly, right, so, you know, for runtime, we know
we, one million time we do something, we don't know how many now seconds, let's pick 30,
this is actually pretty big for one iteration of just append, but just really pick some
number, sometimes it's good, it's just, you know, you can iterate over this number, but
if you don't know where you go, then, you know, how you can make any decisions, decisions.
In allocations, it's a little bit bitter, a little bit easier, because we expect a slice
of six, of one million elements of strings, and as we learn from MachiTalk, every string
has these two parts, one part has 16 bits, which has length and capacity, or maybe capacity
not, but then, yeah, length capacity and pointer, and then there's other parts, which
lies in the heap, but for this, you know, 16 bytes, we can assume that we'll be 16,
right, so it's every element is 16 bytes, so now we just multiply, that's our function,
that's what we all expect, right, and with this, we can, you know, kind of expect that
every invocation of create should, you know, kind of allocate 15 megabytes, but what we
see, we allocate 80 megabytes, right, so already we see that, oh, there might be like easy
ways to do, or something I don't understand about this program, and this is what leads
us to better, to spotting maybe easy wins, and spotting, you know, if we need to do anything,
right, in terms of time, latency, it's already kind of like, more than we kind of expected,
right, but this is more of a guessing, like I just guessed this 30 seconds, right, okay,
so what we do, now we know we are, you know, not fast enough, not allocating, we are over
allocating, right, so then we profile, then we check, okay, we have a problem, now let's
find what's going on, and this is where, on micro level, we can, you know, use profiling
very easily by just adding those two flags, it will gather memory profiles and CPU profiles
in the file, like v1.mempprof, on macro level, you can, there are other ways of gathering
profiles, but you can use the same format, the same tools, there are even continuous
profiling tools in open source, like parkadev, I really recommend them, and it's super easy
then to gather those profiles over time, so this, what we want to really learn is that
what causes this problem, and this is like a CPU profile, and we could spot, and the
wider means it spends more CPU cycles, the depth doesn't matter, this is just how many
functions we have, right, so we can see that create, of course, is one of the biggest contributors,
but the growth slice, right, like why we spend so many cycles growing slice, ideally, I know
how many elements I have, kind of why it doesn't grow me once, right, and then we can check,
and by the way, you can use this go tool, pprof.gttp, locally, I kind of use it a lot
on this file to kind of expose this kind of interactive UI, you can do the same for memory,
but honestly, this is not useful because Append is a standard library function, and they are
not very well exposed, right, so they're hidden, so this is not very helpful, actually CPU
profile was more helpful, because it pointed us to the growth slice, and if we just Google
for that, you will notice this comes from Append, and then you can go to documentation
of Append and learn what it actually does, and as you probably are familiar, because
this is like, should be a trivial case, Append resizes the slice, or assizes the underlying
array, whenever it's full, right, and resizing, it's not super simple, it has to kind of create
a new, bigger array, and copy things over, and garbage collection will kill the old one,
but not fast enough because of the garbage collection, so we kind of aggregate that as
another allocation, right, so this is what happens, and kind of the fix is to just preallocate
right, so to tell, you know, when you create the slice, okay, how much capacity you want
to prepare for that, and thanks for that, so what we do now, okay, we did optimize in
our TFBO, now we test, before we're even measuring, because if you are not testing
if this, you know, this code is correct, then, you know, you might be, you know, yeah, we
would be happy that things are faster, but functionally broken, so always test, don't
be, you know, lazy, run those unit tests, easy, and then, you know, once they are passing,
you can comfortably measure, again, I just changed V2, just to specify another variable,
right, on our file system, and then I can do a bunch that V1.txt and then V2.txt, actually,
I can put like 100 of those variables, it will compare all of them, but here we compare
two, and not only we have absolute values of those measurements, but also a diff, right,
so you can see we improved a lot, and if we check absolute value in regards to our efficiency
requirements, you see that we met our threshold roughly, but like we estimated it, so it's
totally good, you know, 15 megabytes, we have 15 megabytes, and then it's faster than our
goal, so now we are good to go and release it, right, so that's kind of the whole loop,
and you kind of do it until you're happy with your results, so yeah, this is it, and learnings,
again, five learnings, follow TFBO, test, fix, benchmark, optimize, use benchmarks, they
are built into GoLang, they are super amazing, GoTest slash bench, set the clear goals, goals
are super important here, right, and then profile, and you can, I mean, GoLang uses
Pprof, which you can Google as well, it's like amazing kind of protocol, kind of set
of tools, integrated with other, you know, clouds and so on, and use it, you know, every
day whenever I have to optimize something, and then finally, the key is to try to understand
what happens, what I expected, and you know, what's wrong, reading documentation, reading
code, this is what you have to do sometimes, and a general tip, whenever you want to optimize
something super, super carefully in some, you know, bottleneck part of your code, I
mean, avoid standard library functions, because they are really built into generic functionality,
it will test, I mean, it will do a lot of things with, you know, different edge cases
that you might not have, so a lot of times, I just implemented my own parsing integer
function, it was much faster, so this is a general tip that always works, but again,
do it only when you need it, because you might have a box in this code, right?
So that's it, thank you, you have a link here, bwplotka.dev.
Thank you.