Okay, I...
Yeah, it switched off to like...
By itself. I didn't touch it.
So yeah, when you debug, for example, your code,
when you're trying to find out why you have a strange error
or something like that, you can use our long tracing.
And it's very powerful, as we said before.
And for example, you can use tools like DBG or Recon
that are using error tracing underneath.
And the first step is to choose which functions
you want to trace, actually, because you don't trace everything.
Although you can trace what you want,
you cannot trace everything at once.
So you choose like, I want this function to be traced
for this bunch of functions.
And then, when you call these functions,
you get your traces being printed out.
So you get the information of this function is called,
these are the arguments, return values, things like that.
You can get it to console, you can get it to a file,
and you can also send it to a network,
and that's what I have been doing for many, many years.
I was just setting up, for example, I said many years, yeah,
for 15 years, I think, with Erlang.
So I was just setting up a special node
that was like collecting traces for all the other nodes.
So you can also send them to the network.
And, well, afterwards, you either read the traces that you collected,
or you can also search them, grab them, parse them,
do some other operations on them if you want.
But these are just text logs, let's say, mostly.
And the problem is that very often you have to repeat the whole process.
That's because you've traced one function,
but you found out that maybe the problem is in another function,
maybe in a completely different module, and so on and so on.
So you do it, so you repeat, repeat,
and that might be kind of a problem.
So this doesn't scale well.
And what I mean by that is if you try to trace a lot of functions,
well, I found out that at least for me,
when I get like 100 to 1000 traces, it becomes difficult to read,
like for a human to read that amount of information.
Okay, but you can search, for example.
And this also has a limit.
So, of course, this is just a rough estimate, let's say,
but for me, usually, when I have like 10 to 100,000 traces,
then it becomes difficult, because even my system can slow down,
IO can slow down, and actually it's quite surprising,
but sending traces to a file or to a network,
it's actually quite expensive.
And it can slow down the system quite a lot.
And it's a heavy operation,
so sometimes I had traces accumulating for three minutes
after I finished traces or something like that,
and the messages were still in the queue, still being processed.
Yeah, so this doesn't scale that well.
Okay, so let's sum up.
Choosing the function to trace is kind of a guesswork.
Not always, of course, sometimes we know precisely,
but most often I don't know.
I know kind of what I'm looking for, but not exactly,
and that's the problem, but I need to somehow know the function exactly here,
to choose it to be traced.
So, possibly many iterations of the process.
This is, for me, this is like ad hoc logging.
This is very much like logging,
but I don't need to have a log statement in my code.
I just choose dynamically, right now, I want to do these functions to be logged.
And what if the trace behaviors have tests that fail every 20 runs, for example,
do I need to repeat this 20 times?
So what? That's the problem, right?
And answer to some of those issues is Erlang Doctor,
at least for me, and for the people who've used that.
So what's the difference?
So you set up the tracing for entire application, not always.
Sometimes it's not possible.
Sometimes you have to trace individual modules,
but usually you can start with just one entire application.
You capture traces, store them in the ETS table,
and you clear the traces afterwards.
And you can repeat this process instead of repeating everything,
because you've collected enough stuff to query and to find out about different functions, for example.
To query, oh, this was this function called, maybe another one, and so on and so on.
You can do this.
And of course, rarely you have to repeat this,
but for me it's only when I, for example, trace the wrong application,
because the problem is not in my code, it's in a library that I've used.
Then I need to trace another Erlang application, right?
But it doesn't occur that often.
This scales much better.
What are the limits?
So on my laptop, for example,
querying becomes slow at about 10 million traces collected,
which is quite a lot,
but it's like tracing a function, a system under heavy load, for example.
And of course it depends on the size of individual traces,
because you can have big arguments being passed or something like that.
Yeah.
System memory becomes the limit at 4 million at about 50 million traces,
but sometimes it's 10 million, sometimes it's 100 million, it depends.
But basically when you have a few million traces, it's probably too much.
So there is a limit, of course.
So to sum up, very few iterations of the whole process, usually one.
This is for me like ad hoc instrumentation instead of ad hoc logging,
because you're gathering structured information in the ETS table.
I will show you details in a moment.
And use cases, for me there are many use cases.
For example, debugging, system exploration.
I often use it to just learn about the system.
I just run the system, do it with the usual stuff when I'm tracing the whole application,
and I'm just querying what the system did, actually, with the traces.
And you can also do some lightweight profiling without the need to set up the profiler for a particular function.
Yeah.
So let's go to the Erlang doctor itself.
How to get it from GitHub for Erlang or for Elixir?
For Elixir it's called Xdoctor, which looks like a former doctor, but it's just a bit funny.
Yeah, so here is a package of Xdox for both of them.
And yeah.
So how to run it?
Three options.
The first one that I'm using sometimes when doing firefighting,
this is when you don't have it in your shell, but you want it right now, like in a system that's misbehaving or something.
In both tools there are snippets that just download it from the Internet and compile it and run it,
which works in this particular case.
It's probably the best option if you just want it right now.
And yeah, all you need is to have the access to the Internet, which is usually the case.
The second option, which I'm using always in development, is just that you set it up in your .erlang or .iex.exe file.
So that it's always available whenever you start any Erlang or Elixir shell, be it in your project or wherever.
And third option, packaging.
You can always include it in your application, in your software, if you think it's that useful.
Okay, so let's move on.
Let's start.
Examples are in Erlang, but they are also available for Elixir in the docs.
You can find them.
The first thing to do is to start.
It needs a GenServer, so it just starts at GenServer.
And a few other examples how you can start them.
You can choose a different ETS table.
You can just have multiple ones if you want, switch between them.
You can limit the size of a table.
Very useful, like in a production environment, if you need to do some tracing, you just set it to like 1,000 or something.
Just the table will never grow bigger, so you will never consume all memory.
And yeah, there is also a start link.
Okay, so let's set up tracing.
I'm just tracing an example module.
It's a test suite, but it contains functions that we can trace.
It's good.
So yeah, I'm just starting that tracer.
And I can also trace a specific function, like provide a module function argument, whole application, multiple applications.
And add a bit more.
You can trace messages.
You can trace specific processes and so on.
There are a few more options.
Capturing the traces.
Okay, so let's call a function from the trace module.
I'm calling just a sleepy factorial of 3.
It's a function that calculates a factorial and just sleeps for 1 millisecond between each step, right?
It will have some time difference.
That's it.
Yeah, very simple.
And yeah, I'm just...
Okay, now we can stop tracing.
It's a good habit because you don't accumulate traces when you don't want them anymore.
And now what can you do?
Because we've accumulated traces, what can you do with them?
So let's read the record definition.
By the way, I'm using records because they are very performant.
And even maps were giving me five times worse performance for some operations.
So yeah, I'm using records.
Yeah, so let's get all the traces.
So I got all the traces and I don't want to talk about everything.
Let's talk about the arguments.
So these are the arguments and these are the return values, okay?
For calls, for returns.
And I will just introduce the other fields as we go.
Arguments are in brackets.
So now trace selection.
You can do a select.
It's a fancy way of doing this ETS select with ETS from 2MS.
And let's get all function calls.
And for each argument, let's just get this argument.
So I'm getting a list of arguments.
And of course, this is a recursive way of calculating factorial.
So it's 3, 2, 1, 0.
And there is also select slash 2.
And this one takes like any term and looks for that term.
So here it found, for example, it has an argument.
Here it found it as a return value.
But there is more.
It can be hidden inside any lists, maps, tuples.
So it will look recursively inside your data structures to find out anything you're looking for.
So for example, you can look for an error message,
even if it's called unknown error, which occurred to me once.
And I just found this unknown error.
I just put unknown error here and I just found the function that that causes, right, instantly.
Okay, there is also filter.
It's similar to select.
But here you can pass any function.
It's a bit slower, but it has more features simply.
So you can, for example, assign a result to a search, to a variable,
and then you can search in that list again.
Oops, sorry.
Then you can search in that list again.
So you can narrow down your search.
You got like two traces.
Now you search in that two traces, but only for calls and you get only one.
So another way to query it.
And the tracebacks are very important for me because I want to, like, know the source where this originated,
this particular, for example, function call.
So here I'm just looking for any return value of one.
And the sleepy factor of one actually matches it and it returned one.
So this is returned, the traceback of this one.
The call itself is first.
It's right here.
Sleepy factor of one and the rest is just the traceback.
And the sleepy factor of zero returned also one, but it skipped because of some skipping logic.
Yeah, it's details, but it helps you, like, limit the output that you get.
Actually, you can disable that and you can get, like, all the traces with output all.
Then you get, then you have no skipping of traceback that include another tracebacks.
And you can limit the number of much traces.
You can reverse call order.
You can search in a different table, in a list, for example.
And you can get only the first traceback if you want, very useful, just shortcut, let's say.
And you can also get it for a particular record or just an index of a trace because there has just this auto-incremented indexes.
Yeah, and similar to a traceback, you have ranges and ranges look inside.
So traceback is like what's the source and ranges give you, like, all the traces starting with a function call until it's returned.
Everything in between, from one process.
And, yeah, so for example, here we are really looking for any traces that are just for function calls that have one as an argument
and you get a range of traces from the call until the return.
A range options, you can get, like, limit call depth is quite interesting and very useful
because by having one you just get a call and return, which is very useful.
And searching in a list of traces is also possible, getting only the first range if there are many, also possible.
And getting the range for a specific trace.
So quite a lot of options.
I've just, you know, added, I've been adding and adding over a few years of development of these two.
So they're all quite useful.
Utilities, two simple utilities they wanted to talk about.
One is to just look up a trace.
Nothing fancy here, ETS lookup, does it, right?
But then you can execute the trace, which is quite useful for me.
So if this was a function call, I can just execute it right now, again.
This is just, for example, let's say I fix the bug and then instead of writing some long code,
I can just execute a trace and see if the result is the same or different.
Or I can trace again, right?
I can start the traces and trace again.
Okay, now a bit of profiling.
So I find this lightweight profiling very useful because it doesn't put as much stress on the system
as F-Prof, for example, the Erlang profiler.
And it's like instantly available.
I don't have to prepare for it in any way.
So call start, it's statistics aggregated by this function.
So I'm aggregating everything under the total atom.
So I'm getting like four calls and this accumulated time and this own time.
These are equal because I'm just accumulating everything.
But if I aggregated by function argument, you can see that there was this one call with each of the arguments.
And this call took the longest time, but actually it's accumulated time
because its own time was the shortest section, right?
You can also do filtering here.
So you can say when N is smaller than 3 and we just skipped one of them.
So you can do that and you can sort them.
Yes, you can sort them and print them as a table.
We had some just nice utilities to have.
And the last feature I wanted to talk about is function call 3 statistics.
I called it like that because let's say we have a function that calculates the Fibonacci sequence
in a suboptimal way, you probably all know that it's suboptimal.
It branches a lot and let's clean up the traces.
Trace again, call fib of 4 which returns 3, which is the correct value and stop tracing.
So we now have different traces in the table and let's do it.
Let's just call this function with default arguments.
So it says that there is a call 3, I mean by that function calls, returns, everything inside repeated twice
because there is this number 2 and it took 10 microseconds that there is no sleep in this example.
So it took 10 microseconds in total.
And this is how the function call 3 looks like.
So you can see that, yeah, indeed, it repeated twice.
So this can help you find out redundant code.
Yeah.
Okay, so this function also has some options but I don't have time to talk about them.
You can just customize them a bit.
And table manipulation, you can get the current table, dump it to a file and load it on a different Erlang node.
And then you can continue analysis on a different Erlang node.
And that's all I wanted to talk about.
And that's me on a mountain bike. Thank you.