All right, hey everyone, my name is Owen, this is Kavi, we're going to be talking about
Loki today, this is a project that's very near and dear to my heart, been working on
for a while, but I believe it's actually the second FOSDEM top on the subject.
First one right here by Tom 2017 or 2018, so we'll cover some of the differences since
then and let's jump in.
So first of all we're going to get a few things from this talk, one how Loki works a little
bit differently than a lot of things that came before it, and what some of those trade-offs
look like and why I think that's an advantage, and then you can also learn some of the tips
and tricks that we've learned building cloud-native distributed systems that need to be up all
the time, and maybe you can incorporate some of that into your designs later.
So we are both engineers at Grafana Labs.
We work primarily on Loki, the open source project, but then also do a bunch to make
sure it runs as part of our SAS, I'm Owen and this is Kavi again.
And you can find the project here, this is our GitHub repo.
We are operators and implementers, so we both build and operate and run the software.
I use Loki every day to debug Loki, and that's kind of a labor of love, but it comes because
we get paged a lot, well not a lot, I actually shouldn't say that on it, you know.
But we do get paged and so we are very empathetic to that fact.
This is actually the last time I was here in Belgium, this is in Ghent on top of the
Gravenstein, this is not stage, I actually did get paged here.
And so this is actually using Loki to figure out what was wrong with our hosted version
of Loki itself.
But this is my first time here at Fozdom, and it's been a lot of fun so far, so thanks
for having me.
So this was actually coined by a friend and colleague Ed Welch, Loki is a time series
database, but for strings, this is effectively how Loki works at the end of the day.
And so the first off we'll jump into figuring out what exactly a time series database is.
So yeah, what exactly is time series database, right?
So if you think from the like a normal database, all you always see like a key and value, right?
And in time series database, surprise, surprise, you have timestamp.
So what you see is like for every unique identifier, you have array of records, or tuple, what
are you going to call.
So on each record, we'll have a value and a timestamp attached to it, right?
So in this example, as you see, so for this identifier, you have a value v0, a timestamp
t0, and value v1, a timestamp t1, so on and so forth, right?
So this is the mental model, we want you to like, yeah, I mean, keep in mind throughout
the talk, so that you understand some of the decisions, why we made the way we are.
So to see this in action, right?
So this is how it looks like in Prometheus, and this is how it looks like in Loki.
So what is identified here is a unique combination of this, what we call label set, right?
Here it's app equal to NGNX, cluster equal to US central zero.
So this is like a unique identifier, which has this list of records.
And as you can see, the only difference between Prometheus and Loki here is the type of the
value, right?
So in the record, you see timestamp, comma, value, and the value is floor 64 in Prometheus,
which is 34.5, you see here.
And in Loki, it's a string.
It's a log line you ingest into Loki.
So that's what, when we say Loki is a time series for strings, that's what we mean.
So yeah, it's a log line there.
So we definitely take or steal, or however you want to put it, a lot from Prometheus itself.
And this is actually what this looks like in terms of how we index and store data.
We're going to talk a lot about indexing in this talk, or particularly the lack of indexing.
So Loki just basically takes in index and metadata, but doesn't index all content at
all, which tends to be the vast majority of all this data.
So here we're looking at the set of Prometheus-style label values and the timestamp.
All that is kind of controlled and used for routing purposes of queries.
And then the log contents themselves, which, contrary to the size of the underlying graphs
on the screen, the lines, the log contents are much, much larger.
So that's the biggest piece here.
So this allows us to kind of figure out and use our expertise for the systems that we
run and use these labels, which tend to be topological, so the source of your contents.
The application equals API, the cluster, the environment, that sort of thing.
And then slice down to the time period that you care about, and we can use our own expertise
as operators to figure out where we should look.
And so as you can see here, this maybe is a pretty broad section corresponding to about
a terabyte of data over that period, but you can mix and match these things.
And so this goes to a fraction of that down to about 10 gigabytes by just looking at the
sets of applications that we actually care about rather than all of the applications
and replicas deployed in that cluster.
So to give you a bit of a taste of what we mean when we say Loki is performant and Loki
executes code faster, so this is what we mean.
This is the metric we took from one of our internal cluster.
So what you're basically seeing here is this particular Loki cell at peak, it's processing
like 50 terabytes per day, right?
And what you see in the UI, it's a Grafana UI, by the way, where you're running some
lock yield, which is a Kodi language we use to get visibility of your logs that you ingest
right into Loki.
We'll talk about lock yield a bit later.
So yeah, this is specifically a metric Kodi.
This is, like, basically you are trying to figure out the metrics on the fly just from
your logs without any instrumentation, right?
So this particular query is processed like 10 terabytes of data in 12 seconds, which
is almost like one terabytes per second throughput.
So that's what we mean when we say, yeah, Loki is faster performant.
Yeah, so my favorite piece here is that these are actually constructed from logs themselves
are not Prometheus metrics or anything, we log query metadata for every query that comes
into Loki and we're kind of extracting the part of the log line that talks about how
fast it processes data and then accumulating it over a bunch of subqueries to get this
final value and then graphing that over time.
So let's step back now and I like to think about how Loki was designed kind of in response
to what came before it, you know, if we look really far back, we remember using tail and
grep on like individual log files and that's still in use a ton today, but it doesn't work
so well when you start to need, you know, hundreds or thousands of machines and maybe
they're ephemeral, right?
So we had to kind of build new strategies for handling this and a bunch of things have
come up in the past, you know, 10, 15 years.
And sometimes it leads us into the next point where we've accumulated all of this complexity
and sometimes we've really missed that experience and I like this tweet because it's just incredibly
emblematic of that experience that sometimes I just really wish I did have grep on top
of, you know, all of the log aggregation and on top of the underlying complexity and scale
that we've accumulated over the past couple decades.
Yeah, so broadly speaking, like, so that's one of the goal of Loki, at least on the query
side, to have to take the same experience you have before, like with just grep and tail
that you're confident with, can we bring the same experience in this modern cloud native
distributor systems era, right, where you have your logs, like speeding out from different
machines from different applications, yeah, similar setup, right?
So like I mentioned before, Loki has this language called lockium, and that's how you
query your logs back to get some visibility, and this is heavily inspired from Prometheus.
So people who are familiar with Prometheus may already get, could I get a grasp here?
So this particular query what you see at the top is basically saying, like, give all the
logs from this particular namespace, let's say Loki day of 005, and then give me all
the logs that matches only the error in the log line, right?
So as you can see, the experience is like a pipe equal to error.
So you can still combine multiple pipes here.
So that's the kind of like experience we're talking about, right, so, yeah.
So doesn't mean you have to, you can only like grep for only specific pattern, you can
also grep for specific IDs, like use case can be like your order ID or trace ID you
want to find in the logs.
So you can also do some kind of regress match here, and also, like I said, you can also
like put multiple pipelines here, right, you can do under R, doesn't matter.
So it's basically like, first you choose which logs to look for, like a routing, what Owen
was saying, and then you can do all your piping, so, to mix and match.
So that's the idea we're talking about.
So this query is a bit different, as you'd have seen, like compared to previous two examples,
which we called as a lock query, and this is a metric query.
So in the lock query, the output you see after when your query is executed, you will see,
you're going to see like list of logs that matches the particular pattern, right?
So here, it's a metric.
So if you see what this query does, it's similar to the last one, but we added two different
things, right, here, the rate and the sum by.
So what this means is, without doing any instrumentation, like the logs are coming as it is, so you can
really find your error per second rate of all your application aggregated by the container,
which means, so, doesn't matter like how many applications running in this namespace, so
you can aggregate by container just from this metric query.
So yeah, that's the idea we are trying to, like, the experience we want to, like, with
a lock here.
Yeah.
This is probably my favorite part about Loki of all the things, right, the ability to extract
information at query time ultimately means that you can be reactive instead of proactive.
You don't have to, you know, figure out a schema and make sure that you're recording
something in a precise way before the bad thing happens, right, because a lot of the
time it's the unknown unknowns that get us.
And so this, the ability to extract this structure, you know, and particularly to do it from logs,
really allows you to figure out when things go wrong rather than having to re-instrument
before you can get that information.
So next up, we're going to talk a little bit less about the experience querying it and
more about how it's constructed.
So this is the kind of design choices that we make.
So particularly individually being able to scale different parts of the system, particularly
tuning your kind of preference for cost versus latency.
On the read path, Loki runs, is intended to run across commodity hardware, and our only
real dependency is object storage.
So we store everything cheaply in a generally managed solution, although you can, you know,
run your own object storage or use file system backends if you prefer.
So this is how Loki ingestion path architecture looks at the high level.
So when you send logs to Loki, and this is what it goes through.
So the key takeaway here is, like Owen said, the only external dependency you see here
is the object store.
So everything else is a component of Loki.
So of course we have like different deployment models.
So usually all these components can be put together in a single binary.
So if you're new to Loki, you are starting just to play with it at maybe like a small
scale.
So you just run it as a single binary, as a single process, and you send all the logs
to a single process.
And the only, you just put the bucket name and you're done.
So all your logs are getting ingested into Loki, just fine, right?
So that's what when we say like, yeah, like less dependency in running Loki.
Yeah, in this case, it's an ingestion path.
And so one of the things about not indexing the log contents themselves means that it's
really easy to ingest data from a bunch of different sources.
So maybe you work at an organization, and one team writes in one language, one team
writes in another language.
They have no standardization over the formats that they're using, the schemas of the logs,
if they're using structured logging.
So you can have one team in JSON, one team who just pulls in NGINX logs, and then another
team that uses something like log format.
And that's all OK, because Loki doesn't really care.
We just index the source of where this came from, along with the timing information.
So you can, each individual team, in that sense, can extract that information when they
query it and choose what and how that they actually do care about in their logs.
Yes, speaking of query path, right?
So this is high-level architecture on the query side.
So again, these are all like Loki components, except like a two dependency here.
One is object storage, of course.
The other one is like optional, which is like a cache.
So again, the same pattern applied here, right?
So you can combine all these Loki components into a single binary, and you can just run
it as a single process.
And all you need to do is just point to the persistent object store, and then cache for
the performance.
And that's it.
So you're good to go on the read path, right?
So again, if you want to run in a highly available fashion, let's say you hit some scale and
you want to tweak some things.
So this is how particularly we run in our internal like SAS.
So here you have more control in a way, say you can tweak individual component, you can
scale individual component, and yeah, that's the idea.
So again, the key thing here is the simple external dependencies.
So yeah, and it's really powerful to be able to develop on a single binary running all
these subcomponents or things that eventually run in microservices in a single process, and
then being able to scale that out depending on your tolerance for scale and running HA
or data replication, that sort of thing.
So because the index in Loki is very minimal, it's just this routing information or the
data topology, really.
It allows this to be much, much smaller than you would otherwise think.
So I actually pulled this from one of the clusters that we run internally.
It's close to 40 terabytes a day and just shy of 140 megabytes of index.
So it's much, much smaller.
And this gives us a lot of benefits when we talk about some of the next stuff.
Yeah, so if I were you, probably I'd be asking this question, okay, folks, you're talking
a lot about tiny indexes, right?
How on earth do you make the query faster?
So also this thought comes from the idea like either you are like from an academic background
or like a practitioner who is building a distributed systems, it's always been taught like if you
want to make something faster, you index it, right?
So here we are sharing our experience like where if you want to index everything to make
everything faster, so at some point in scale, your index is going to be much larger than
the actual data, right?
And in our experience, like handling huge index creates much more problems at scale.
So that's a whole idea here, right?
So let's understand like how low key makes the query faster with the tiny index, right?
And this is how.
So don't let the image scare you.
So let's understand piece by piece here.
So at the top, you see the query that comes in, right?
So for the sake of discussion, let's say this query is asking for the data for 1 hour period,
right?
Time period.
So the query path architecture what you saw before, what the first thing it does is it
takes this query, the huge query, and it try to make it like a sub-query by time split.
So in this case, it splits this 1 hour query into 4, 15 minutes query.
We call it a sub-query, right?
And the trick is it doesn't stop here.
So low key index is designed in such a way so that it can look into the index and say,
hey, this many data, like this many bytes, it needs to touch.
So we can dynamically decide how many worker pool you need to process that query.
So that's where this performance comes from.
So for example, like in this case, let's say this 15 minute sub-query is touching, I don't
know, like 10 gigabytes of data.
So and you can plan accordingly like how many worker pools I can schedule this query into,
right?
So let's think about this for a while, right?
So now the key takeaway here is you get to have a control over your cost versus performance.
So you start with low key and you hit some limit, right?
And at some scale, you're going to hit the limit.
And you can increase the performance just by adding more query pool, like more query
workers.
So yeah, that's like a key control we want to have with the low key.
So yeah, that's how we make query faster.
So the next thing we're going to talk about is retention.
We get asked this a lot, particularly wanting to store things like application logs for
some period of time.
You know, we use 30 days as kind of our standard, but it could be whatever your use case is
and then things like audit logs for much longer periods of time.
This is pretty easily tunable in low key.
But the important part here is we were talking about the index size earlier.
As we don't index the log contents themselves, retention is very, very easy and cost efficient
to do because all of our data is stored in object storage.
And so if we extract that kind of earlier index sizing slide out to what it would look
like in a year, this is roughly what we get.
And so again, we just use this index for routing information, which means that all of the data
is effectively live.
There's no like rehydrating or, you know, hot and cold storage tiers.
Everything is served out of object storage generally all the time.
Now you can, there's some nuance there.
You can put like caches in a couple of places and that sort of thing.
But the idea that you can use object storage as your primary back end is very powerful,
especially when you consider cost over long periods of time.
All right.
So yeah, we've been saying low key has been built for the operators and for dels, right?
And when it comes to operation, yeah, if you've been run any database or any distributed
systems at scale, so you always want to keep on like top of the latest release of whatever
the product you're running, right, to get the latest optimization, latest features,
so on and so forth.
And the other use case is like sometimes you want to migrate your data, right, from one
persistent store to another one.
In this case, maybe one GCS bucket to another one, even across the cloud provider, right?
You sometimes you find like maybe S3 is better.
Like I can go with S3.
I can change from GCS to S3.
So with all these use cases, can we do all these operations with zero downtime?
So can we do that?
So that's something like we can do in low key.
We have been doing it many times.
So to give you a complete example here, and this is one of my favorite part of low key
config, like we call it as a period config.
So what you're seeing here is we have something called like schema version.
So whenever we change anything with index or any new feature comes in, and if it's like
we use different index format or something, we change this version.
Like in this case, you see V11 to V12, right?
And so what you can do is, let's say you want to start using this new feature from
Jan 2023, right?
All you need to do is go and put the version V12 from the start date and you're done.
So low key can understand this and it can work with both different schema version.
So this is how you can like upgrade your schema and production like lively without downtime.
So this is one example.
The other one is like the migration, right?
Like I talked before.
So you may want to move your data within the cloud provider or across the cloud provider.
For example, it's the same thing here.
So from 2023 of Jan, I need to store all my new data into like S3 instead of GCS, right?
So you go back and you change this one config and you're done.
So low key again understands this.
So when the query comes in, it checks whether which data it's asking for, like which time
range it's asking for, and it can go and fix the data accordingly, right?
And yeah, again, without downtime.
So it also works really well with the retention.
Let's say if you have 30 day retention and yeah, after 30 days, you don't care.
So all your new data is stored to the new bucket.
So yeah, this is FOSDOM all about the community.
So we launched low key at 2019 as open source and we have like active community going on.
So these are some of the ways you can reach us.
We have a public Slack and we have a community forum.
And every month we also have a low key community call.
We alternate between US and EU time zone.
So yeah, come say hi.
Yeah, we are happy to, and if any of the things which we talked about excites you, come talk
to us.
We'll be like more than happy to have a new contributor to the project.
So yeah.
Yeah, we should have asked this probably in the beginning, but is there anyone out there
using Prometheus?
Yeah, a couple.
All right, a lot.
Yeah, good.
What about any low key users?
Okay, not bad.
Thanks.
That makes me really happy.
Anyone run into hard configuration problems with low key?
Yeah.
Oh, no hands.
That's weird.
No, I'm just kidding.
Yeah, we got some work to do on.
Yeah, so things to take away from the talk.
Key is meant to largely function like a distributed grep that you can also kind of pull metrics
and analytics out of.
It's low footprint, storing things in object storage and not relying on schemas.
And we really target kind of easy operations with that as well as low cost.
And then please come join the community.
Come talk to us.
We're going to be hanging out for probably at least 10, 15 minutes if you have any questions.
We'd love to hear from you.
All right, thank you.
Yeah, I can share.
Any questions?
Hey, great talk.
I was wondering with the amount of query sharding you're doing, how are you kind of synchronizing
the result in the end?
Or is it like synchronized in the UI at the end?
Or like, do you need to sort things because that might be very expensive?
All right, so I had a little difficulty hearing that, but I think the question was, how do
we synchronize sharding in the query engine?
Is that in the end when you show it in the UI?
When we show it in the UI, it happens a layer down in Loki itself.
So we've already merged everything.
I think the real answer to that is probably a lot longer than I can give in this question,
but talk to me.
It's one of my favorite things to talk about.
Thanks.
Great talk.
This was mentioned several times that the main power of Loki is that it indexes only
labels basically, and it doesn't index actual log messages.
But to make log messages searchable, you want to have many labels in this case, and many
labels often lead to cardinality explosions, how you deal with it.
Is there any good recommendation?
How many labels I should have?
What are trade-offs here?
Yeah, so the way that I think about this is you probably want to index where the log
came from less than what's in it, right?
So we definitely added the ability to kind of like, especially in PromTail configs, which
is the agent that we write, the ability to like add extra things in there, and people
have been really clever about what they put in.
Unfortunately, that can also be somewhat of a foot gun at times.
So I'd say index the things that correspond to your topology, right?
Where the logs come from.
So environment, application, cluster, that sort of thing.
And less things that have to do with the contents of them themselves.
There's probably a somewhat longer answer there, too, and then we've also been doing
some recent work to make some of this less of a user concern, particularly about distribution
of individual log stream throughput.
So if you have one application which is logging like 10 megabytes a second, that can be really
harmful on Loki in some ways.
And so people got clever around splitting that out into different streams.
But in the future, we're going to be doing a lot of that automatically and transparently
behind Loki's API.
Just to add one thing on top of what Owen said, like in log yield, which we didn't show here.
So if you have something in your log lying itself that you want to treat it as a label,
you can do it on the fly with the log yield query language itself.
So we have some like a parser.
If it's a log form, if it's a JSON parser, you can use it like labels, the things in
your log line itself.
Hi.
Thank you for the great talk.
I have many questions, but I'll ask one.
Do you have any tips in terms of scaling query and caching layers?
Like from my experience, usually they're very overutilized, but when people start querying,
you get all the crashes.
Do you have any code and ratios or anything?
Yeah.
So this is one of the things like how we expose configurations around query parallelism and
controls.
I wish I had a do over on this a few years back, because there's a couple different configurations.
Largely around for every individual tenant, how many subqueries you want to be allowed
to be processed per query at a time, and then there's also things like concurrency for each
of your query components, right?
How many go routines should you associate with or should you devote to running queries
independently?
But yeah, I got some work to do to make that easily digestible, if that's fair.
So this year at work, I end up kind of giving to Loki for 10 minutes about 20 gigabytes
of log from a 500 machine.
It was just a one-time experiment that they ran from time to time.
And we started using Loki.
We optimized it as intended.
So using index query was great.
At a certain point, my colleagues come to me and say, we want the log string for each
individual machine in a file.
And I started asking Loki for this, and he took ages to extract this information.
Now I hope you're going to tell me, this is not the use case for Loki.
You did very well to use RC's log and just pushing things in a file.
But if you have another answer, I'll be glad.
We didn't catch the question fully, but my understanding is you're asking, like, can
the log yield find logs coming from a single machine?
Is that right?
Am I getting it right?
So you're talking about, I guess, some kind of federation, maybe?
So why do you want to store it in the file in a single machine?
Because they wanted to run analysis on a specific stream of log from a specific machine.
So technically, you can store log files in the file system.
So instead of using object storage, technically, you can use a file system.
That's completely possible.
But we encourage to use object storage when it comes to scale, because that's how it works.
Yeah, is the question behind the question there changing where you actually store it
so that you can then run your own processing tools on top?
Yeah?
Okay.
Yeah, that's actually a relatively common ask.
It's not something that we support at the moment.
We kind of have our own format that we store in object storage or whatnot.
We do have some tooling, one's called chunk inspect, which allows you to kind of iterate
through all of the chunks, which could be from a particular stream or log file.
But it's not incredibly batteries included at the moment, if that makes sense.
Hello, I have the use case that I store some telemetry data with my logs sometimes, like
a metric or sometimes, which I don't want to be indexed, but I also don't want to encode
in the actual log message because it's already structured data.
Is it possible to have fields that are not indexed or data that are not indexed?
It's funny that you asked that.
Yeah, so there's kind of a growing question around non-index metadata like that, right?
That's not actually stored in the index itself, but like includes some degree of structure.
I know we see this as kind of a current need, particularly for like hotel formats, so it's
something that we're looking into right now, actually.
So thanks a lot, everyone.
Thank you.