So, before we go on with the next security or later topic, we're going to talk about
something completely different, and that is about elastic search internals by Martin
from the Bulgaria jug, and maybe also about security in this context.
So, we're going to talk about something completely different, and maybe also about security in
this context.
Test, test, test.
Thank you.
So, people coming in, please move to the middle of your row so that there's space on the side
so people can sit.
We're working in Cisco together, so a lot of people coming, so we can start.
If you're standing along the side, please take a seat.
So, hello, everyone.
My name is Martin, and I'm a consulting architect at the European Patient Office.
I've been also doing a lot of consultancy on elastic search in the past two to three years.
So, just before we start with this session, how many of you are using or have used elastic
search in a project?
Okay, more than half of the people.
So, why this talk at FOSDEM?
So, multiple reasons, in fact.
When I've worked with elastic search, I realized that even though it has quite a good documentation,
in many cases, you need to go into the public code base and see what's in there, and to
understand how it works.
I've had questions from many people, how this functionality works, or how can I achieve
something with elastic search.
And not always it's clear from documentation or blocks over the Internet what you can achieve
with elastic search.
So, in this short session, I'll try to show you how this elastic search works internally,
and I'll talk about the elastic search architecture.
So, first of all, we'll do a 360-degree overview of the elastic search stack, which I believe
most of you are familiar with.
Then I'll go into the elastic search architecture, and at the end of this short session, I'll
show you how you can write a very simple elastic search plugin.
In most cases, you won't need to write an elastic search plugin because there is quite
a rich ecosystem of elastic search plugins that you can use.
But many companies find that that's not always the case.
So, sometimes you need to either customize something in elastic search or write your
own plugin to achieve something.
All right.
So, let's talk briefly about the elastic search stack.
In the middle, we have elastic search, which is a Java application.
It's being updated quite oftenly.
There are a lot of features being implemented in elastic search, especially in the latest
few releases.
And around the elastic search server application, there are different applications that are
being built to allow you to work more easily with elastic search, such as Kibana.
Kibana is a user-rich user interface for elastic search that allows you to achieve multiple
things, so not only querying elastic search, but Kibana allows you to also visualize data
that's already in elastic search or build different dashboards that are quite nice,
especially for management.
Also, if you want to put different data from a variety of sources in elastic search, you
can use LogStash.
So, originally, LogStash was implemented to provide a way to aggregate logs into elastic
search.
But over time, LogStash evolves to an application that is used to integrate data in elastic
search, not only log data, but any kind of data.
So, you can think of LogStash as a log aggregation pipeline that allows you to put data in elastic
search.
And on top of that, we also have a different set of so-called bits applications that are
lightweight log shippers that allow you to collect data and put it either directly into
an elastic search or through LogStash into elastic search or different other data sources.
The specific thing about the bit applications is that they are lightweight in nature, so
they are supposed to not consume a lot of resources such as CPU and memory.
And in that reason, they allow you to collect log data or other data and put it into elastic
search.
Now, you can think of elastic search as a web server built on top of the Apache Lucene
library.
So, the Apache Lucene library is an actively developed Java library that is used by different
applications that want to implement some kind of search functionality.
And elastic search is one of them.
So, I'll show briefly in a few slides how elastic search interacts with the Apache Lucene
library.
And another way to describe elastic search is a document-oriented database.
So, elastic search is used by different projects not only for searching, but also as a NoSQL
database.
So, I had a few projects where elastic search was used purely as a NoSQL database, not as
a search engine.
And one can think, okay, elastic search is a Java application.
Why I cannot use Apache Lucene directly?
And the reason is that elastic search provides a number of features that are missing in the
Apache Lucene library that allow you to implement search in your project way more easily than
using directly Apache Lucene.
Some of these features are, for example, JSON-based REST API, which is quite easy to use, quite
easy to write search queries, to index data into elastic search, and so on.
There is also a really nice clustering mechanism implemented in elastic search that allows
you to bring and scale your elastic search cluster quite easily, something that's not
possible if you use directly Apache Lucene in your project directly.
And also, it has a number of other features, such as, for example, caching, that allow
you to improve the performance of your search queries, and so on.
Now, the basic data structure used by elastic search is the so-called inverted index, and
indexes are stored on disk in separate files or Lucene segments.
Search can be performed on multiple indexes at a time.
That's one of the capabilities of elastic search.
And in earlier versions of elastic search, documents were logically grouped by types.
That was effectively deprecated as a version 7 of elastic search, and it's expected to be
dropped.
In order to ensure score relevancy when you search for some data in elastic search,
elastic search uses a set of different algorithms to score results relevance.
In the later versions of elastic search, this algorithm is BM25.
In earlier versions of elastic search, this was a simpler algorithm which is called TFIDF.
And the base of those algorithms is the fact how many times does a term occur in a document,
and how many times does this term occur across all documents that are currently indexed in
elastic search.
Based on that, by default, elastic search scores every result that gets returned by your search
query, and by default, it returns results sorted by relevant score.
Now, why would you use elastic search in favor, for example, of a relational database?
Well, it provides faster retrieval for documents in way more scenarios than a traditional
relational database can do.
So, as you know, traditional relational databases provide faster searches through indexes.
However, indexes in relational databases have many limitations based on the type of
SQL queries that you write.
In elastic search, the inverted index data structure provides with the capability to cover
way more scenarios for searching using more complex queries.
And for that reason, many projects choose to use elastic search as a search engine.
Now, documents also in elastic search might not have an explicit schema, as you have in
a relational database, and that's typical for many no-SQL databases.
An explicit schema, however, can be defined on the fields, and certain fields can even
have different types mapped to them.
This is needed because sometimes you need to use different kinds of search queries based
on the field type, and some field types pose limitations.
So, that's why you might need to have multiple types on a single field in elastic search.
Now, this was brief about what is elastic search and how it works.
Now, let's see what the architecture of elastic search.
Elastic search, as I mentioned to you, is designed with clustering in mind.
By default, in later versions of elastic search, if you start, if you create an index, it has
one primary chart and one replica chart.
So, what is a chart?
Now, an elastic search index contains one or more primary charts that distribute the data in the
elastic search cluster.
Below that, an elastic search chart is, in fact, a Lucene index, and a Lucene index is,
in fact, the data structure that stores the data on disk in terms of Lucene segments.
Lucene segments are the physical files that store data on the disk.
Now, when you index data in elastic search, you might have also replica charts.
Replica charts provide you with the possibility to enable high availability and data replication
at the level of the elastic search cluster.
So, two types of charts, primary and replica charts.
The more notes you add to the elastic search cluster, the more data gets distributed among
charts.
Now, it's very important that up front you plan the number of primary charts based on
the data growth that you have.
It's very difficult to change later in your project lifecycle the number of primary charts
you would need to re-index data.
However, if you want to change the number of replica charts, that's more easy to do later
in time.
So, it's very important that you plan up front what's the number of primary charts on an
index that you create.
Now, by default, elastic search tries to balance the number of charts across the notes that
you have.
And one of the other capabilities that elastic search provides you is that if a note fails,
you still can get search results, or so-called partial results can be returned, even if some
of the notes in the cluster are not available.
Now, by default, elastic search determines the chart where a document is indexed based
on a relatively simple formula.
You get the hash key of the routing key of the document.
This is the document ID, which can be generated in different ways.
You can generate it from elastic search.
If you don't specify your application, you can supply the document ID, and so on and so
forth.
And you'll take the modules, the number of primary charts that you have defined on the
index, where you index the document.
Now, as I mentioned, by default, the routing key is the document ID, but you can also
use a different routing key.
And one interesting technique that some people use to enable distribution of data in the
elastic search cluster is by specifying a custom routing key that allows you to enable
so-called chart routing.
This is a technique that allows you to specify at which particular chart you want to send
the document to be indexed.
But that's a case that's used in some specific scenarios.
In most cases, people rely on the default mechanism that elastic search uses to distribute
data in the cluster.
Now, by default, new nodes are discovered via multicast.
If a cluster is discovered, a new node joins the cluster if it has the same cluster name.
If a node on the same instance already runs on a specified port, and if you try to run
another node on that instance, elastic search automatically gives you the next available
port.
Now, however, in some cases, in some companies, multicast addresses are disabled for security
reasons.
And that's why the preferred mechanism to join new nodes in an elastic search cluster is
by using unicast addresses.
In the elastic search YAML configuration, you just need to specify one or more existing
nodes from the elastic search cluster so that they can join that existing cluster.
And in that list of unicast nodes, you don't need to specify all the nodes in the elastic
search cluster.
You just need to specify at least one node that has already joined the cluster.
Now, when you bring up an elastic search cluster, there are some considerations that you need
to take.
First of all, as I mentioned, sharding, it's very important for you to consider what should
be the number of primary shards that you define on the elastic search index, and the number
of replica shards, which is more easy to change over time.
You also need to consider how much data you store in an elastic search index.
Indexes with too small amount of data are not good, because that implies a lot of management
overhead.
And the same is for indexes with too many amounts of data.
I've seen some cases where people store, let's say, more than two, three hundred gigabytes
of data in an elastic search index.
And that really slows down search operations and other operations of that index.
And people start wondering, okay, why is my indexing slow?
Why are my search queries slow?
And in many cases, the reason is that because data is not distributed properly in the elastic
search index.
The preferred amount of data that you should keep in an elastic search shard is between
five and ten gigabytes, roughly speaking.
So if you have more data that you want to put on a shard, you should consider splitting
that data.
So you either use more shards in the cluster, or you split the data into so-called sequential
indexes.
So for example, you might have daily, weekly, or monthly indexes.
Now, this is what I mentioned.
So you should avoid putting too less data in the elastic search cluster.
Also, if you have too many shards defined on an index, that also introduces performance
and management overhead.
So you should consider rather splitting the data in the index rather than bringing too
many shards on a single index.
And determining the number of shards should be a matter of upfront planning.
Now, apart from putting the fact that you need to avoid putting large amounts of data
in a single index, the main strategy that people use is to use, for example, prefix
when they split data into indexes.
For example, you can put prefix for daily, weekly, or yearly indexes.
And if you do that, it's a good practice that also you use aliases to reference data,
directly reference a particular index in your application, but rather use aliases.
In terms of concurrency control, elastic search does not provide pessimistic locking,
like, for example, you have in relational databases.
If you want to establish some form of concurrency control in elastic search in order to make
sure that you don't have unexpected race conditions, so elastic search uses optimistic
locking for concurrency control.
The way this works is when you index a document, there is a version attribute that can be
specified.
And if there is already a document indexed with that version, then the operation is
rejected from elastic search.
Concurrency control can also be achieved with the two fields that can be specified
when you index the document.
If sequence number and if primary term parameters, if they already match the document
that's indexed, then this operation gets rejected.
So if you want to establish some form of concurrency control in elastic search, you can
use this optimistic locking provided by elastic search.
In terms of high availability, you can create one or more copies, or so-called replicas
of an existing index.
The number of primary charts is specified when you define the index mapping, or you
can change it later.
Once an index request is sent to a particular chart, determined based on the hash of the
document ID.
The document is also sent to the chart replicas.
And one interesting property in elastic search is that the replicas are not used only for
high availability, but also used for searching purposes to improve performance.
So when you have replica charts, they also participate in the search requests that you
have for elastic search.
Now, this mechanism for improving performance is really nice, but this doesn't mean that
you need to supply to increase the number of replicas because, of course, that increases
management overhead.
So it's also a matter of determining how many replicas up front you would need.
And later on, if you plan to scale your cluster, you can also increase the amount of replicas.
So you should not put a lot of replica charts also at the beginning when you define your
indexes.
Now, how is a chart request processed?
Now, if we want to index a document in elastic search, what happens?
We send the request to a coordinating node.
This is one of the nodes in the elastic search cluster.
And this coordinating node sends the request to the chart, to the node in the cluster where
the document needs to be indexed and stored in Lucene segments.
When the document reaches the elastic search node in the cluster, the particular chart,
it gets sent not directly to the disk, but to two in-memory areas.
This is the memory buffer and the transaction lock.
Now, the memory buffer gets flushed every second to the disk.
So when you index a document in elastic search, you cannot expect it to be available right
way for searching purposes.
But there is also a parameter that you can use to enforce it to be written to disk right
away before waiting for this one second to be flushed on disk.
There is also another area, which is called the transaction lock, where it gets flushed
not so often.
It gets flushed every 30 minutes or when it gets full.
So the important takeover from this is that when you index a document, you should not
expect it to be available right way for searching, but you can enforce it too.
What happens if you send the search query to elastic search?
First, the search request gets sent to one of the nodes in the elastic search cluster,
the so-called coordinating node.
Then we have two phases.
First is the query phase.
It asks all the shots, primary and replica shots, hey, do you contain some data for that
search query?
And this information gets returned to the coordinating node.
Based on that information, the coordinating node determines which nodes it needs to query.
And on the second fetch phase, it sends the request to the shots that have some data for
that search query and return it back to the client.
Now, in terms of how is the elastic search called base structured, this is a snapshot
from the GitHub code base of elastic search from the public code base.
Now, what I'm speaking about in this presentation applies for the public code base in elastic
search because of version 7.16, there was a licensing change, and there is a lot of
controversy in the open source communities whether elastic search is still open source
or not.
So we can have a discussion about that after the session.
I'm not going to go into the details, but the main thing about this licensing change
is to protect elastic search from other vendors willing to provide elastic search as a service,
not from people willing to customize elastic search or to use it for their in-house projects
and so on.
So this is the structure of the elastic search code base that has been like this since the
Apache license code base.
So elastic search gets built with GitHub actions.
You can see also the definition in the.github folder.
The main server application is in the server folder.
The documentation that gets generated on the official elastic search website is in the docs
folder.
We have the main modules for the elastic search server application in the modules folder and
the internal plugins in the plugins folder.
An implementation of the REST based Java client for elastic search, the high level and the
low level REST funds are in the client folder, and the distribution folder, you can find
the gradle scripts that allow you to build different distributions of elastic search
for Linux, Windows, and so on.
Now, I would say the structure of the code repository is very logical.
It's easy to navigate.
So you can just go into GitHub, and if you need to see, for example, how is a particular
plugin or module implemented, you can just go to GitHub and check it out.
Now, internally elastic search is comprised of different modules.
And in earlier versions, elastic search used the modified version of Google GIS for module
binding, but they're slowly shifting away from Google GIS in favor of their own internal
module system.
So modules are loaded on startup when the elastic search server starts up.
And in this simple example, I've shown an example of how modules were bound internally
when the node starts up.
So we use a module binder.
The earlier versions, B was a Google GIS binder.
And then we bind particular module classes to their implementation.
And then wherever you need them, you can reference them in the elastic search code base.
It's a very simple dependency injection mechanism.
Now, when elastic search starts up, you can imagine it's a simple Java application.
The main class is Orc elastic search, bootstrap elastic search.
It boils down to calling the start method of the node class.
And the start method, in fact, loads up all the modules of the elastic search node.
Now, some of these core modules are, for example, modules that provide the REST API of elastic
search module that allows you to establish clustering and elastic search, or so-called
transport module.
There is a module that allows you to build plugins for elastic search, and so on and so forth.
Now, how does elastic search internally interact with loosing?
When you start up the node, the node also exposes, provides different services that are
used by the modules of elastic search.
And, for example, if you want to, when you start up a node, there is a createChart method
that gets called, indexServiceCreateChart, to create and initialize the chart that is
part of this elastic search node.
And then, if you want to index a new document, it boils down to calling indexChartApplyIndexOperation
on primary.
Then, this boils down to calling the index method on the indexChart class.
And the indexChart class goes down to an internal engine class that calls index into loosing.
Then, that calls internal engine at docs.
And at the end, we just call indexWriter, which is a class from the Apache Loosing Library,
at documents.
So, it boils down to calling different methods from the Loosing API.
And on top of that, we have a lot of initialization and services happening.
So, in a way, you can think that apart from all the functionality that elastic search
provides, the integration with the Apache Loosing Library just boils down to calling the
different APIs that Apache Loosing provides.
And last but not least, I'll show how you can build a very simple elastic search plugin.
Now, if you see the elastic search code base, it already has some building plugins that
you can use.
And there is a very nice elastic search plugin utility that you can use to manage plugins,
to install them, remove them, and so on and so forth.
If you build your own plugin, you can use the same utility to install the plugin, and
it gets placed in a folder in your node installation.
So, if you install a plugin, you need to make sure that it's installed on all the nodes in
your cluster.
Because many plugins are cluster aware, it needs to be installed on every node in the
cluster.
Elastic search plugins are bundled in ZIP archives, along with their dependencies, and
all of them must have a class that implements our elastic search plugin's plugin class.
There is a plugin service which is responsible to load the plugins in elastic search.
Now, let's see how we can create a very simple ingest plugin that allows you to filter words
from a field of an index document.
So, if you index a document, you can specify from which field, which words you want to
filter out.
This is a very common scenario, for example, if you want, for example, to implement that
allows you to clear contents from documents and so on and so forth.
It's probably one of the simplest plugins you might have.
So, first we have a filter ingest plugin class that extends the plugin class and implements
ingest plugin.
We have different interfaces for the different types of plugins you might have for elastic
search, and ingest plugin is one of these types of interfaces.
Then you specify you implement the get processors method because an ingest plugin needs to have
processors that you can define that do something on the documents before their index.
And the get processors method, what we do, we get a filter word from the parameters that
we supply on the ingest processor that we define in elastic search.
And then we get the filter field.
So, we have two parameters, the word that we want to filter out, and from which field
of the document we want to filter it out.
Then we create a map of processors, and in that map we put the filter word processor
that we create from this class and return it.
You can also have multiple processors defined in that plugin.
Now, what does the filter word processor look like?
The filter word processor extends abstract processor from elastic search.
It, again, comes from the core class of elastic search.
And we have an execute method.
In the execute method, we get the document that we want to index.
This is the ingest document.
We get the value from the particular field that we want to filter out, and then we replace
that value with the empty string.
And then we set back the value on top of that field and return the document.
This, when you index a document and you specify that ingest processor,
applies the filtering on that document before it gets indexed into elastic search.
Now, those two classes, if you want to build a plugin, you also need to supply some simple
plugin metadata, then build it, for example, with Maven or with Gradle,
and then you can install it with the elastic search plugin utility.
And in that manner, you can build any plugin you would like for elastic search.
And since we are running out of time, I'm not sure if we have some time for one
or two questions, maybe.
Do you have time for?
Yes, of course.
Okay, so if anybody?
Yeah?
Hey, thanks for your insights.
We saw how too many cats can go out and fall into the pool.
Yeah.
Yes?
I was curious, how does one know how many cars are going to be in charge?
Well, I would say it depends on upfront estimation of how much data do you expect to put in that
index.
So we need to do an upfront finding, okay, in the first phase of my project,
how many, let's say, gigabytes of data I would have.
And based on that, you determine how many initial set of shots do you put,
and if those shots still have a lot of data, then you consider partitioning the index.
And it's a matter of upfront planning to determine that.
Okay?
Yeah?
What is the structure used for store indexes and data?
It's inverted index.
This is the data structure.
Yeah?
Inverted index.
It's just called an inverted index because it's an app between terms,
and if for each term you have a pointer to the document that contains that term.
So it's called inverted index.