Okay, well some of you may be here for my first presentation, this one is going to be different
in much more technology focused, if you will.
And we will talk about their distributed storage in the cloud, right?
And my goal of this presentation is kind of provide you a very general overview of options
which exist.
I am not an expert, right?
And probably something I'm going to say is I'm going to be wrong, right?
So if it is, then say, say like, this is fucking wrong, Peter, you know, so I can fix my slides
when I talk next time, I have the wrong stuff, right?
So don't be shy.
Be engaged and that's going to be more fun for all of us.
So the thing I would say to start with, we discussed about, as I believe there are different
ways you can approach your cloud, right?
One is where you really kind of lock in with the cloud provider and then another one is
what you really use, one of the really there open source solutions out there.
And as I spoke in my previous presentation, we can see what, well, like I would imagine
that is how the cloud was originally taken.
Well, I won't spend too much time on this because I already had a presentation, but
and also because we are not have too much time.
Now one thing what I often have people asking me is about their, the open source, right?
Which I think this conference is about.
And if you are thinking in the open source from, from the business standpoint, right?
We often see a lot of those different companies, right, which promote themselves as open source
or somewhere around open source.
But how do you know if it is, if it is for real, right?
And of course, one of those you can look at the, like open source, you know, license
and, and so on, right?
And this is all there.
Good stuff.
Another is also to make sure what you ask yourself, right?
So maybe even kind of company representative some of the questions, right, about how things
look, right?
One is you always think about how you can deploy that kind of solution product, right,
on your own without getting any additional cost, right?
Because software may be kind of open source, but they can, right?
And the source is available, but well, actually maybe their, you know, binaries are provided
only to four people who have a commercial subscription.
Well, in this case, it's maybe technically open source, but on the practical side, there
is some of those, some of those problems, right?
And especially I have seen some open source projects, which would essentially, you know,
withhold details about the build process, right?
So it's not easy, right, to, to do that, right?
Then another question I always like to look at is a choice of vendors.
If you need any help, right?
For many, for many companies, just saying, hey, we're just doing to go ourself is not
going to work, to work, I want to hire somebody.
And in some of them, you kind of license this around open source, there have been kind of
some restrictions.
Well, you know what?
You cannot, well, you know, provide the consultant services around the software, right, or something
like that as a license, right?
And I think the third very valuable thing about the open source is to see wherever you
can improve the software for your purpose.
If something in an open source doesn't really fit, right, can you contribute to that?
And again, that I think is another very interesting property of open source software where, where
maybe different shades, right?
Sometimes open source vendors maybe, well, maybe more or less open to that kind of things.
Well, now with that, maybe open source public service announcement, I would touch briefly
about open source.
I think I spoke about that, the previous speaker spoke about that as well, that is a fantastic
API, right, and that is something we are going to focus on here, right?
And why I mentioned Kubernetes here, right, as we're going to talk about the open source
storage in the cloud, I will focus a lot about, hey, what exactly choices you have in the
Kubernetes environment, right, because if you're really speaking about the cloud, right,
modern large-scale application, a lot of that is now being built around Kubernetes, okay.
Now this storage in the cloud, what does that really correspond to?
Well, there are a lot of different storage types we really can consider those days, right,
and here is the list which ranges from the simple stuff as a node local storage all the
way to the databases, right.
I define storage myself in a very general way in this case, hey, you need to store the
data somewhere, well, that is a storage, right.
Now there are some of those things like a node local storage is relatively simple, right,
they are, you know, direct replacement from like a, well, file systems we have on our
operating system for a long time, the others, such as databases, can be very complicated.
It is not just well, a database, right, but we can see databases being different by data
model query language, like a virus internal design decisions and so on and so forth, right.
Even if you look at the data model, right, these are some of the most common data models
which you would see and what is interesting, over the last, I think, like maybe 10 years
by now we see really this explosion of the different special purpose databases versus
approach before where we, I think, had like relational databases absolutely dominating
in the dominating ecosystem.
What is also interesting in this regard is what we are having the databases right now
not just being, you know, focused on a single data model, but many databases are able to
support multiple data models, right, which is, I think, is a big trend and even potentially
speak multiple, multiple protocols, right, like here are some examples, right, if you
look at the ClickHouse, which is data analytical database, it is able to talk ClickHouse but
as well as PostgreSQL and MySQL protocol, right, so their ideas, hey, you know, wherever
programming language and libraries you already use, you can just connect to us and, you know,
find your queries, you know, fantastic idea, right, or the time series database, Victoria
Matrix also is able, is implementing things like in FluxDB and Graphite API for data ingest,
again, I think, very, very smart.
We also see some frameworks which allow us to do some of conversion and translation,
for example, FerroDB, the projects allow you to use PostgreSQL back end with MongoDB
front end, right, or Amazon released recently Babylfish, which turns your PostgreSQL in
Microsoft SQL comparable database, right, so a lot of this, I think, you know, interesting
integration is going on those days.
If you look at their databases, we also see a lot of difference in the databases for a
purpose and design, right, like we are speaking about operational analytical, how is it used,
how it's internally structured and so on and so forth, right, and why am I listing that
is because if you look at their, at the complicated environments, right, with a lot of complicated
database, it's very unlikely you will be limited only to one database, right, of course, as
the previous speaker mentioned, well, you know, you probably don't want to have 50 because
that is way too, too much complexity, right, and you want to be very mindful about how
you introduce them to you, to your environment, but it's probably going to be more than one,
than one of those days.
Now we also speak about besides storage, about distributed storage, like why is that important?
Well, if you think about this, that is all of redundancy performance and scale, right,
I mean, saying if I just have a storage which is not distributed, right, which is the kind
of really sort of one device only, I will be limited and all of those.
I think this is even more important in the cloud, right, because if you look at the age
before the cloud, often we would be in a case where we have some one very powerful, very
redundant server, right, maybe with a hot swap rate and redundant power supplies, right,
and we expect that beast is never going to go down, right, well, that is not how we operate
in the cloud anymore, right, we assume any company in the cloud is going to die, right,
and they actually do die more frequently, right, if you look at the stats within like
a meantime within failures, let's say for VMs compared to what you could get with some,
you know, beast from the past, well, it's going to be different, right, but that means
we need things distributed at least from a higher ability standpoint.
Okay, the fact, let's look a little bit about the storage types as promised.
One is our commodity storage types, right, and this comments to the previous talk I did,
this commodity storage types, they are pretty much the same in every cloud, there are minor
differences but they are, I would say, like a commodity building blocks, right, they have
a relatively simple interface and usually it is relatively easy to migrate, so the lock
in, right, the word we don't like on this track, right, is going to be relatively low
with them, one is node local storage, I mentioned, hey, well, it's pretty much every major
and even your second tier cloud typically offer you some kind of local storage, right,
and it can vary in terms of, you know, performance it offers and so on and so forth, but that
is pretty much the same, right, from what that gives you, right, and that is fantastic,
right, but again, that is where I would, if you are looking from that, I would focus on
the performance because that is where surprises can await you, right, and saying, hey, this
cloud vendor and that both have a storage, right, one of them has its, you know, implemented
as a very fast NVMe flash storage, other something not so fast, well, that may have a very big
difference for your application.
The second most common one would be the network block storage, right, that's typically how
we store the data so that can, in the cloud, so it can survive their depth of instance
and Amazon would be EBS, right, and all the other cloud has something similar.
We also have some additional solutions in this case coming from the proprietary vendors,
right, like from those vendors, right, which, you know, provide you some additional features
and there is actually quite a lot of different solutions which exist if you want to roll
out their block storage in the open source, right, and I think this is kind of very cool
and that shows how things are evolving in the open source space, right, we had sort of
this block storage idea for a long time, so a lot of projects evolved, right, and they
have a lot of choices.
The next type of storage in the cloud would be your file storage, right, like when you
can say, hey, I can mount something locally, not as a block device, but as a file system,
in many cases that would be your NFS or SMB compatible file system, or both, right, again,
all the cloud will support something other file systems, there are a number of major
proprietary cloud vendors, they support sort of solution in this case, and again, in open
source there are also solutions in this case, right, and you can see there is like some
connections, right, so many open source protocols which just say, hey, we are focusing on the
storage, right, they may provide different interfaces, right, and well, and that kind
of makes sense.
The next one would be the object store, right, and that is I think a very important component
which appeared in the cloud, and that is interesting in the new commodity storage, right, because
if you think about the age before the cloud, we always had that, you know, local files
system, we had like, you know, network servers, right, with your remote file systems for very
long time, but we didn't really have anything like S3, right, until the, like at least kind
of in a common use, right, and that has appeared and used a lot those days as a building block
for many applications, because it's actually very cool, right, it's kind of bottomless,
right, you can access it on HTTP directly so you don't have to, you know, process the
pass data through your application all the time, right, it's very scalable and so on
and so forth, even many databases those days, again, like both proprietary and open source
are now starting to be built by using object store as a back end instead of your conventional
file system.
I think what is interesting in this case is what there are a lot of also object store
cloud vendors which exist, right, so it's not just Amazon, like, or even kind of major
cloud anymore, right, and here you can see like two types of commercial vendors, right,
our usual suspects, NetApp and Portworx, they do have a solution for S3 compatibility, right,
but also we have solutions like Wasabi or BlackBase, right, which are offering you S3
which you can, compatible services which we can use as less costly replacement or kind
of like a supplemental to your main cloud, right, for example, you may say, well, you
know what, I have my stuff in Amazon but I want to make sure also back up somewhere
else, you know, just so well, there are numbers of vendors out there, and then if you want
to like run the storage in your, in your, well, locally, right, there are also now number
of vendors, and I specifically wanted to flag the Minio in this case because I think they
have been the most successful, right, as providing S3 compatible interface in their, for private
cloud in those days, okay.
Now let's look at the databases and data stores.
I think the interesting thing about the database and data stores is what unlike the previous
storage types which are kind of relatively commoditized, right, have a relatively simple
interfaces and relatively simple to, to replace, like if you store data in S3, right, and now
want to store it in Minio, well, guess what, right, you have a different endpoint, maybe
have some, you know, little configuration differences, but that is not a big deal.
It's a very, very different, right, and even so-called, I would say, like a similar offerings
actually often end up to be very, very distant because of their, well, of a lot of complexity
which can, which exist in a database space, right, so that is, I think, where using some
open source solution is especially important.
So let's look at some of the databases in this case.
One, what I would call Q stream data pipelines, right, wherever we want to call it, right,
that is increasingly is a very important component of modern, you know, data-driven architectures,
right.
You often want to say, hey, we have a data comes in, right, and maybe it kind of flows
to a number of, you know, consumers being maybe, you know, processed alone, along the
way, it's kind of your data plumbing, right, it's not conventional database, but it's
very important.
What I think is interesting in this case, right, is what we are actually a lot of options.
Well, you see at the Amazon AWS, right, and they probably would have more services than
that, right, we have a huge amount of solutions in this case, some of that is because they
kind of started first, right, maybe implement something, and then open source solution exists,
right, and in general, right, because Amazon has a huge number of different services those
days, I think it's like more than 200.
If you look at the proprietary solutions, in this case, you can see Kafka is being, I
think, the most common solution these days for building your plumbing, and then additionally
we can see this technology, right, Panda coming up, which is saying, hey, we are providing
to you something which is Kafka compatible, remember, I mentioned earlier, right, what
those days, people are often building comparability of existing protocols, but it is faster,
simpler, yada, yada, right, I put them in the proprietary side, like specifically right
Panda, because they are one of those companies which started as an open source and then later
changed the license to, you know, something not quite open source, we do have a lot of
solutions in open source, it's good to point out what Kafka instead, right, is Apache open
source project, right, Confluent has a commercial offerings build on Kafka, but Kafka itself
is an open source as well as actually many other open source solutions in this space,
what I think is interesting in terms of like Qs, there is also often certain solutions
which exist in the given programming language ecosystem, right, so you will find what often,
you know, go long people will have their own choices compared to the Java people, right,
and so on and so forth, if you look at relational databases, well in the cloud we have a lot
of choices often ranging from providing you wrapped and extended open source databases
to also proprietary database available in the cloud, right, if you want Oracle or Microsoft
SQL, typically that also is available on most of the cloud.
What you also see in this case, there are a lot of proprietary solutions in this case,
right, which exist, right, and in many cases you will find those coming from your proprietary
vendor or you see a lot of companies those days which are providing the proprietary management
service right around open source databases, so for example you will find Avent here, right,
which is on like one extended provide the management services for a lot of open source
databases, but I still put them as proprietary vendor because if you can say, hey, you know,
is there this open source version of your kind of fancy GUI, right, so instead of paying
you can I take it around in my own data center, well the answer would be no, right, well,
so foundation like its solution includes open source data components as a core database
but as a whole it is not, right, and that applies to many vendors in those case.
Now, if you look at the open source, there are actually a lot of databases available
both from like an old guard like MySQL, Postgres, MariaDB, right, as well as the new folks
in the block like Ugabyte, TyDB, Percon also provides our own version for MySQL and Postgres
but typically that requires more of, I would say like manual work to deploy, right, compared
to databases as a service which exists in a proprietary space.
Here are some choices in the analytical spaces, right, that is I would think one of the big
decisions for relational databases because of kind of building database which optimizes
for transactional work load and analytical work load is kind of quite different, right,
they're designed and shownly very, very different, right, and so there are typically different
choices out there. There is a little bit of overlap those days, right, some database position
themselves as HDAB, Hybrid Transaction Analytical Databases but you know, typically the databases
are good one thing or for another. Here are some relational and analytical databases
proprietary, right, you can see number of very common solution here and then you also
have a number of open-source solutions in this case as well, right, I think what is very
interesting is what as you look at the analytical standpoint, right, they are, it's also like
a very big focus, right, if you know, very large amount of needs, right, so for example
if you look some databases mentions here like you know, Preston, Trina, right, and saying
hey we want you to provide information so you can take a data from all the different
data sources, right, and join and query wherever you directly, right, that's very valid use
case, right, something like you know, click house, focus on saying hey we provide you
sort of like a real-time analytics, right, if you want to insert the data and then have
it available for a query in the next second, well that's something what we focus on, right,
or TIDB as I mentioned they go the HDAB database, okay, I have some sign to speed up, so the
other class of databases which is quite important is the document store, right, I think if you
look at for many, you know, simple applications, some new developers, right, you just say hey
you know what, SQL relational databases, yada yada, too complicated, right, you want just
to stuff our JavaScript objects directly in the database and work with that natively,
not trying to spread them on normalized schema in relational database, but all of the cloud
vendors major one they're offering their property solution in this space as well as we do have
number of property solutions in this case, like I would say MongoDB and Couchbase are
probably the most popular in this regard which come in both cloud and enterprise space. Now
if you look at the open source, that is where I would have to say like both open source and
source available, right, because well frankly the most popular document database is MongoDB
which few years back ditched open source license, right, and well, so it is not open source
solution anymore, right, if you're looking for open source compatible, right, where is an
early stage open source MongoDB comparability, where is early stage project FerretDB which
provides interface for Postgres, right, which I mentioned. One thing I would point out here is
what relational databases, actually the lot of work recently is being much better for document
store, right, specifically in JSON support, where you take MySQL, Postgres, right, or even SQLite,
all of them are also usable, right, so in some cases when you say well you know what I want to
have some document store but I don't really completely hate open relational databases, that
also can be choice. Key value stores, that is another important model, in this case I think
it's interesting what they really can go in a two different buckets, right, one is hey we are
using that for caching, it's kind of in memory, transient, if you lose it we don't care but we
want it to be fast, there is number of solutions here, from a proprietary non-cloud solutions,
I think Redis is a main leader, in this case, right, we have both Redis Enterprise and the
cloud, if you look at the open source key value storage, solutions in my opinion, we also have a
key value, or I would say like a key value plus plus, right, because some of those solutions have a much
more powerful language than key value, would be, you know, DynamoDB, CosmosDB, Bigtable, right,
in a cloud space, Redis cloud and the enterprise versions of open source solutions, that's what
exists here, and here are some examples of open source solutions, right, which have a key value
stores, right, and again like a key value store plus plus, right, I mean, you would find like
especially, I respect mentioned here, they are, well, do much more than, you know, a key value
store, right, Cassandra as well, but they I would say not, don't position themselves as I would say
like powerful as document stores, so yes, we have also time series databases, that is another class
I wanted to cover here, right, again, you can see solutions from a proprietary vendors, from a cloud
vendors, proprietary vendors, and probably what they're most interested here is their open source,
it is also interesting what the time series database is also relatively new in technology,
which has a lot more, I would say, choices those days. Well, let me finish over also mentioned
maybe per corners role, right, in all of this and what we are trying to do, right, what we try to do
is to really see to push boundaries on what possible if specifically open source databases,
like, hey, you know what, if you want to have something which is, you know, totally open source,
our focus is on my school MongoDB and Postgres, right, I mentioned MongoDB is not open source
anymore, well, but that's not our choice, right, but I'm strict choice and we are having as much
of our tooling even for MongoDB open source as possible, and what you build is 100% open
source software around that, right, if you look at our distributions from my school MongoDB and
Postgres, right, generally include a lot of their features with enterprise companies need,
like, you know, auditing, authentication, whatever, but it is completely open source,
and we focus both on your kind of conventional or old deployments on Linux as well as we have
operators for Kubernetes, right, I think you have, like, some of the more advanced databases
out there and all that stuff, again, besides MongoDB is open source, we don't have any
proper resolution, plus we do have per quantum monitoring management, which we position as a
single tooling where you can monitor and manage databases, you know, you can get something similar
to a database as a service experience with Kubernetes backend, and again, that is all,
you know, 100% open source, which you can play with if you choose. So, to finish it up with our
storage in the cloud, right, as you probably have seen me going through that, right, some of you,
I see falling asleep, some of you rolling in your eyes, and that is totally appropriate to
action, right, because there's a lot of shit out there, right, that's like, like, lots of
options out there, so important to know here, hey, there is no one size fits all, right,
you guys can look with fits for your job for what your applications need, but one thing I wanted
you to come out of this case, it's like one last, most important takeaway is what we could see in
the all, like, where we slice and dice it, right, all those areas that have been a choice of more
than one open source solutions available in every single class of storage you may need in the cloud,
so that's all ahead. We have a little time for questions.
Hello, my question is about an interesting tool, which used to exist, and Takane used to have it
in the MySQL package, Handler Socket, so I think it was kind of discontinued, and I don't think it
supports MySQL starting from 5.7, so is there any movement in the direction of supporting this
kind of tool, which enables you to access your relational database both ways in a traditional
SQL way and in a highly available... Well, that's right, so the question is about the Handler Socket
interface for MySQL, right, and yes, and there was this, you know, interface, right, graduate,
it's, I would say, came mostly out of use, right, and we, you know, stopped supporting that,
it's, of your corner as well, there's a couple of replacements, right, one is which I think generally
cover most of the use case of what Handler Socket did, one is MySQL supports memcache protocol,
right, so if you look at for key value store, memcache comparability is out there, and then
there is also something called docstore, right, that is the MongoDB-like protocol, right,
which allows you to store documents, like JSON documents in the MySQL, that is an
other choice, right, so I think within those two, well, it covers most of the Handler Socket use
case as well. Okay, thank you, Peter.