Okay, welcome everyone, starting with the next session, so yeah, try to find a place,
if you're free to use the reserved spaces if nobody comes. Let's welcome Patrick on the
talk on SQL on stuff like this.
Alright, hey everybody, I'm Patrick Donnelly, I know I got Red Hat slides up, but I'm actually
part of the storage group at Red Hat that got moved over to IBM as of this year, so technically
I'm at IBM now, if that matters for anybody who wants to ask me questions. Today I'm talking
about SQL on CIF, it's kind of a small project that started about two years ago, it was actually
a COVID project for me while I was dealing with the newborn baby that had lots of, or
some time on my hands, but anyway, this is kind of like an overview for what we're going to talk
about. Yeah, go ahead. Yeah, I do.
We don't have a PA, unfortunately.
Oh, yeah, I just have to speak up, so I'm naturally soft spoken, so if you can't hear me in the back,
just wave your hand and I'll try to speak up more, okay?
I'm okay right now, alright. Where was I?
So just a quick canvassing the audience, who's used CIF before?
Oh, wow, okay. Who's used SQLite before?
Fewer people, that's interesting, okay, but not much fewer.
Alright, so I'm going to quickly talk about CIF and what it is.
I won't spend too much time on it due to the time I have available in my talk.
I'll give you a brief introduction to RATOS for anyone who's not familiar with it.
Then I'm going to talk a little bit about SQLite.
Then some typical storage patterns we use for storing data on RATOS.
I'll give you an introduction to this new library, LibCefSQLite.
And then I'm going to talk about how we use that today within CIF,
just to show that this library is not being used by anyone.
Although I am interested if anyone's using it in the community.
I'll give you a brief, go through a brief tutorial,
an interactive tutorial using the library,
and then I'll end the talk with some retrospective and talk about future work.
So what's CIF?
CIF is an ecosystem for distributed object storage.
It's composed of numerous projects centered around managing a large storage cluster.
The underpinning of CIF is RATOS, which I'll talk about on the next slide,
but it's basically a distributed object store.
Most people don't use it, use RATOS directly.
What they use instead is the storage abstractions we built on top of RATOS,
which provide the more popular storage mechanisms that people are familiar with,
including CIFFS, which gives you your file storage as a distributed file system,
RGW, providing the S3 object storage gateway,
and RBD, which gives you your block device storage on top of RATOS.
CIF has kind of evolved more and more recently to become more user-friendly.
If you had maybe poor experiences in the past with CIF,
I encourage you to give it another shot.
The dev team has dedicated a lot of time recently to improving the user experience,
and also taking the hassle out of managing your storage cluster out of the experience.
There are things like monitoring device health.
We now have a very mature dashboard for interacting with the CIF cluster,
and the cluster management itself is now largely being done through CIFADM,
which, as you saw in the previous talk,
you can start up a CIF cluster with just a simple command and then start adding hosts to it.
It's never been simpler.
Oops, went backwards.
So what's RATOS? RATOS is a number of object storage demons that run on physical disks.
They can be hard disks, SSDs, and VMEs.
On top of these object storage devices, we have this concept of pools,
which allows you to have various administrative policies
regarding what kind of hardware the pool should use,
how the data should be replicated.
Clients of RATOS talk directly to the primary object storage device for a given object,
and you can look up which object storage device an object belongs to in constant time
using a library called crush.
You don't need to use that directly. That's just under the covers.
And then, as part of the name suggests, reliable, autonomic object storage.
Distributed object storage, the cluster heals.
Self-heals, it's autonomic, and the replication is done automatically.
You don't have to worry about how any of that works.
So what's an object?
The object storage device is composed of a number of objects,
and that is the logical unit you have when you're storing things in RATOS.
An object is composed of three different parts. You can use one or all.
They have the data blob, which is analogous to, like, a regular file.
Like, you put data in the file, you get data out of the file.
You have key value X-Satters.
This is sort of an older technology that was used in the early days with CEPFS
for storing certain information about files, which is typically very small data.
It's not usually used anymore, except in some parts of CEPFS.
Now, the key value store that's used most often in CEPFS, also RGW,
is OMAP, and that is much more of a general purpose key value store used today.
So this is how you interact with RATOS through these objects.
Now, it's not that simple to take, like, a number of objects,
distribute all over the cluster and try to build something with that,
because you've got consistency issues that you have to deal with.
You've got to manage how you're going to stripe the data across all these objects,
which is why we have these more popular abstractions that I talked about,
CEPFS, RBD, RGW, which is how you typically interact with RATOS.
So what I'm going to talk about today is a SQLite library
that operates alongside these other three storage abstractions,
gives you something on top of LibRATOS, but you can also now run SQL on CEPFS.
So how do you typically do application storage on RATOS?
Well, we have various bindings you can use to talk to RATOS.
We have the typical CC++ bindings, which are part of the broader project,
also used within CEPFS.
We also have a Python interface, which is used for manipulating the objects.
That's somewhat used in the broader community for various projects,
but also within CEPFS we use it for some of the new CEPFS manager daemon,
which I'll talk about more later.
And again, it's not that simple to stripe data across the objects,
which is why we have these other abstractions.
One of the more notable exceptions is this LibRATOS striper,
which is one of the ways you can create a file concept on top of objects,
where you open and close, read and write and sync to a number of objects,
and it looks like a regular file.
That was developed by some folks at CERN,
and it's mostly, I think, in terms of use, it's stayed confined to that space.
Well, even though we do have these other storage abstractions,
it's still useful to talk to RATOS directly,
because sometimes you want to do something that is not dependent
on these other storage abstractions,
which may, in the case of within CEPFS internals,
may not actually be available,
which is why a number of CEPFS manager plugins,
the CEPFS manager has a number of Python modules,
and they talk directly to RATOS.
So this was something I wanted to address,
because it was a little bit awkward,
and I'll talk about that more.
So a quick overview of SQLite.
For those who've never used it before,
it's a user application library for allowing you,
the X is a SQL engine,
and lets you store a SQL database as a regular file,
usually two files.
It'll be a journal, and then the database object itself.
And depending on how you use it,
the journal is transient, may come and go.
It's widely recognized as one of the most used SQLite engines
in the world.
It's very popular.
They estimate on their website,
there's billions of SQLite databases in use.
It's at least tens of billions at this point,
because it's in every Android phone.
So it was easy choice to make.
It's a very simple library,
and bindings exist for numerous SQL systems.
In particular, of interest to me, was Python.
Actually, extending SQLites is fairly simple.
They have this VFS concept, virtual file system concept.
It lets you swap in different virtual file systems as needed.
The basic one is the UNIX VFS that's what comes with SQLite
by default, and it's very intuitive.
It just passes on open, read, write,
closed off to the local file system for execution.
So Libsef SQLite is a library for a SQLite VFS library
that lets you put a database, SQLite database in RATOS.
It's composed of two parts.
Libsef SQLite and simple RATOS striper.
I'll talk about simple RATOS striper on the next slide.
The use of this library does not require any application
modification, and that's kind of like the killer feature here,
because you can just set some environment variables
and modify the database URI,
and you can automatically start storing your database in SEF.
And all of these, you know, the journal objects,
the database objects are striped across the OSDs.
You don't need to do anything differently.
The simple RATOS striper is based loosely off
of the Lib RATOS striper developed by CERN.
The main reason I didn't end up using CERN's library
was because it had some locking behavior
that was not really desirable for a highly asynchronous use
case, and I didn't want to modify their library
out from under their feet, so I just wrote a simple version.
It provides the primitives that SQLite needs,
open, read, write, close, sync, and all the writes
are done asynchronously, and then the sync call
that comes from SQLite actually flushes them all out.
So these are all stored across RATOS with these names,
you know, foo.db, and it's got like the block number
associated with the database, and so on.
Using LibSeph SQLite, again, it's very easy.
You just have to load a VFS library.
This is done with the SQLite command.loadLibSephSQLite,
and then you just provide a URI for the database.
This is the pool ID or the pool name,
the namespace within that pool, which is optional,
and then you give the database a name and specify
the VFS is Seph, and that's it.
It just works.
You may have to specify some environment variables
if you're using the SQLite binary to give it,
tell it which Seph cluster to use,
or which Seph configs to read, things like that,
but that's all fairly unobtrusive, not obtrusive.
Within the Seph manager, so the Seph manager
is one of the newer demons in Seph
that takes care of certain details
of managing your Seph cluster
and trying to provide easier interfaces.
A particular interest to us is one that handles
health metrics that come from the OSDs,
giving the Seph manager information
about the smart data associated with the disks,
being able to anticipate failures in disks,
again, Seph trying to reduce the management burden
of storage clusters, and then also a portal
to higher level commands like managing volumes
within Seph that is a subvolume concept
that's used by OpenStack or Kubernetes, CSI.
Within the Seph manager demon, what I observed
was that there were several modules
that were just storing data in the OMAP key value
of a particular object, and it turned out
this doesn't scale very well.
We know it won't scale well because if you have
more than 10,000 key value pairs in a single object,
the performance starts to degrade.
In fact, you'll start getting cluster warnings
that there's objects with too many key value pairs.
So it was also pretty awkward in terms of how it was being used,
and just by how we were managing the data,
it was a perfect match for a SQL database,
except it was not very easy to put a SQL database
on Seph at the time.
In fact, Jan here worked on SNAP schedule,
which is a module for creating snapshots
and maintaining snapshots in Seph FS
and handling retention policies,
and that actually used a SQL database
that was flushed to RATOS objects and then loaded
in anticipation of the project that I'm working on now,
and that's all been updated now to use this
of Seph SQLite library.
All right.
So in terms of how it actually looks within the Seph manager,
on the left we have a schema.
It's fairly simple.
Just creating a table with the device ID
as the primary key, and then another table
with device health metrics.
With the time we got the metrics,
the device ID associated with that metric,
and then the raw smart text.
And then they actually put the device metrics in the database.
It's as simple as this.
Within the manager I've taken out a few unnecessary lines
just for space or unnecessary keywords
for space in the SQL.
You create the device ID,
which just calls another SQL statement
to insert into this table,
and actually execute the SQL statement
with the Epic dev ID and data.
It's that simple.
And now that's stored, persisted, and RATOS.
So here's a quick Libsep SQLite in action series
of gifts I've created.
Here we're running the Seph status command,
just showing us the state of the cluster.
We have two pools right now, a.manager and an A pool
that I'm creating for this demo.
Here I'm purging A just to show that there's nothing in it.
It removed one object.
And here I'm just listing all the objects within this pool.
There's none because I just purged it.
So that's just a starter.
And then here we're actually going to run the Libsep SQLite.
So to do that, again, I mentioned there were
some environment variables.
If I'm using the SQLite command directly,
I have to specify some environment variables.
So the library knows what to do.
Here, because this is a dev cluster,
I have to tell it to use the library path
associated with my build.
I specify which Seph config to load, which key ring associated
with the admin user that I'm going to specify here.
I was actually going to also add some logging data,
but I ended up not doing that, just to save space.
And then here I'm actually running a SQLite command.
I'm loading the Libsep SQLite library.
That's one of the first command that Libsep SQLite is going to run.
And here I'm opening a database in pool A,
namespace B within that pool,
and then a database named a.db with vfsf.
All right, now I'm in SQLite.
Here I create a simple table with an integer column.
There's the schema, exactly what I wrote.
And then we're going to insert into the table one value
and then dump it.
So it's now in RATOS.
And now just to confirm that, I'm going to run the RATOS command
on the pool A, list all the objects in the pool.
You can see namespace B. I have this a.db.
So now I'm going to use this striper command.
I'm actually, if this database were composed of many objects,
you can use the striper command to actually pull the database out.
And you can see here I've done that.
It's an 8k database.
It's small because there's just one table with one value.
And I loaded that locally.
I pulled it out of RATOS.
Sorry, the GIF loops.
I pulled it out of RATOS.
And then I now have the database as a local file.
I ran SQLite on that local file database.
And just dumped it to confirm that it actually wrote the data out
to RATOS correctly.
And I can pull it out of RATOS and verify that it actually worked.
So here's another demo with just rerunning the same SQLite command
I had earlier.
And sorry, this is going to be a big paste.
But I'm creating a table.
This is just some magic in SQLite to basically create an infinite loop.
And I'm just going to insert a number.
I think it's 100,000 integers into the table.
And just see how many objects are in the database now.
There's four objects composed of that database.
So I think for time reasons, I'm not going to go through the performance notes,
but it's on the slide if you want to look at it later.
And just as a retrospective for Quincy, when the database got used live,
it's being used in the two manager modules right now,
the device health and the SNAP scheduling module.
It's been fairly successful.
We had a few minor hiccups that weren't really too much related to the library.
And just for some future work, I want to add support for supporting concurrent
readers.
That's not yet possible right now.
All readers and writers obtained exclusive locks when accessing the database.
But there's not a technical reason why we can't add concurrent reader support.
And then I also want to look at adding read-ahead performance,
improving read-ahead performance.
Because right now every read call in Libsef SQLite is synchronous.
So that's the end of my talk.
Thank you.
Do we have any time for questions?
Sorry, no time for questions to this session.
You have to find Patrick and I'll send him.