And our next talk is by Sean, and he's going to talk about
Bottle Rocket.
Thanks.
Thank you.
Yeah, so I'm going to talk about the Bottle Rocket Container
Optimized Linux Distribution.
My name is Sean McGinnis.
I work, I'm an engineer on the Bottle Rocket project, and I work
at AWS.
So I'll just go over what a container optimized Linux is.
I'm assuming most people in this track probably have an idea.
But go over the basics, talk a little bit about what Bottle
Rocket is, show a little bit so you get a feel for it.
And what I'd really like is to see others get involved.
So the mission statement, I guess, is that it's a free and
open-source Linux-based operating system for hosting
containers.
So what does that mean?
Before we get into the motivations behind that, it's
interesting to look at general purpose Linux distributions.
And some of the challenges with using those, when you have
hundreds of nodes in a Kubernetes cluster, you really have
a lot of workloads that you're running, containerized
workloads that you're running, and you need to optimize how
you're using resources.
So with most general purpose distros, the configuration is
immutable.
You can log into the machine, you can make changes, make
adjustments, add, install extra services.
Out of the box, a lot of them come with a default baseline
set of services that you might not necessarily need when
you're just trying to run containers.
And that uses more system resources and then also
creates more of a security risk of there's more attack
service available.
And because of that, because you can log in, you can change
things, you can tweak configuration settings, those
kinds of systems are easier to become pets, where you've
really customized that node exactly how you want it.
And you're less likely to just blow that away and spin up a
new one, especially when, OK, I've made some changes there.
I don't quite remember what I changed, because I was
troubleshooting something.
There might be something important there.
So now I need to take care of this node, and it
becomes my pet.
So for container optimized Linux distributions, especially
for BioRocket, really try to optimize for just the services
that you need running on your Linux machine to be able to
run your containers.
That means less resource usage for things that aren't very
important to you.
It means less attack area for someone to compromise that
machine, get all kinds of added benefits, faster boot time,
smaller image sizes to transfer around.
So with BioRocket, we try to make things as small as
possible, just what you need, and try to make it more secure
by default.
And I don't say secure by default, because that's
impossible, try to make it more secure.
So really locking things down, making sure that if someone
were to try to compromise your host, they're going to have a
hard time doing it.
And it's open source.
It's BioRocket is not a general purpose
operating system.
If you're looking to do other things besides container
workloads, BioRocket's probably not the right
distro for you.
It is backed by AWS, but it is not an AWS-only solution.
Coming from AWS, it is very well integrated with AWS, but
I hope that doesn't stay the primary case for long.
And it is not a container-based OS.
So what I mean by that, and this comes up a lot in
conversations where there's this confusion where you talk
about a container distro.
And different people kind of already have a preconception
of what that term means.
So the two paradigms that come up are the distro, that's
kind of the base image that you build on top of.
So you've got a file from BioRocket versus your OS.
You're running, and then on that you spin up containers.
And it really is.
When we talk about BioRocket, being a container
optimized Linux, it's that second one.
BioRocket is not something that you would use to create a
container image.
A little background of BioRocket, if you see the date
in the bottom left there, we launched March 2020, which
there are a few other things going on around that time.
So didn't quite make the big splash on the launch there
that we had hoped for.
But it's great now to actually be back in person in front of
people, being able to talk about the work that we've done
in the last two years, or three years, and hopefully get
the awareness out there a little more.
BioRocket, right now, we build and distribute
different variants.
The variants term for us is how we try to optimize things
for your specific scenario.
So if you're running Kubernetes 1.22, there is a
variant specifically for Kubernetes 1.22, and Amazon
ECS, and VMware.
The reason to have these different variants is back
to that idea of no extra overhead.
So for these, for the metal variant, we try to limit the
number of kernel drivers that are loaded.
The kernel drivers that you need between a metal
deployment, where you're running on an actual server
hardware, versus where you're running in a virtual machine
instance, are different.
We would know if you're running in a virtual machine,
there's a very small subset of the available drivers that
you need to actually do that.
So anything extra is taken out of there.
Any specific agents that are needed to integrate well, like
in VMware, we have those baked into that variant.
So you pick the variants that you want for your scenario,
and that gives you a well-integrated option.
Now, Bottle Rocket isn't far from the leaders in container
optimized distributions.
CoreOS is one that really popularizes the whole idea.
FlatCars is very popular.
I just wanted to acknowledge that, make sure that
everyone's aware, there are other options out there.
None of the things here are meant to say one is better
than the other.
They all approach things from slightly different angles,
just because there's maybe a smaller list for one.
If you're using that platform, then you're well-integrated.
Maybe that is the best option for you.
But Bottle Rocket is trying to address all of these similar
problem spaces that the other distributions are doing.
We all come at it a little different way, just
like how you have Ubuntu and Red Hat and many, many other
general-purpose distributions.
So to dive into Bottle Rocket a little bit, there really
isn't too much more than these blocks.
We have the base Linux kernel.
System.bedu is used to manage things.
And we actually run two different container-y instances.
And the reason for this is, again, security.
Everything on the left-hand side, host containers, are
things that are used to manage the node.
They're things that have a little more privilege, that
might be able to access things that, regular pods that are
running on the container-y instance that's used, say,
with Kubernetes, that would be a little more locked down.
And then if there is any security vulnerability that
helps isolate things.
It's an API-driven configuration.
So when you deploy an instance, you can give a
configuration that actually, even though it ends up being a
file with settings, everything goes through the API.
And that's what actually sets the values for what happens
when this instance boots up and runs.
The host containers, so again, the things with a little
more privilege, host container is really where, how you
would access the machine.
So the actual ball rocket base itself, if you look at this
actual Linux kernel running, all the systems on there,
there's no shell.
There's no SSHD.
It really is isolated.
You need to physically connect some way to this instance.
So how do you actually do things?
And that's where the control container comes in.
That provides an environment that you can connect to and
has a few of the tools that you may need if you need to
actually interact beyond the API configuration.
One step above that, then, is from that control container,
you can launch an admin container.
And that actually does then give you the option to run SSHD.
It lets you access more of the system settings.
But it's something that's not run by default, and it's not
something that we recommend you keep running.
Really, it's there, kind of a break glass if you need it.
You need to get in the node.
You need to be able to troubleshoot some things.
This is how you would do it by running an admin container.
And then there's also other privileged things that we call
bootstrap containers.
Those are things that, they're container images that you
can customize and spin up to be able to do special things.
We've seen some cases where, OK, maybe I have some specific
file system requirements, or I need to do some special thing.
Bootstrap container is one of those host containers that's
a little more privileged that you can have it when the
system just starts up, initializes, that can do some
custom things for you.
So to see what this actually looks like, I talk a bit very
vaguely about distributions.
I'll just show you what it's like to interact with a
ball rocket host.
So typically, the only way you would interact with a host is
through whatever container orchestration you're using.
You shouldn't care what's running, what nodes are
actually part of your cluster.
You just have a Kubernetes cluster, and you use it.
So you can use things like kubectl to get
information about nodes.
You could describe the nodes if you'd like.
And if we do that, you can see here, it's an OS image,
ball rocket 1.12.
But most of the time, you would just be running your
services, your pods, your load bans, demon sets, those
types of things.
If you need to connect, then you actually need to connect to
the console.
So if you're on a bare metal instance, this is actually
plugging into the display port.
If you're using a hosting platform, this is actually
a hosting platform, this is whatever console interface
that gives you, and you're presented with this.
This is the controlled container that lets you
actually access the host.
So back to that diagram, within these host containers,
under that container instance, right now, the shell that I
have is actually this controlled container, because
there's no shell on the base OS itself.
So this gives you a little information to help you get
started.
There's the API client.
And I can use that to get my Kubernetes settings.
But if I look at Kubernetes, it puts everything under
configuration under Etsy Kubernetes by default.
There's nothing there.
There's no shell.
I can't do, let's see, I can't access file, can't make
changes here.
But there's nothing on here that shows you that this is part
of a Kubernetes cluster.
And that's because we're just inside that control container.
This isn't actually the host OS, the host file system.
The trick that I have there is there is a hidden mount that
will give you some access to the root file system.
But if you notice, I didn't point it out, there's enable
admin container, which is one of the commands that that
banner recommends, or lets you know about.
And that's what actually gets you into this admin container
that has more access.
So if I do enter admin container, I
knew I should have reset things.
Normally, you just enter admin container.
So that spins up that container instance, because it's
not running by default.
It's only when you need it.
So I started up an admin container.
Now I have a shell within the admin container.
So now I'm from there, and now that I'm in that container,
I have a little more access.
I can see some more files, but it's still not going to give
me full access yet.
So there's a tool called Shellty.
Now that I have that, now I have access to the actual
underlying file system.
So now I can go in, that's a Kubernetes, I want to take a
look at the kubelet config.
All that information is there.
So yeah, aren't command line demos exciting?
So in addition to being able to access the system only
through these controlled mechanisms, we try to
limit anything else that could be running, that could be
used.
There's a read-only root file system.
So even if a container running in your Kubernetes
cluster somehow was able to break out, we'd only have
access to the read-only file system.
It can't make any changes there.
We also use de-imverity as an extra layer of precaution.
So even if something happened, that adds some checks.
And things are locked down.
So really it'd be very difficult to compromise a system.
And then we also use SeLinks.
So there's multiple layers of protections in place here to
try to limit things.
There's PCI compliance.
Sorry, I don't know what happened to that slide.
And we are looking at FIPS compliance in the future to be
able to show that the system really is secure.
So I mentioned it's a read-only file system.
The way Bialarocket is distributed is it's image
base.
There's no YUM, there's no DNF.
You can't go in there and install extra packages.
So it is a static image.
So one nice thing about that is when you, if you want to
upgrade a node to a new release of Bialarocket, the way
it works is it'll actually download that newer image
and write it to the second partition of the root disk.
And then upgrading really is just switching over and
pointing at that new partition.
Because everything, all the settings that you have are
persisted as part of what was set through the API server.
It can switch over to this new image, reboot when it comes
up.
It reads all those settings again, uses the new image,
and post is running.
We do provide a few tools.
There's a command line interface to be able to
check for updates.
That's a pain, especially if you have hundreds of nodes.
But there's things like Bialarocket update operator,
which will handle a lot of this for you.
So if you have a Kubernetes cluster, you can schedule when
you want maintenance activities to happen.
That will automatically go out and look for new versions
being released.
And then it'll take care of interacting with Kubernetes
to coordinate off nodes, get workloads drained off into
others, upgrade those nodes, and then allow things to move
back.
So cleanly over a period of time, it'll get all of your
nodes within the cluster upgraded to the new version.
Or, and this is my preference, just replace the nodes.
They're not customized.
You know you haven't left any special thing on the file
system that you need to worry about, am I going to lose
something if I get rid of this?
You just spin up new nodes, have them join, and then you
can get rid of the old nodes and fresh system every time.
Either way works.
The configuration, like I mentioned, most of the time
you're passing in a user data file.
And that's in the Tamil format.
But really, we're an equal opportunity markup project.
So depending on how you're doing things, there's the
YAML, if you use something like EKS, everything's
configured in YAML, so you can have settings there.
I showed the API client, if you actually do want to go and
make changes on the command line, you can use that API
client and set and give it a JSON string of the settings
that you'd like or the Tamil.
And on this URL at the bottom in the repo, there's a full
listing of all the different settings that you can do with
those configuration files.
Now, the ballerocket handles things slightly different
than a lot of other distros.
So that can be a stumbling block when you look at how, if
you want to adopt ballerocket or you're trying it out and
trying to see if it works.
So there's a few things to be aware of, I guess, when you
do that.
One common thing that I've heard from users is, well, my
company requires that I run this anti-virus agent on all
of my hosts.
If they've containerized that agent, great.
If they haven't, that's an issue.
Like I said, there's no DNF, there's no YAML.
You can't go in there and install software.
So really, anything that you need to run on there, any kind
of host agents that integrate with systems, can be run in
privileged containers that can do a lot of things.
It just has to be containerized to be able to
run on ballerocket.
And then, like I said, accessing it.
A lot of sysadmins, they're used to, OK, I've got all
these nodes out there.
I know I just SSH in, and that's how I access my system
and do things.
So that can be a stumbling block, too.
Things are done a little bit differently because things are
so locked down, pros and cons.
So because things are locked down, you need to enable that
admin container.
Then you can enable SSH if you need to, but it's not going
to be there by default.
And then the last thing I wanted to bring up, just
because these are two AWS-initiated projects,
both open source, I'll be talking to someone about
ballerocket, and we'll go into some detail.
And it seems like we're both on the same page, talking
about the same thing.
And then, somehow, they say something, and we're like,
oh, wait, no, you're actually thinking about a
different project.
So Firecracker is another thing.
That's actually for a virtualized solution.
So that's ballerocket, Firecracker, explosive things,
but not the same thing.
So my main motivation for getting out and talking about
things like this is love to see more people get involved.
Everything that we can right now is up on GitHub, under the
Bottle Rocket OS org, everything is Apache 2 and MIT
licensed.
We try to publish a roadmap under that org, so if you're
curious what's happening, take a look there.
But we'd like to, the people working on the project would
like to hear from folks about what they'd like to see.
If you have ideas and you want to bring pull requests, love
that, to actually work on Bottle Rocket, we have the
Linux kernel, obviously C, but most of the Bottle Rocket
pieces of putting all this together is in Rust.
So you will need Linux, you will need Rust, and you will
need Docker to be able to do builds and things like that.
We have a Bottle Rocket SDK image that we published, so
that has the specific Rust, like the version, and there are
some Go pieces, so it has the Go tool chain.
But you do need a base requirement on your machine
to be able to actually run things.
And I say a decent amount of CPU memory and storage.
I can't really give an exact number.
You're compiling, you're building a distro.
So if you want to do that in a two core, eight gigabyte VM
on your laptop, you're going to have to be patient.
That'll take some time.
So really, the more CPU, the more memory that you have, the
better that whole situation is going to be.
But there is a building.md file in the repo.
If you are interested in that, take a look, and that will
go through everything to get you set up to be able to
actually check out the repo, make changes, and compile it.
Another area that I hope is going to help to get people
involved, we're calling right now Autotree builds.
So the variants that I spoke about, having these variants
that are very optimized for different situations.
If you wanted to build your own, say you have your own
container orchestration platform, and you want to
integrate BioRocket with it, right now you would have to
fork the whole repo and do everything within BioRocket
to get repo itself.
So we're looking at ways, how can we separate things out
and make this easier?
So if you have a customized BioRocket image that you'd like
to make for your company, for your home lab, how can you do
that without having it pulled everything in?
So if you are interested, you can subscribe to this, 2669
in the GitHub BioRocket issues.
And then even if you're not a developer, you don't have the
resources to build your own, to be able to compile
everything, if you're not interested in that, that's fine.
We'd love people to just join our community meetings.
Let us know what you're looking for, let us know if
there's anything missing from BioRocket, become part of
BioRocket itself.
So that happens right now every other week.
And we manage it through Meetup just to have an easy way to
communicate when those are.
There's a HackMD, you can throw your ideas in there, and
we can discuss them.
So I'd love to see anybody join there.
So that's meetup.com, BioRocket-community.
And with that, I'll open it up if there's any questions.
Hey, thanks for a great presentation.
Given that the file systems are immutable, where do logs go?
Does BioRocket itself log?
And I understood that the cubelet is also running
non-containerized.
So where do the cubelet logs go if you use cubelet?
Yeah, there are some very targeted areas where we
mount a tempfs.
So things like how I was talking about all the settings
through the API, you need to use those to spin up
cubelet or to run cubelet.
It needs to know its settings and needs to read that from a
configuration file.
So yeah, so if we have a read-only file system, how
does it do that?
On boot, we mount these tempfs mounts in specific places
where they're needed.
And then based on reading the configuration settings that
gets written out with the template, so if changes
somehow happen in there, your reboot comes back up and
you're exactly how you have things originally.
There's a question in chat is, is there a version of
bottle rocket which is built to run on KVM lib world?
We, there, in the repo, let me see where did I put that.
In the repo, there is, sorry, it must just be
under building, there are instructions.
Bottle rocket can be run on QMU, so if you, that's a great
development tool too, is if you want to make changes and
spin things up and just have it running, see how it works,
you can run it as a virtual machine, yeah.
Thank you for your...