So, hello everyone, first of all, this is not my talk.
I've been receiving this talk because my colleague didn't make it to get the visa on time.
So I'm sorry, I don't know anything about Kubernetes.
I'm usually more into low-level stuff, kernel, and embedded.
But I will deliver the talk with the notes that I received, and if you have questions,
you can directly direct it by email to my colleague.
I wouldn't be able to answer.
I'm sorry for this in advance.
Okay.
So before getting to the architecture and principle of the QBOS, let's define what
it's all.
So there is a cloud-native development that is encouraged by Docker Kubernetes communities.
And many infrastructure is being cloudified.
But some of the problems with the general-purpose operating systems reappear in this cloud-native
environment.
So you have container management, workloads scheduling, automatic service deployment,
rollbacks of updates, and so on.
That's all capabilities that are provided by Kubernetes, but it is unable to control
the cluster-node operating system directly.
So the first problem in cloud-native environments is the desynchronization between OS and Kubernetes
that are managed and controlled completely separately.
Also Kubernetes, like the operating system management, needs a key, upgrades, user access
control, all these things.
And then you can have like the ops operation guys or patient people, sorry, that need to
complain ridden and task between the two systems.
The maintenance are therefore poorly synchronized usually, and the greater modification of the
OS components can affect the availability of the OS and which require additional monitoring
from Kubernetes.
So an example is that you have operation staff that must block the nodes to stop new workloads
from arriving in order to upgrade the OS without interfering with the Kubernetes.
And after everything is clear and everything is updated, you can unblock the node again.
So this makes it complicated and expensive.
So another issue is the OS version management.
So if you have a standard package manager and you can add, remove, modify packages independently
on the OS, at the beginning you have an image which is clean, but then you start differing
from your different instances.
So you have like what they call OS version splitting.
So you will have different packages installed on different nodes.
The version of these packages can also differ, security updates and all that stuff.
So you have this divergence that appear over time.
So if you want some integrity and consistency that you want to ensure for your OS nodes,
this can harm this constraint.
And yes, so if you want also to update to a major version, it's also more difficult.
So other people have worked on this problem.
So rebuilding the operating system is an approach that has been taken to solve these problems.
So previously you have many technology packages that are part of the OS that are moving to
containers.
So the old guest OS is less reliant.
We rely less on the guest OS so it can be replaced by a lightweight operating system
with less services that are on and so on.
So container OS is a lightweight operating system designed to run containers.
And so like on the figure on the right, there is an OS OS and it's not the OS running inside
the container.
So you have three important aspects, minimalism, usability and atomic updates.
It means that you will only include what you really need as components in the host OS.
So the container OS requires a Linux kernel, container engines like Docker, container D,
and security mechanisms such as SE Linux to ensure the security.
And other applications that are running containers are running containers because you don't need
it in the host.
And this can also reduce the attack surface because you have less in the host OS.
Emutability is that you use a read-only file system that can be configured at the start
of the deployment and also reduce the risk.
And the atomic update is that you do the upgrade for the entire OS and not individually for
packages.
So the core OS was started in 2013 and was the first widely used container operating
system.
You also have a system like AWS bottle rocket, flat car, and container optimized OS.
So QBOS, it's a container operating system built on OpenOiler, which is a distribution
maintained by Huawei.
So QBOS main design concept is to use Kubernetes to manage the operating systems.
Once you have QBOS that has been installed on a cluster, the user only knew the Qube
control command and YAML file on the master node.
The OS of the cluster worker node can be managed.
And this OS on QBOS is connected to the cluster as a Kubernetes component, putting it in the
same position as the other resources in the clusters.
And containers and operating system can just be matched in a unified way through Kubernetes.
So OpenOiler based reconstruction is used so that the operating system can be updated
optimally, like to avoid the problems I introduced before.
So now we are going to go a little bit in more depth about QBOS.
So the first feature is the ability to manage the OS through directly Kubernetes.
So we use API extension, custom resource, CRD, to design and registering in the cluster.
We use Kubernetes operating framework to create customized controller for the OS to monitor
and manage it.
Then this Kubernetes operating framework, we use it to create customers.
So the user only need to modify this CR, enter the expected OS status to the cluster, and
the QBOS and Kubernetes handle this, and you only have to manage it in the control plane.
So the next one is atomicity management of the OS.
QBOS upgrade is an atomic dual zone upgrade.
It does not include packet manager.
The change of each software package corresponds to the change of the operating system version.
Then the OS version corresponds to a specific OS image or RPM package combination.
Each software update as shown in this diagram is an OS version update.
So you avoid the version splitting problems, and the cluster nodes remain consistent at
all times.
So QBOS is lightweight with unnecessary components removed to reduce the attack surface and enable
faster start-up and upgrade.
So this is a diagram of the QBOS overall architecture.
So you have two main parts.
The first with three different components, OS operator, OS proxy, and OS agent.
In the red box above the diagram, which are used for Kubernetes cluster docking, complete
OS monitoring and management.
And the second part is the QBOS image creation tool.
The user can use QBOS scripts to generate QBOS images from the open or lower repo source,
which supports the generation of container image, virtual machine image, and so on.
So the three main components I mentioned, like OS operator, proxy, and agent, are critical
to the ability to manage cluster using Kubernetes.
The OS operator and proxy are the operators we mentioned earlier.
The OS operator will be deployed in the cluster as deployment and daemon set, and will communicate
with Kubernetes to issue upgrade instructions.
The operator is a global OS manager that monitors all cluster nodes.
When a new version of the OS information is configured by the user, it determines whether
to upgrade and send a great task to each node.
The proxy is a single node operating system manager that monitors the current node information.
When the operator sends a great notification, it will lock the node to expel the pods and
forward the OS information to the agent.
The agent is not included in the Kubernetes cluster.
The real executor of the OS management communicates with the proxy via Unix domain sockets, receive
a message from the proxy, and perform the upgrade rollback and configuration operations.
So the upgrade process, we will use the work process as an explaining example.
So we consider how the different components communicate and interact.
First the user configures the OS information to be upgraded via Qt control and enable files,
such as OS version, address of the OS image, number of nodes to be upgraded concurrently,
and so on.
Then when the OS instance changes, the operator begins the upgrade process, labels the nodes
that must be upgraded, and limits the number of nodes to be upgraded each time to the number
specified by the user.
Then the proxy checks to see if the current node is marked as an upgrade node, locks the
nodes to expel the pods, and retrieves the OS information from the cluster before sending
it to the OS agent.
After receiving the message, the agents will download the upgraded package from the address
specified by the user, complete the upgrade, and restart.
After restarting, the proxy will detect that the node OS version has reached the expected
version and will unlock the node and remove the upgrade level of the node.
So this is the complete upgrade process.
Then finally the file system.
So how do we design and upgrade the file system in QBOS?
It adopts a dual-area upgrade, like mentioned earlier, to upgrade the OS, so you have two
root partitions, the upgrade of partition A is to download the updated image for the partition
B, and then modify the default bootloader as the B partition after, and then you restart
from the B by default, and the opposite happens for the next upgrade.
So it's a classical dual image thing.
The file system of QBOS is recently, which improved the security, but we also support
persistent data partitions.
The union path, which is mounted as an overlay, and the files in the image other than the
user change can still be seen.
There is a writable path, which has a writable file layer to the image using the bind mounts.
The files in the image are not displayed, only user data is stored, and there is also
the boot partition, which contains the bootloader files.
So we determine the main concept of QBOS and design, and implemented a set of components
to complete the OS management, and we intend to continue completing more functions based
on this process.
One thing is the ability to provide a configuration, like in the grid process, the configuration
is delivered to the node via the Kubernetes cluster on the cluster control plane to ensure
the consistency of the configurations of the nodes, and given that some of the configuration
must be complete before the nodes join the cluster, more configuration capabilities to
the QBOS image creation are planned.
Then there is the improved upgrade capability.
We have realized the function-based OS upgrade, and we will provide upgrade strategies that
user can customize, such as upgrading based on the cluster node label to provide more
upgrade solutions.
In addition to the rich functions, we intend to improve the usability of QBOS by displaying
the upgrade of configuration process and improving the image creation tool so that user can
more easily customize the image.
Okay, and that's it.
Sorry again for the functions, but for the question, you can always shoot the colleague
in the middle.