So, once again, hello everybody, welcome to my talk.
This talk is going to be about OSV, evolution of OSV towards greater modularity and composability.
Thanks for introducing me.
So I've been contributing to OSV since 2016.
Here I go, in 2015 I heard about OSV in one of the conferences and then a couple of years
later I was nominated to be one of its committers and my greatest contributions to OSV include
making OSV run on Firecracker and significantly improving ARCH 64 port among other things.
So I'm not sure if you can tell it but OSV is actually my hobby so I'm not like a real
current developer like many of previous speakers are so it's actually, you know, I work on
it in my night when I feel and I have a day job so I don't represent my company that I
work for so this is all my personal contribution to the project.
So in today's presentation I will talk about enhancements introduced by the latest release
of OSV 057 with the focus on greater modularity and composability but I will also discuss
other interesting enhancements like lazy stack, novel ways to build ZFS images and improvements
to the ARM port.
Finally I will also cover an interesting use case of OSV, seaweed FS running on OSV which
is a distributed file system.
So as you can see in this talk besides the title, modularity, I will actually try to
give you like state of the art where OSV is, how it has changed recently and a little bit
of where it's going hopefully.
So I know there are probably many definitions of unique kernels and each of them is a little
bit different right but so I'm sure most of you understand what unique kernels are but
just a quick recap with emphasis on how OSV is a little bit different.
So OSV is a unique kernel that was designed to run single and modified Linux application
on top of hypervisor whereas traditional operating system were originally designed to run on
a vast range of physical machines.
But simply speaking OSV is an OS designed to run single application without isolation
between application and kernel or it can be thought as a way to run highly isolated process
without ability to make system calls to the host OS.
Finally OSV can run on both 64-bit X86 and ARM V8 architectures.
Now a little bit of history, so OSV for those that don't know, OSV was started in late 2012
by the company called Cloud Use Systems and they built pretty strong team of 10, 20 developers
I think.
I wasn't one of them but they pretty much wrote most of OSV but at some point they basically
I guess realized they have to make money I'm guessing so they basically moved on and started
working on this product you may have know CillaDB which is this high-performance database
but I think they took some learning so and after that basically I think OSV did receive
some grant from European Union so there was some project on that and I think there may
have been some companies also using OSV but honestly since then it's been really maintained
by volunteers so like me like there's still some people from CillaDB, Nadaf, Harrell and
others that contribute to the project you know I would just single out Fortisks and
Akis which actually was the one that implemented Virtio FS as a you know for very interesting
contribution to OSV and obviously I would like to take this opportunity to invite more
people to become part of our community because honestly you may not realize it but our community
is very small so it's just really me, Nadaf and a couple of other people that contribute
to the project so I hope we you know we're gonna grow as a community after this talk.
So quick recap of a little bit of how OSV looks like what the design is so in this slide
you can see major components of OSV across layers starting with G-Lipsy, the top which
is greatly based actually on Musil, then core layer in the middle comprised of ELF dynamic
linker of VFS, virtual file system, networking stack, thread scheduler, page cache, RCU,
read copy update, page table management and L1, L2 pools to manage memory and then you
have a layer of device drivers where we OSV implements Virtio devices on both of our
PCI transport and MMIO transport and then Zen and VMware among others and obviously
and one more thing so as we can run on KVM based hypervisors like QMU like Firecracker
I did test also OSV on cloud hypervisor which is I think Intel's hypervisor written in
Rust and then I personally didn't really run OSV on Zen so I know that the Zen support
is a little bit dated probably and I'm not sure how much it has been tested.
I did test on VMware Vbox, virtual box and I think on Hyperkit at some point.
So I will just I want to go into more detail about this diagram but I will leave it with
you just as a reference later.
So in the first part of this presentation I will about modularity and composability
I will focus on new experimental modes to hide the non-Gilipsi symbols and standard C++
library.
I will also discuss how ZFS code was extracted out of the kernel in form of a dynamically
linked library and finally I will also explain another new build option to tailor the kernel
to a set of specific drivers.
I call them driver profiles and another new mechanism to allow building a version of kernel
with a subset of Gilipsi symbols needed to support a specific application which I think
is quite interesting.
So by design OSV has always been a FAT unicolonal and which has been some sort of some of the
criticism and by default provided a large subset of Gilipsi functionality has included
full standard C++ library and ZFS implementation drivers for many devices and has supported
many hypervisors.
So on one hand it makes running arbitrary application on any hypervisor very easy using
a single universal kernel.
But on another hand such universality comes with the price of bloated kernel with many
symbols and drivers and possibly ZFS that is unused.
That's causing inefficient memory usage, longer boot time and potential security vulnerabilities.
In addition C++ application linked against one version of LeapSTD C++ different than
the version the kernel was linked against may simply not work.
For example that happened to me when I was testing OSV with.NET and the only way to
make it work was to hide basically the C++ standard library and use the one that was
part of the.NET app.
So one way to lower memory utilization of the guest is to minimize the kernel size.
By default OSV comes with a universal kernel that provides quite large spectrum of Gilipsi
library and full standard C++ library and exposes over a total of 17,000 symbols and
most of those are very long as C++ symbols that make up the symbol table.
So the question may be posed why not have a mechanism where we can build a kernel with
all known Gilipsi symbols hidden and all unneeded code that is unused garbage collected.
So the extra benefit of fewer exported symbols is increased security that stems from the
fact that there is simply less potential code that is left that could be harmful.
And also that way we can achieve better compatibility as any potential symbol collisions for example
and mismatch standard C++ library which I mentioned can be avoided.
So the release 057 added a new build option called conf-hide symbols to hide those non-Gilipsi
symbols and the standard C++ library symbols.
These are enabled if enabled in essence most files in a source tree of OSV except the ones
under Lipsi and Musil directories would be compiled with the flags of visibility hidden
and only if that build flag is enabled.
On the other hand the symbols to be exposed as public like the Gilipsi one would be annotated
with OSV Asterisk API macros that translate basically to attribute visibility default
and the standard C++ library is linked with the flag no whole archive.
Those SV Asterisk API macros basically would be like OSV Lipsi API or OSV Pthreads API
OSV Lipsi API and so on basically that match all then I think around 10 libraries that OSV
dynamic linker exposes.
Finally the list of public symbols exported by the kernel is enforced during the build
process based on the symbol list files for each advertised library like for example Lipsi
SO6 and is maintained under the directory exported symbols.
So these files are basically list of symbols that are concatenated using the script called
generate version script and which goes into version script file and then is fed to the
linker as an argument to the version script file.
So in order to now remove all unedited code basically garbage all files would be compiled
with the function sections and data sections and then they would be linked with the flag
GC section.
Now any code that needs to stay like for example the bootstrap start point or dynamically
enabled code like the optimal memcpy implementation or trace point patch size is retained by
putting relevant kept directives and relevant sections in the linker script.
The kernel L file built with most symbols hidden is roughly 4.3 megabytes in size compared
to 6.7 which is reduction of around 40%.
This great reduction stems from the fact that the standard library standard C++ library
is no longer linked with whole archive.
The symbol table is way smaller and unused code is garbage collected.
Please note that the resulting kernel is still universal as it exports all glipsy symbols
and includes all the device drivers.
And as a result of this size reduction kernel boots also a little bit faster.
Well this all sounds great so one may ask why not hide most symbols and standard C++
library by default.
The problem is that there are around 35 unit tests and some also applications that were
written in the past that rely on C++ symbols and they basically would not run if we hide
all of those symbols.
And those are basically used to, they were implemented in the past and it was done sometimes
out of convenience, sometimes basically out of necessity.
So to address this specific problem we will need to expose some of those OSVC++ symbols
as the API expressed in C.
So we'll basically define very simple C wrapper functions that we'll call those C++ code.
Well I can use this one.
A good example of modularity improvements made in the release 057 is extracting ZFS
code out of kernel as a dynamically linked library, LibSolarisSO, which effectively is
a new module.
To accomplish that we changed the main OSV make file to build new artifact, LibSolarisSO
out of ZFS and Solaris file sets in the make file, which basically used to be linked into
kernel.
The new library has to be linked with a bind now flag and OSV specific OSVmlog node to
force OSV dynamic linker to resolve symbols eagerly and populate the mappings eagerly
as well.
This basically is done to prevent page faults that would lead to potential deadlocks as the
libraries loaded and initialized.
The init function ZFS initialized called upon the libraries loaded creates necessary
thread pools and registers various callbacks so that the page cache arc, which is adaptive
replacement cache from ZFS and ZFS depth driver can interact with relevant code in
the ZFS library.
On another hand, the OSV kernel needs to expose some around 100 symbols that provides some
internal free BSD originating functionality that LibSolarisSO depends on.
OSV borrowed some code from free BSD and actually a good chunk of this code was actually implementation
of ZFS, which right now is outside of the kernel.
Finally, the virtual file system bootstrap code needs to dynamically load LibSolarisSO
from bootFS or read-only-FS using DL open before mounting ZFS file system.
There are at least three advantages of moving ZFS to a separate library.
First off, ZFS can be optionally loaded from another file system like bootFS or read-only-FS
partition on the same disk or another disk and I will actually discuss that in more detail
in one of the upcoming slides later.
Then also, kernel gets smaller by around 800 kilobytes and effectively becomes 3.6 megabytes
in size.
Finally, there are at least 10 fewer threads that are needed to run non-ZFS image.
So for example, when you run read-only-FS image on OSV, with one CPU it only requires
25 threads.
The regular Linux Jalipsi apps should run fine on kernel with most symbols and standard
C++ library hidden, but unfortunately many unit tests which I mentioned and various internal
OSV apps which are written mostly in C++, so-called modules, do not, as they had been
coded in the past to use those internal C++ symbols from the kernel and we have to do
something to deal with that problem.
So in the release 057 we introduced some of the C wrapper API which are basically in C
style convention and then we changed those modules to use those C wrapper functions instead
of C++ code.
The benefit is that down the road we might have some newer apps or some newer modules
that would use those C wrapper functions and it also may make OSV more modular.
As you can see some of those, one of the example is, for example, OSV gets all threads which
is basically a function that gives a thread safe way to color, to iterate over threads
which, for example, is used in an HTTP monitoring module to list all the threads.
A good example of OSV specific modules that uses some internal C++ symbols is HTTP server
monitoring.
We modify the HTTP monitoring module to stop using internal kernel C++ API.
We do it by replacing some of the calls to internal C++ symbols with this new module C
style API, symbols from the slide which you saw on the slide before, for example, SCAD
with all threads, with this new OSV get all threads function.
In other scenarios we fall back to standard G-Lipsy API, for example, the monitoring app
used to call OSV current mounts and right now it uses basically getMTNT and function
and related ones.
So the release 0.57 introduced another built mechanism that allows creating a custom kernel
with a specific list of drivers intended to target given hypervisor.
Obviously such kernel benefits from even smaller size and better security as all unneeded
code, all unneeded drivers are basically excluded during the build process.
In essence we introduce a new build script and makefile parameter, driver, driver's profile.
This new parameter is intended to specify a driver profile which is simply a list of
device drivers to be linked into the kernel and some extra functionality like PCI or ACPI,
these drivers depend on.
Each profile is specified in a tiny include files with the MK extension under conf profiles
arch directory and included by the main makefile as requested by the driver profile parameter.
The main makefile has a number of basically if expressions and add conditionally given
driver object to the linked object list depending on the value of 0 or 1 of the given conf drivers
parameter specified in that include file.
The benefit of using drivers as are most profound when they are used with when you build kernel
and hide most of the symbols as I talked about in one of the previous slides.
It's also possible to enable or disable individual drivers on top of profiles as profiles are
basically list of the drivers but the number of configuration parameters that where you
can specifically for example include, which I'm going to be actually showing here, you
can include specific driver.
One may ask a question why not use something more standard like when you config like for
example what Unicraft does, well actually OSV has this specific build system and I didn't
want to basically now introduce another way of doing things so that's where we basically
script build uses the various effectively parameters to for example to hide symbols or
specify specific driver profile or list of other parameters.
So as you can see in the first example we built default kernel with all symbols hidden
and the resulting kernel is around 36, 3.6 megabytes.
In the next example we actually use, we built kernel with the VIRTIO over PCI profiles which
is like 300 kilobytes smaller and then in the third one we built kernel which is intended
to for example for firecracker when we include only VIRTIO block device and networking driver
over MMO transport and then just to see basically in a fourth one just to see how large the
driver's code in OSV is when you basically use driver profiles base which is basically
nothing, no drivers, you can see that roughly 600 kilobytes of the driver's code is roughly
600 kilobytes in size.
And then in the last one actually option is where you can specify, you use basically
driver's profile and then you explicitly say which specific drivers or you know driver
related capability like in this case ACPI, VIRTIO FS and VIRTIO NET and PV panic devices
you want to use.
Actually with the new release of OSV 057 we started publishing new versions of new variations
effectively of OSV kernel that correspond to this I thought you know interesting build
configuration that I just mentioned and in this example the OSV loader hidden artifacts
are effectively the versions of OSV kernel built with most symbols hidden and then for
example which will be at the top for both ARM and X86 and then for example right here
in the second and third and fourth artifacts basically version of the kernel built for
micro VM profile which is effectively something that you would use to run OSV on Firecracker
which only has VIRTIO over MMIO transport.
Now the release 057 introduced yet another built mechanism and that allows creation
of a custom kernel by exporting only symbols required by a specific application.
The extra such kernel benefits from the fact that again it's a little bit smaller and
tasks offers better security as in essence all unneeded code by that specific application
is removed.
This new mechanism relies on two scripts that analyze the built manifest, detect application
L files, identify symbols required from OSV kernel and finally produce the application
specific version script under app version script.
The generate app version script iterates over the manifest files produced by list manifest
files pi, identifies undefined symbols in the L files using objectDump that are also
exported by OSV kernel and finally generates basically the app version script.
So please note that this functionality only works when you build kernel with most symbols
hidden.
So I think what is kind of interesting worth noting in that approach is that you basically
run a built script against given application twice.
Basically first time to identify all symbols that application needs from OSV kernel and
then actually second time we do is to build the kernel for that specific app.
In this example we actually generate kernel specific to run a simple going app on OSV
and when you actually build kernel with symbols around I think 30 symbols by going pi example
the kernel is effectively by around half megabytes smaller and it's around 3.2 megabytes.
So this approach has obviously some limitations.
So some applications obviously use for example DLSM right to dynamically resolve symbols
and those would be missed by this technique.
So in this scenario basically for now you have to manually find those symbols and add
them to the app version script file.
Basically a lot of Jalipsi functionality is still in OSV in Linux CC where all the system
calls are actually implemented is still basically references all the code in some of the parts
of the Lipsi implementation so this obviously also would not be removed.
So obviously we could think of ways of finding some kind of build mechanism that could for
example find all the usages of Cisco instruction or SVC on ARM and analyze and find all this
only code that is needed.
In the future we may componentize other functional elements of the kernel for example the DHCP
lookup code could be either loaded from a separate library or compiled out depending
on some build option to improve compatibility while also planning to add support of statically
linked executables which would require implementing at least clone BRK and arch PRCTL Cisco.
We may also introduce ability to swap built in version of Jalipsi libraries with third
party ones for example the subset of libm so that is provided by OSV kernel could be
possibly hidden with the mechanism that is discussed and we could use different implementation
of that library.
Finally we are considering to expand standard PROCFS and CISFS and OSV specific parts of
CISFS that would better support statically linked executables but also allow regular
apps to interact with OSV.
A good example of it could be implementation of net stat like type of capability application
that could expose the networking in terms of OSV better during runtime.
In the next part of the presentation I will discuss the other interesting enhancements
introduced as part of the latest 0.57 release.
More specifically I will talk about lazy stack and new ways to build ZFS images and finally
the improvements to the ARH64 port.
The lazy stack which by the way is actually the idea that was felt off by Nadav Harrell
which maybe is listening to this presentation effectively allows to save substantial amount
of memory if an application spawns many p-threads with large stack by letting stack grow dynamically
as needed instead of getting prepopulated ahead of time which is normally the case right
now with OSV.
So on OSV right now all kernel threads and all application threads have stacks that are
automatically prepopulated which is obviously not very memory efficient.
Now the crux of the solution is based on observation that OSV page fault handler requires that
both interrupts and preemption must be enabled when fault is triggered.
And therefore if stack is dynamically mapped we need to make sure that the stack page fault
never happens in these relatively few places where the kernel code that executes with either
interrupts or preemption disabled.
And we basically satisfy this requirement by refolking the stack by reading one byte,
one page down per stack pointer just before preemption or interrupts are disabled.
So a good example of that code would be in a scheduler right when OSV scheduler is trying
to figure out what the next threat to switch to.
And obviously that code has preemption and interrupts disabled and we wouldn't obviously
want to have page fault happen at that moment.
So there are relatively few places when that happens and this idea is to basically pre-fault
this code.
So to achieve that we basically analyze OSV code to find all the places where the IRQ
disabled and preempt disabled is called directly or indirectly sometimes and pre-fault the
stack there if necessary.
As we analyze all call sites we need to follow basically five rules.
The first one do nothing if the call in question executes always on the kernel thread right
because it has pre-populated stack there's no chance that page fault is going to happen.
Second one is do nothing if the call site executes on other type of pre-populated stack.
The good example of that would be the interrupt and exception stack or Cisco stack which are
all pre-populated.
And the number three rule is do nothing if the call site executes when we know that either
interrupts or preemptions are disabled because we don't need to somebody already probably
pre-faulted that.
And then pre-fault unconditionally if we know that both preemption and interrupts are about
to be enabled right.
And otherwise pre-fault stack by determining dynamically basically by calling the preemptable
is preemptable and IRQ enabled functions.
And now the idea basically if we only always if we did if we followed only rule number
five which actually this is what I tried to do in the very beginning the first attempt
to implement lazy stack it will be actually pretty inefficient.
I mean I saw pretty significant degradation of for example context switch and other parts
of the OSV when I dynamically checked if preemption and interrupts were disabled.
So this was accessible pretty painful to basically analyze the code but I think it was worth it.
As you remember from the modularity slides the ZFS file system has been extracted from
the kernel as a separate shared library called LipsolizeSO which can be loaded from the different
file system before ZFS file system can be mounted.
This allows for three ways ZFS can be mounted by OSV.
The first and original way assumes that ZFS is mounted at the root from the first partition
of the first disk.
The second one involves mounting ZFS from the second partition of the first disk and
at an arbitrary non-root point for example slash data.
Similarly the third way involves mounting ZFS from the first partition of the second
or higher disk at an arbitrary non-root point as well.
Please note that the second and third options assume that the root file system is non-ZFS
obviously and which could be like read-only-FS or boot-FS.
This slide shows you the build command and how OSV runs when we follow the original and
default method of building and mounting ZFS.
For those that have done it, there's nothing really interesting here.
This is a new method, the first of the two new ones where we actually allow ZFS to be
mounted at a non-root mount point like data for example and mixed with another file system
on the same disk.
Please note that lib-solaris-so is placed on the root file system typically read-only-FS
under USR-lib-FS and loaded from it automatically.
The build script will automatically add the relevant mount point time to each ZFS.
The last method is basically similar to the one before but this time we allow ZFS to be
mounted from the partition from the second disk or another one.
It's actually what happens with this option, I noticed that OSV would actually mount ZFS
file system by around 30 to 40 milliseconds faster.
Now there's another new feature we used to run in order to build ZFS images and file
system we would use OSV itself to do it.
With this new release there's a specialized version of the kernel called ZFS loader which
basically delegates to this utilities like Zipple, ZFS and so on to mount OSV but there's
also now a new script called ZFS image on host that can be used to mount OSV ZFS images
provided you have open ZFS functionality on your host system which is actually quite nice
because you can mount basically OSV disk and introspect it, you can also modify it using
standard Linux tools and unmount it and use it on OSV again.
Here's some help on how this script can be used.
Now I think I don't have much time left but I will try.
So there's also, I will focus a little bit on the AR64 improvements, I will focus on
three things that I think are worth mentioning, the changes to dynamically map the kernel
during the boot from the second basically gigabyte of visual memory to the 63rd gigabyte
of memory, addition enhancements to handle system calls and then also handle exceptions
on a dedicated stack.
As far as the moving memory, virtual memory to the 63rd gigabyte so I'm not sure if you
realize OSV kernel is actually position dependent but obviously the kernel itself may be loaded
in different parts of physical memory so and it used to be before that release that you
would have to build different versions for Firecracker or for the QEMU.
So basically we, in this release we changed the logic in the assembly, in a bootloader
where we basically OSV detects itself where it is in a physical memory and in essence
the, you know, dynamically the early mapping tables to then eventually bootstrap to the
right place in the positional code.
So now basically you don't need to, you can use the same version of the kernel on any
hypervisor.
Now we had system calls on ARM, we had to handle the SVC instruction, there's not really
much interesting if you know how that works and what is maybe a little bit more interesting
was the change that I made to make all exceptions including system calls to work on a dedicated
stack so before that change all exceptions would be handled on the same stack as the
application which was, which wasn't you know really, which caused all kinds of problems
and it was, for example, that would effectively prevent implementation of the lazy stack.
So to support basically that we would, you know, SV which runs in EL1 in a kernel mode
we would basically take advantage of the stack selector register and we would have, we would
basically use both stack pointer register SPL0 and SPL1.
So normally OSV uses SPL1 register to points to the stack for each thread.
So with the new implementation what basically we would do before the exception was taken
basically we would switch the stack pointer selector to SPL0 and once basically the exception
was handled it would basically go back to normal which was SPL1.
I think I was going to skip C with FS because we're running very little half time left but
you can read it on that.
Yeah we've also added netlink support and we've made quite many improvements to VFS layer
so both actually of those netlink and VFS improvements were done to support C with FS
so there are basically more gaps that have been filled by trying to run this new use
case.
So just briefly as we are pretty much at the end of the presentation I think in the next
releases of OSV whenever they're going to happen I would like to, I would like us to
focus on supporting statically linked executables, adding proper support of spin locks because
OSV for example Mutex right now is lockless but under high contention it would actually
make sense to use spin locks and we have actually a prototype on that on the mailing
list and then supporting ASLR, refreshing Capstan which is a build tool which hasn't
been really out because we don't have volunteers, improved for a long time and then even the
website and there are many other interesting ones and so I would, as a last slide I would
like to basically use this as occasion to thank basically organizer Razvan for inviting
me and everybody else from the community of Unikernals and I would also want to thank
ScyllaDB for supporting me and Dorlaor and Nadav Harrell for reviewing all the patches
and his other improvements and I also want to thank all other contributors to OSV and
I also would like to invite you to join us because there are not many of us and if you
want to have OSV alive we definitely need you and so there are some resources about
OSV, there's my P99 presentation here as well and yeah if you guys have any questions I'm
happy to answer them, thank you.
Thank you Voldemort, thank you.
So any questions for Voldemort, yeah please Marta, just ask it's going to be a bit of
the mic.
Okay I have two questions, first when you have spoken about the symbols, about the
G-Lipsy symbols and the symbols for symbols, do I understand it correctly that the problem
is that the kernel might be using some G-Lipsy functions and the applications might be linked
to its own G-Lipsy and so-so symbols apply basically?
Well not really, they would use the same version it's just you know and there's no problem
with for example malloc, like malloc we don't want to expose malloc but there is a good
chunk of OSV is implemented in C++ and all of those symbols don't need to be exposed
because they inflate the symbols table a lot and they are not, they shouldn't be really
you know available to, visible to others and yeah I mean now I think OSV exposes if you
build with that option around I think sixteen hundreds of symbols instead of you know seventeen
thousands.
So it's really about the binary size there?
Yeah, yeah basically binary size and with in case of C++ library avoiding a collision
where you build OSV with different version of C++ library versus you know the application
that.
Yeah okay so this is the case I'm interested in, so have you thought about maybe renaming
the symbols in the kernel image during link time, maybe adding some prefixes to all the
symbols so that you can have them visible but they would not clash?
That's an interesting idea, I haven't thought about it yeah.
And Marty the other second question, yeah.
Yeah that's just a quick second question, so when you have spoken about the latest tag
you said that you pre-fold the stack to avoid the problematic case when it drops in preemption
disabled, so basically when I'm thinking about it you still need to have some kind of upper
bound of the size of the stack so that you know that you pre-fold it large enough to
not get into the issue.
So my question is why not then have the kernel stacks in all fixed size because if you already
need to have some upper bound then why not have a local upper bound for the whole kernel?
Wouldn't it be just easier?
Well I mean this is for applications threads only, so for application stacks where the
kernel threads would still have the pre-populated fixed size stack, yeah so because I mean there
are many applications like good example is Java that would start like 200 threads and
all of them right now are proposed like one megabyte and all of a sudden need like 200,
so this is just for application.
Okay so basically my understanding is wrong, so you have the user stack and the kernel
stack is the same stack?
Well no, it's in the same virtual memory but yeah, I mean when I say kernel stack I mean
in OSV basically there are two types of threads, there are kernel threads and there are application
threads so basically application threads use their own stack, but when they enter the kernel
so to speak they are still reusing the original stack right?
I mean application threads use application stack and kernel use kernel and when I say
like some kernel code obviously because unicernel as the code executes in an application it
runs on application stack but it might execute some kernel code as well which yeah, yeah.
Thank you, any other question?
Okay thank you so much, let's move on.