Okay, so let's start.
So hi everyone, I'm going to be talking about advanced camera support on all-winner SOCs
today.
So the topic is quite similar to the previous one in that we are also going to talk about
sensors that need particular care and processing.
So unfortunately I will be maybe explaining some of the things that were said by Kirin
just earlier, so sorry about that for the people who just attended the previous talk.
But hopefully you'll also learn a thing or two in addition.
So I'm Paul, I work at a company called Butlin, we are an embedded Linux engineering company,
so we do services around that.
And I have contributed a number of things, especially to the Linux kernel, so I worked
on the Sidrus VPU driver, a little bit of DRM things on the all-winner side, but also
others.
And I made a training that we give about displaying and rendering.
But today I am here to tell you about the camera support for all-winner SOCs using mainline
Linux.
So we are going to talk first about general things related to image capture technology
and just complex camera pipelines in general.
So let's start with kind of an overview of a typical capture chain.
So we start with the optics, of course, where the light is kind of focused on a particular
area where we have our sensor.
So the optics are usually passive from a software perspective, but not necessarily because you
can have coil drivers to change the focus and things like that, so it's not always a
passive thing.
So after that you have the sensor, which actually samples the light and produces data from that.
And we want to transmit that data to something else, typically like your CPU, where you can
do something with the data, like display it or encode it.
But in between the acquisition and actually receiving the data, we need to do some processing.
And that processing can sometimes happen on the sensor itself, in which case we talk about
some embedded processing, or it can be on the other side of the interface.
So on the side that receives the data, typically your SOC or your CPU package or whatever.
So this processing step is really the one that is necessary to produce good looking
pictures from, let's say, just samples coming out of an ADC.
So typically what we call a row or barrier sensor will produce not pixels, but data in
a biopattern, which is a grid of red, green and blue filters that is applied on the sensor.
That gives you information for each of these different channels.
And we kind of need to translate that to pixels.
So this is called debiring, and this is how we get pixels.
But these pixels typically look very bad.
So you need to apply a number of processing, number of operations and enhancements to have
something that looks like a nice picture.
So a number of things need to be done.
For example, the brightness that you get from your ADC is linear, and we want to apply some
gamma curves to that to make it look nice to the human eye.
Typically, there's some dark level current that we have to subtract, for example, because
the zero value that you get from the sensor is not necessarily the, well, the darkest
value that you get from the sensor is not necessarily zero, so you might need to subtract
an offset, things like that.
There's usually a lot of noise, and the colors will be off, so you will need to do some white
balancing, things like that.
So all of these different steps take place in what we call the ISP, the Image Signal
Processor, and there's basically three domains in which we apply these enhancements.
The first one is the Biodomain, so that's really the first step that we apply to the
data coming from the raw sensor.
At the end of that step, we get some RGB data that we also want to enhance, and at the end
of that, we typically convert it to YUV representation, and then we can also apply some enhancements
to that data, and at the end, we get a YUV picture that is, like, ready to be displayed
and encoded, for example.
So yeah, that's kind of a list of the different enhancements that we apply, so I mentioned
a number of them already.
I'm not going to go through the list, but you get some idea that there is really a
lot of things to be done here, and it actually takes quite some processing power to do that.
So that's why typically we consider that it's not something you can do in real time with
a CPU or it's going to fully load your CPU just to produce pictures and let alone encode
them and things like that.
So lots of things to do.
That's really the base that you need to have something that looks right.
There's more advanced stuff that you can have in addition, like the lens shading correction,
so it's about the fact that the lenses will typically be darker on the edges than they
are at the center, so you want to kind of even that out.
That's also an operation to do.
Dewarping, that's when you have, like, a very short focal, and things look, well, the geometry
looks distorted, so you need to kind of readapt that.
That's also very intensive in terms of calculation.
Stabilization can also be involved if you have, like, a very shaky footage, especially
from, like, a smartphone or something like that, so you want to also apply a stabilization
step to your picture, and then, finally, you might also want to apply some style to your
picture, so that will typically be a color lookup table where you can decide that you
want to make it look, I know, like, CPiaton or something like that.
This is also some processing that you will need to apply.
So I mentioned that there's basically two types of ways to deal with this operation.
The first one is to have the ISP in the sensor, in which case it's typically quite simple,
and when it's in the sensor, you get the data directly ready from the sensor, but when it's
not, you get just the raw Bayer data, and you need to do all of these different enhancement
steps on some system on a chip ISP, so that's typically a hardware block that is dedicated
for the purpose in your SoC.
So nowadays, many multimedia-oriented SoCs do have such blocks, and in order to properly
configure that, you might need some specific calibration data that really depends on the
sensor, sometimes on the environment that is used, things like that, so it's kind of
highly specific to the setup that you have.
So it's kind of just an illustration of the different steps, so that's the kind of picture
that you would get as a raw thing from the sensor, and the steps you might apply to have
something in YUV at the end that looks kind of okay.
But it's not just about statically configuring an image processor to produce something good.
Some parameters actually depend on the thing that you are shooting, so there's basically
three things that you need to adjust depending on the situation.
The first one is focus, so of course that implies that you have control over some coil
to change the focus of the lens, but obviously your picture is going to look very different
if it's out of focus or if it's sharply focused, so that's one of the things that the ISP
is also involved with to basically tell you whether an image is sharp or not.
There is white balance, which highly depends on the source light that you are using, especially
the color temperature of that light, so if you are in broad daylight, it's not the same
as being in a room with some particular type of lighting, so it needs to adjust to that.
It will typically have an impact on how the image looks.
And of course exposure, so what is basically the window of luminescence that your sensor
is going to sample, so if you are in a very bright environment, you need to apply a different
gain than when you are in a very dark environment, so that's also something that needs to be
adjusted depending on what you are shooting, and this is also something that the ISP is
going to help for by telling you basically how bright or how dark the scene is.
So yeah, you can adjust those, especially exposure, you can adjust with three parameters,
so if you do like photography, you probably know about that.
You can change the aperture of the lens, you can change the exposure time, so for how long
you are waiting for light to come in to charge yourselves that will be read by the ADC, and
you can increase the gain, which will also increase noise typically.
So advanced users will typically want to control these parameters manually to have exactly
the picture that they want, but in most cases people just want to take their phone out and
shoot at something and just press a button, it just works, so the idea is that we want
all of these different parameters to be adjusted like automatically.
So that's what we call the 3A, so the 3As are automatic exposition, auto focus and auto
white balance, and that is typically again something that will be done with the ISP.
So it works with a feedback loop, okay, there's a number of algorithms in the literature that
exist that are known to be able to do that correctly and efficiently, but the way they
are implemented in the actual ISP hardware really depends, of course, on the hardware
itself and how it was designed and what the like register interface is to configure these
different things, and that is often considered to be the secret source of the manufacturer
of the ISPs, so that's the information that is often kind of difficult to get and that
they don't want to release, and so that's why sometimes you end up with like a big binary
blob that does all of this and you don't really know what's going on.
Okay, so that was for the kind of, yeah, parameters for the image enhancement.
Now let's take a little bit of a look at the hardware interfaces for the capture.
So historically there's been different ways to transmit pictures from one side to another.
There used to be analog interfaces which are now mostly deprecated, so let's not really
focus so much on those.
And then we have typically two types of hardware interfaces that are used for cameras.
First one is the parallel, also called DVP sometimes, and that's when you basically just
have like one line of data per bit, you have some sync signals, so it's a little bit like
a display, a parallel display if you should know about that, and so you just kind of send
the data like that, and there's also more advanced interfaces which are also more robust
to noise that typically work with serial lanes, so there is MyPy CSI2 and other ones like
LVDS, SDI, High Spy.
So those are kind of the high end interfaces that allow you to stream a lot of data, they
typically go pretty high speed, they are more robust to noise, so they are considered to
be like the advanced ones.
So that's the one MyPy CSI2 that we are going to focus on.
Through my particular use case involving the Alwinner platforms, so in case you're not
familiar with the Alwinner platforms, they are ARM SoCs, made by this company called
Alwinner from China, they are widely available, especially on these kind of form factors as
developer boards, and there's a number of these platforms that support MyPy CSI2 and
that have an image signal processor, so it means that we can connect a raw Bayer sensor
and get the data from that, pipe it through the ISP, and then get a picture at the end.
So that was kind of the goal of the project that I had involving these platforms.
So the scope was on the V3 and A83T platforms using two different image sensors, OV8, 865
and OV645648, which are, like I just said, MyPy CSI2 sensors that provide raw Bayer data.
And these sensors don't really have an onboard ISP.
I think one of the two actually has one, but it does very little, so you still need to
do a lot on the receiving end of the interface.
So that was the goal.
That's the state of all-winner camera support in general with the mainline channel, because
we wanted to use the mainline channel, of course.
Let's first take a look at the general all-winner platform support.
So there is a community called Sanxi, or Sanxi, I think, which has been working towards mainline
support for all-winner SOCs, so it's very advanced, there's lots of people involved.
You can check out this link, the Linux mainlining effort, which kind of lists all the features
of the different SOCs and how they are currently supported in mainline Linux, and it's pretty
impressive nowadays.
Many of the features are supported, especially for the older SOCs, because of course it takes
time to get it right.
But the multimedia areas are often the ones that come last in support, because they are
typically a bit complex to implement.
So when I started the project, there were two drivers for capturing data.
The first one is the SunFry CSI driver, which covers the first generation of these all-winner
platforms.
It was the hardware then evolved into a second generation, which is supported in mainline
by a driver called SunSixi CSI.
After that, all-winner made a new generation of platforms, which have a third generation
of CSI, which is currently not supported.
So the devices that I was interested in, so the V3 and A83T, work with the second generation
driver.
So this driver basically allows you to receive images from the parallel interface, but it
didn't support MyPi CSI 2, and it didn't have support for the ISP.
So there was actually some support for these features in the downstream vendor all-winner
kernel.
So they do have some code for that, but the ISP part, especially, was implemented as a
binary blob, so it was like a static library that was linked to the kernel, which is not
necessarily very legal, but never mind.
So there was actually very, very few resources regarding how the ISP works on these platforms.
Okay, right, okay.
So generally speaking, how do we support cameras in Linux, at least at the kernel level?
So there's this API called v4l2 that I think you've all heard of just before, and probably
many people know about.
So it's really about supporting anything that produces pixels that the CPU can receive.
So it supports lots of different types of devices, not only cameras, but also, I don't
know, things like skaters, DVBT receivers, lots of different things, now decoders, encoders,
things like that, so really lots of different devices related to pixels.
And typically the way it works is that you have one device node that corresponds to a
driver, so typically dev video zero, and that device node gives you access to an API from
user space where you can do all the different things that are necessary to get a picture
from user space.
So typically negotiating the pixel format that you want to receive, doing the memory
management like allocating the buffers, how many buffers you want, et cetera, queuing
and decuing buffers.
So user space provides a buffer to the driver which will fill it with pixels and then return
it to the application, and then the application has a buffer that has pixels in it that it
can use to, again, display or encode them or whatever.
So this video device works well for, I would say, all-in-one devices where you basically
just receive the finished data from a device like a USB-UVC camera.
So the camera itself will do all of the processing inside, and it will just give you the final
result over USB, and you get that through this API on Linux.
And yeah, typically you need some DMA interface to do that transfer.
But in the case of a more complex pipeline, especially when you have multiple components
involved, like with the ISP, with the Mypy CSI2 receiver, with a particular sensor that
you can control directly, then you end up with a situation where you have multiple devices
in the pipeline, and you kind of need to configure each one of these devices individually.
So this called for a more advanced API, which is the subdev API, which allows not only to
have one big device for receiving the data, but also side devices that you can use to
configure each component in the chain.
And there is also the Media Controller API that allows you to kind of control the topology
between these devices.
So the subdevs typically just represent one of the parts of the pipeline, and they typically
cannot do DMA.
So they will be connected from and to other devices through some interfaces that don't
involve writing the data to memory.
So it could be a FIFO, or it could be an actual hardware interface like Mypy CSI2.
And basically, the top-level video device will be in charge of kind of calling the next
subdev in the chain, which we'll call the next one it's touch to, and et cetera.
So that, for example, you can coordinate starting the stream and starting all the elements at
the same time to start receiving an image.
But these subdevs still need to be parented to the V4L2 device.
So basically, they need to be all controlled under the same top-level entity to be able
to, let's say, coordinate between one another.
So for that, there is an API in V4L2 that allows you to register the subdevs with V4L2
device.
So again, that's the parent controlling entity, which is easy to do if all of the support
for the subdevs are in the same driver, because you have access to that V4L2 dev pointer.
But it can also happen that you have multiple drivers involved throughout the tree.
So for example, you have one driver for your sensor, one driver for your DMA interface
to transfer the data, one driver for your ISP, and you could even have more.
So in that case, the drivers don't know exactly which other driver they should be attached
to.
So in that case, there is a asynchronous subdev registration interface, which allows
you when basically you have, for example, a sensor driver to just make that subdev available
to whichever driver is going to need it later.
So the subdev drivers will just make the subdev available to the rest of the world.
And then the top-level drivers will need a way to identify which subdevs they actually
need and to get a handle of them, which will allow registering these subdevs with the top-level
V4L2 device.
So the way that this kind of linking is done is through the FW node graph, which is typically
implemented in device tree.
So it uses the port and endpoint representation that maybe you've seen in some device trees
implementing this, and this description also allows describing some characteristics of
the interface.
For example, if you have a sensor that is on a MyPyCSI interface, it can use a different
number of lanes.
So in MyPyCSI 2, you can have up to four lanes, but maybe the sensor only uses two.
So you have to kind of be able to share this information.
And this is also done through this FW node graph description.
So you have some device tree properties that you had to indicate that.
And then the drivers can call these endpoint pass helper to actually retrieve the information
about the interface.
So to illustrate, on the left side, we have some sensor here.
So we have the port and endpoint representation.
The remote endpoint allows you to connect two sides together, and you have these extra
properties here like the data lane and link frequencies that really describe the characteristics
of the bus, so at which frequency it should be running and how many lanes it should have.
And then on the other side, you have the same thing.
In this case, the link frequency is controlled by the sensor, so you only need to provide
it there, but the data lanes is present on both sides.
So that's how you can link basically different devices and allow the top level driver to
retrieve access to the sub devs that you want to use.
So this is very flexible, of course, because then the same, for example, sensor driver
can be connected to lots of different platforms and lots of different situations.
So it's really the driver itself doesn't know about how it's connected.
It's really the device tree and the FWU node graph that tells you how it works.
So back to async notification, just quickly to illustrate how the top level driver would
gain access to a sub dev.
So first, it has to match using that FWU node graph representation.
It has to match a particular sub dev and the top level driver registers a notifier which
has a number of callbacks that will be called when a particular device becomes available
and then it can pretty much bind to that device and then the matching sub dev will be registered
with the top level vfoil to device and then everything can be linked together and the
top level driver actually has a pointer to a vfoil to sub dev that it can use to apply
some actions like stop streaming, stop streaming or configure the format or things like that.
So this is how it kind of all works together.
So yeah, that's also when the media controller API comes in.
So the media controller API is there to control the topology of how these different devices
are actually connected between one another.
So it also implements particular functions.
So you can say this block attached to this sub dev is an entity of this kind, okay?
And each sub dev has an associated media entity which lists pads which are basically in and
out points that you can use to connect other devices.
And then you can create links between these pads which represent the actual connection
in the hardware.
So for example, you could have multiple links that are possible for one device and then
you could decide to enable one at runtime.
So for example, if you have a multiplexer or something like that, that would be a typical
case where you would just select one of the inputs and have just one output.
So this is really the API that allows you to configure the topology of the whole pipeline
and how everything is connected together.
There's also some runtime validation to make sure that when you connect two entities, they
are configured with the same pixel format so that everyone agrees on what the data will
be, the data that will be transferred.
And there is a user space utility called media CTL that you can use to configure these links.
So for example, here I'm configuring pad number one of this sub dev to be connected to pad
number zero of this sub dev and the one indicates that the link should be enabled.
So yeah, it's a bit blurry.
This is kind of just to give you some kind of big idea or kind of a head start on that,
but it's definitely complex, so it's normal that it seems a little bit blurry, it's just
in case you have to work on that, then you know what are the things involved in this.
So in the end, we can end up with very complex pipelines, okay?
So each of the green blocks are sub devs, okay?
So they represent a specific functionality that can be connected in different ways.
And the yellow blocks are the actual DMA engine, so the video nodes that are visible from user
space that programs can connect to to receive the data.
But of course, if you haven't configured the rest of the chain properly, then there will
be no data available.
So this is really what you use at the end when everything is configured and everything
is ready and it works.
Okay, so that was for the general pipeline integration thing.
Now let's talk about ISPs more specifically.
So ISPs are just a kind of sub dev and media entity.
And they typically have an internal pipeline with multiple things in it, so we don't necessarily
represent the internal pipeline unless it's relevant.
So there will normally just be one sub dev for the ISP.
But this sub dev will have highly specific parameters.
Like I said, it depends on the hardware implementation.
So the representation of the parameters that you give to the hardware will differ from
one implementation to another.
So it means that it's actually very hard to have like a generic interface that will work
for every ISP and that would be the same.
So instead of that, in V4L2, there is actually driver-specific or hardware-specific structures
that are used to configure the ISP sub devs.
So the way it works is that we have, so one or more capture video devices that's the same
as the dev video zero where you get the typical data, the final data that you want.
And we have extra video devices that we can use to configure the ISP and to get side information
from the ISP.
So these are the meta output and meta capture video devices.
So the meta output is there for parameters.
So in V4L2, output is when you provide something to the driver, not when you get something
from it, which is a bit confusing, but that's where it is.
So with that, basically, you will also use the same Q interface as you have with a video
device.
But instead of having pixels in the buffers, you will have particular structures that correspond
to the parameters of the ISP that you are going to fill with a particular configuration.
And then you can push that as a buffer to the video device, and the ISP will be configured
to use those parameters.
For the meta capture, which is the data provided by the ISP, you get the typical feedback information
from the ISP, so essentially it will be statistics about how sharp the picture is, how dark the
picture is, things like that, so that you can use this information to create a feedback
loop and then provide new parameters in the output video device to properly configure the
ISP to respond to a change in the scene or something like that.
So for example, if you switch off a light and turn a different one on that has a different
color temperature, for example, then you will get the information from this statistics,
and you will be able to adjust the parameters to respond to that change.
So that's how it works.
Here is an example from the RK ISP, the Rockchip ISP1, where you can typically find this same
topology.
So the ISP is here.
It actually has extra sub-devs before having the video devices for capturing the pixels.
But you also find this statistic video device and params video device.
So the params will take a particular structure here that you can configure, and the statistics
will take another one with the information provided by the ISP.
Okay, so that gives you kind of a big overview of how all of this is supported in V4L2 in
Mainline Linux.
So now let's take a look at the thing I actually worked on for the all-winner cameras.
So using, again, these same interfaces for the particular use case of all-winner cameras,
or cameras, you know, interfaced with all-winner SoCs.
So in the all-winner second-generation hardware implementation, we have MiPy CSI2 controllers,
which are really the components connected to the actual bus, the actual MiPy CSI2 bus,
which are separate hardware blocks that are connected through a FIFO to the CSI controller,
which is really just a DMA engine that will get some pixels in and write them to memory,
basically, with some formatting and timing things.
But essentially, that's what it does.
So this CSI controller again was already supported in Mainline, but not the MiPy CSI2 controllers.
So the CSI controller actually also needs to be configured specifically to take its
input from the MiPy CSI2 controller instead of the parallel interface, which is the only
choice that was supported before.
So that's one of the things I had to add support for.
So there was a lot of kind of reworking of the CSI code to support that, even though
the biggest rework was actually to support the ISP.
We need to get some information from the sensor to properly configure the MiPy CSI2 interface
on the receiving side.
So for that, we use a V4L2 control that the MiPy CSI2 controller is going to retrieve
from the sensor driver through the subdev interface again.
So it knows what the clock frequency of the bus will be.
And we also use a part of the GenericLinux-Phi API to do that, because MiPy CSI2 works with
a physical, let's say, protocol or physical implementation called DeFi from MiPy, which
is kind of like the physical layer implementation that is used by this interface.
So there needs to be some configuration about that.
And yeah, for that, we use the Linux-Phi API.
Now if we look more closely at the platforms that I got interested in, first, for the A83T,
there was actually some source code provided in the all-winner vendor releases that we
could use as a base to implement a driver, a proper mainline driver.
So it has lots of magic values in registers, so sometimes it's just writing things to registers
and we have no idea what it means, but we basically just took that in and did the same
and it just worked.
So there's still some magic involved, but that's unfortunately not so uncommon, so we
just have to deal with it.
The DeFi part is separate, so it has different control registers, but that was also supported
in that all-winner SDK downstream code, so we could also just reuse the same thing and
it worked.
For the A31 and V3 supports, so it's like, again, the second generation of all-winner
SOCs, we have a different MiPy CSI2 controller from the A83T, so it was necessary to write
a separate driver for that one.
There was also reference source code available and some documentation in one of the user
manuals of the platforms, so that was, again, sufficient to write a driver.
It turns out that the DeFi part is actually the same controller that is already used for
MiPy DSI, which is a display interface that uses the same physical layer encapsulation,
I would say.
So there was actually already a driver for the DeFi block used for MiPy DSI, in which
case it's in transmit mode, because when you want to drive a display, you push pixels
out, but in that case, we reused that driver but configured it instead in receive mode
for MiPy CSI2, so we could get pixels in.
So that was also a change in this driver.
But it was then necessary to indicate in which direction it should be running, so there were
different approaches that were possible for that.
So I think at the end, we settled for a particular device tree property to configure this mode.
So the kind of outcome of this work was first some series to support the MiPy CSI2 controllers,
so about 2,600 added lines, so pretty big, that's two new drivers here and here, some
changes to the DeFi, like I just mentioned, and some device tree changes, so that's most
of it.
I started this work in October 2020, and it was merged in the next 6.0 in June 2022.
So now these drivers are upstream, and you can use them, and they work, and I actually
got a number of people writing to me and saying that they actually have been using this in
different situations, and apparently it works pretty well, so I'm pretty glad about that.
It's pretty nice.
So that was for the MiPy CSI2 part, and let's say the big part of the work was supporting
the ISP.
So the ISP is connected to the CSI controller as well, but on the other side, meaning that
the data will flow from MiPy CSI2 to the CSI controller to the ISP.
So there also needed to be some configuration to be able to support that, especially big
rework was required because when you start using the ISP, the DMA engine that is used
to write the data to memory is no longer the DMA engine of the CSI controller.
So the CSI has to act like a regular subdev, okay?
It's no longer the final, let's say the final sync for the data, but it's just one more
element in the chain.
So the driver had to be reworked to support this different mode of working, where it will
basically not register itself as the parent V4L2 device, but instead it will register
itself as a subdev and the parent V4L2 device will be the ISP driver, which is again a separate
driver.
So that required quite some rework, and also to support both modes, obviously, because
not everyone is interested in using the ISP or not every platform even has an ISP.
So yeah, so there needed to be some, yeah, some rework to support that.
What else to say, it has, I don't know if I put it here, but it has some weird way of
configuring it, basically, in a typical hardware, you would just like have some registers and
configure them, and then the effects will be applied on the next frame or something
like that.
But in that hardware, it actually has a DMA buffer, where you write the new values of
the register, and then you trigger some update bits, and the hardware itself will go and
read from the DMA buffer and copy that data to its registers synchronously with the virtual
synchronization, so when you receive a new frame.
So it's very odd as a way of working, but that's how it does.
So like if you write directly to the registers, it won't actually do anything.
You need to write to a side buffer, and then tell the hardware to update its registers
from that buffer.
So yeah, it's a little bit weird.
If you look at the driver, you'll see that there is this buffer that is allocated for
that, so that's the reason why, that's how it works, and that's what the old winner code
is doing.
So that's how it's done.
So that's the final pipeline that we have with the sensor here, connected to the Mypy
CSI 2 subdev, which is a separate driver.
Then it goes through the CSI driver, which in this case is configured as a subdev only.
And then it goes to the ISP subdev, which provides a DMA capture interface where you
have the final data that was processed, and that should look good.
And it also has another video device for the parameters.
Like I described with the RockTip ISP, this one is implemented the same way.
So we also have a specific structure to configure it.
Currently, there is no support for the statistics, but in the future, when such support is added,
there will be another video device connected to this ISP subdev to be able to provide the
feedback data out.
OK, so yeah, that's pretty much what I just said.
Few details about the currently supported features in that config parameters buffer.
Currently, we support the buyer coefficients, so we can translate from the buyer raw data
to actual RGB data, and we can tweak how much of each color channel we put in.
So that will typically allow different color temperatures, basically.
We also support 2D noise filtering, which is called BDNF, so it's bi-directional noise
filtering, which basically is like a low-pass filter, so it will remove the high-frequency
stuff in your picture, and that will make it look smoother and nicer.
And also easier to encode, which is one of the big reasons why you need to do noise filtering.
And yeah, that's the main two features, so there's still a lot to be added.
That's just the scope of what our project was at the time, but there's definitely a
lot of room for improvement, so the ISP itself has numerous hardware capabilities, and so
those could be added later in the driver.
So it was, for that reason, submitted to staging in Linux, because we don't yet support all
the features, so we don't yet have a complete description of that structure, and since it's
part of the API, we want to make it clear that it's not finalized yet, so there will
be some additions to this structure to support other features that are currently not implemented.
So this code was submitted in September 2021, and it was merged in November 2022.
So this is also in Linux 6.2, so you can get that with the update, so that's pretty nice.
This change was much bigger.
You can see it's 8,000 lines of additions, so it's a whole new driver, and a big rework
of the previous 6 ICSI driver, which was more or less a complete rewrite of the driver,
so it's pretty big.
Just to finish on what is left to do in this area, so currently the ISP only supports the
V3 platform, but the same hardware is found on the AT3T, and there's a few other chips
that have previous versions of the same hardware, so they could be supported in the same driver,
so that's something that could be done in the future.
I mentioned that there is no statistics currently, so that is also something that could be added
in the future.
It has numerous other features that we could support, scaling rotation, and of course all
of the modules inside the ISP for all the different features that I mentioned, and we
don't have any 3A algorithm support in user space to do this feedback loop implementation,
so that is also something to be worked on.
And of course, doing that would be a great fit for Lib Camera, so Teran has just talked
about it, so I won't go over it again, but that's definitely a good fit for supporting
an ISP with Mainline Linux, so hopefully it will soon be well integrated in Lib Camera.
Someone recently submitted patches about this, so it's like going towards this direction,
so that's pretty nice.
That's pretty much the end of this talk.
I just wanted to mention that Bootlin is hiring, so if you are interested in this kind of stuff,
how to support everything, you can reach out to us and we have positions available, also
internships, so feel free if you're interested.
And that is pretty much it for me, so thanks everyone, and now I'll have questions if there's
any.
Hi there, that was fantastic, thank you, who knew it was so complicated.
The last time I looked at some of this with NXP free scale parts, we were using GStreamer
with V4L sources coming into it, and a lot of the headache was that there was loads of
buffer copying all over the place, and there were different memory maps and different access
for different components to different memory maps.
So with what you're explaining here, typical use case might be, we do this image processing,
then I want to encode it with H.264265, maybe I want to push it into a GPU to do some kind
of image analysis with AI machine learning techniques.
Could you say something about how that hangs together with buffer copying and so forth?
So basically nowadays the V4L2 framework has great support for DMA buff, which is a technology
used for buffer sharing across different devices.
So with that driver you could absolutely reuse the same memory where the ISP is producing
the picture and use that as the source for an encoder or even the GPU, because DRM also
supports DMA buff pretty well.
So you could do all of that with zero copy, that's definitely all supported, and I didn't
have to do anything special to have that work, it's just the V4L2 framework has that now.
So unless your hardware has weird constraints like the GPU can access this part of memory
or things like that, which are not really well represented currently, but in the general
case it should work pretty well.
So yeah, basically when we have an encoder driver for these all-winner platforms we will
definitely be able to directly import ISP output to encoder input and no copy and low
latency.
Yeah.
So.
Anyone else?
Yeah, thanks for your talk and for supporting, hopefully, more mainline Linux so we have
more phones available.
I have a question about the support for artificial network declarators.
Do you have any idea if this is somehow integrated into the kernel stack in this way?
I mean, it's a lot of work like this as is, but well.
Yeah, so the AI accelerator stuff, that's not really the same scope as the camera stuff,
but that is definitely moving forward.
There is an axle subsystem that was added to the kernel quite recently, which is based
of DRM for some aspects.
And I think more and more drivers are being contributed towards that, so the main issue
currently with that would be that the compilers to compile the models into the hardware representation
are typically non-free and probably going to remain so in a number of cases.
So feel free to push for free compilers for these models to your hardware provider or
whatever.
Any more questions?
You mentioned patches for the camera for the ISP.
Could you point them to me?
Sorry?
Could you point me to the patches you mentioned, and do you have plenty of work on the camera?
It's Adam Pig, right?
Sorry?
It's Adam Pig's.
That's just for the CISAs receiver as far as I'm aware, just not for the ISP.
Maybe I went a bit fast over that.
So it's actually patches on the driver, the SunSix ICSI driver side to implement things
that Leap Camera expects.
So I think you know the one thing I'm talking about.
So do you plan to work on the ISP support, the Leap Camera?
So personally, I would be very happy to do so, so we're just looking for someone to fund
that effort.
So if, you know, someone with lots of money and interest, please come and talk to us.
No, but seriously, I know that people would definitely be interested in that, so it's
good to spread the word that we are available to do that.
We just need someone interested and serious about funding this, but we would definitely
be very happy to do it.
So yeah.
Okay.
Cool.
Thank you for a great talk.
And that's the end of the question.
Thank you.