Happy? Hello, everyone. Thanks for attending. We were doing okay. Is that my device or
I don't know if that's me or okay. So today's talk is written by Yakubo Mondi. Unfortunately,
he couldn't attend today. He's his back. So I'm stepping in. So what I'm talking about today is
not work I've done. It's about his experiences working on the Python Pro. I don't want to touch
you now. So my name is Kiran. Just like Yakubo, I'm an embedded camera engineer with ideas on
board. We've been working on VFRL2 kernel drivers and for some time now, lib camera. We can be
found on IRC matrix. Anyway, you want to get a hold of us at GitHub if you need or after the
chat. And today we want to talk about how we perceive the Linux camera stack on both desktop
and mobile environments. And starting with the kernel, we see lib camera as being a big part
of that to support the platform abstractions. And on top of lib camera, we see lots of
applications desiring to use pipe wire. So we're going to look through there. And the overall
goal is that applications shouldn't care what platform they're running on. They shouldn't care
if they're running on a PC, a desktop, or a Libon 5 or a Pinephone Pro. And equally, any
application you want to run or camera framework, they should all be able to say, hey, I want to
talk to the camera. Give me some pictures, please. And specifically today's talk is about the
Pinephone Pro. Jacobo has spent some time over the last three months or so, or more, trying to
make sure that we can bring up the Pinephone Pro with lib camera and standard applications. And
the Pinephone Pro is an interesting device because it's, I think it's promoted as like a test
ground. So it's like there's no official software, but it's a good device that people can play with
and develop their own software. Interestingly for us, it has an RK3399, which is a chip that has
an ISP. And it's actually a device that we have already been supporting for several years now.
It pretty much was one of the first devices we started sporting with lib camera. And part of why
we created lib camera is because cameras got complex. This is a slide I presented a few times.
But on the left, we can see that beyond having just a single video node where you might say,
UBC, give me some pictures, cameras started getting more complicated. They have multiple
components and you want to configure them. And the one on the left has now been removed from
the kernel. And the N900, which already has a lot of different nodes, that's 13 years ago. So if
you can imagine, there's a lot of cameras out there now that are even more complicated than
all the components there, particularly lots of crawl components. And with all those new components,
it's very difficult for applications to know what to do with each of those things. Suddenly every
application has to be aware of every platform. And that's going to lead to a lot of replication of
code. Each camera application is going to have to deal with media controller to configure the
pipeline, has to talk to V for L2 to get frames, has to talk to sub devices to configure parameters
on the sensor itself. And that changes for every platform. It's different on a Rockchip,
it's different on Raspberry Pi, it's different on an Intel. So lib camera's goal really is to fill
the gap of that abstraction so that applications only have to look at one API again. And it sits
on top of V for L2, it's not a replacement for V for L2. But what we have is a pipeline handler,
which deals with the platform abstraction. And we have a component called the IPA. And that's
crucial for devices like the Pinephone Pro, with an ISP and raw Bayer sensors, because you need
control algorithms. And the IPA in lib camera provides the space to do that. On top of lib
camera itself, we have a native lib camera API, which is C++. We've got Python bindings,
there's people developing Rust bindings. The Rust bindings are actually giving us C bindings,
I believe. Aside from that, we've got Android HAL integration, which is important and comes up
later. And integration into frameworks like G-Streamer. And as I said, the Rockchip for
LK3399 is one of the devices we started supporting when we started lib camera, particularly on the
Chromebook, Chrome tab. But it's actually a really interesting platform because it's in a lot of
small-ball computers as well. So it's readily available hardware, you can plug in off-the-shelf
cameras from Raspberry Pi and play with it. And what I actually really like is recently we've
been working on the IMX8M+, which has the same ISP core in the chip. So the same code that we've
written for Rockchip also works on the IMX8M+. So I've mentioned that these cameras are now
complex and we've got this thing called an ISP, which is kind of getting in the way of people
getting images out of their cameras. And the reason for that is the cameras themselves are now raw
biosensors. And that needs a lot more processing and support to get good images from, particularly
the underlying format is in a Bayer format, which most applications don't want to process.
So that data is fed into the ISP, but the ISP needs to be managed. It produces something called,
well, it produces statistics, usually custom to each platform. And there has to be code or an
algorithm to process those statistics to then generate control parameters to configure the ISP
for the next frame. And ultimately, then that will process in a loop and produce you some images
that the applications will expect, either YUV or RGB. And we already have an implementation
for this. This is one of the things we started early. I believe a lot of this implementation
is derived from Raspberry Pi, so it's quite compatible with the implementation that Raspberry
Pi have at the moment. But we've got various components, like AGC, to handle how bright
the image is automatically or set manually. White balance is important, then lens shading
and the kind of three that you have to start with. But all that code is open and already
existing in Live Camera. The kernel driver itself has been in Mainline Kernel now since,
I believe, 2020. And it was destaged in 21. Helen from Collabra was working on that. And
since then, it's still had active development. There's fixes that go up. And we've been working
on it to extend support for the IMX8M+. And so the kernel side and the Live Camera side is looking
pretty good. We've got support for processing the images. We've got the kernel drivers.
But when we go back to the Pinephone Pro, for quite a long time, there's no driver in the
Mainline Kernel for the front camera, 8858. And even though there was a driver for the back camera,
it wasn't tuned and it wasn't supported very well. It wasn't tuned, really. So Pinephone Pro has been
left behind from Live Camera for quite some time because no one was actively working on this.
And it just meant that you couldn't use Live Camera on a Pinephone Pro. And then Yacobo
has been working on this in collaboration with others who wanted to push this forward and make
it work again. And Nicholas Roth started this back in October, I think, where he wanted to get
Wade Road running on a Pinephone Pro. So he was trying to find out what the missing piece is,
what do we need to up three. And this talk really derived from the work that he kick-started.
So he submitted support for the rear camera, front camera, to Live Camera. And
he based that on the kernel driver that was in the Pinephone Pro
so self-hosted driver, not self-hosted, Meggy's tree. And interestingly, the driver was,
it hadn't been posted upstream, so it hadn't had any kind of review process.
And it exposed itself as a name with M00F underscore over 8858. So it was encoding
properties in the sensor name about where it is and its location. And that's not very good for
Live Camera because it's not generic because then we can't have a handle that says
only match the front camera in location zero when we want that to support every device that has
the sensor. So the upstreaming process actually highlights where things need to be cleaned up.
This has gone through some iterations. And Yakobo, who would have been talking, has taken this on to
completion and it will land in 6.3. It's accepted in the next media tree now. So that's getting in
March, I think. The support required for Live Camera, we moved that and made a release last week
for 004. So now we've got a kernel with the ISP driver. We've got the sensor drivers and Live
Camera support all upstream and mainline. The other sensor needs a lot of work still.
Interestingly, it's supported by Raspberry Pi and the Raspberry Pi kernel has a lot of downstream
patches. So if anyone wants to get involved, this is a really good opportunity to look
at what is in the Raspberry Pi tree, take some of those cleanups, get them suitable for mainline
and post them up. So if anyone's there, what's next to make it good? There's lots of patches
to upstream still for the 258. The next stages really are about camera tuning. And that's part
of the process that we're trying to provide in Live Camera as a framework. We've got a camera
tuning tool and that's really about helping the control loops know how to process the images. So
we have a camera tuning tool which is being developed and can be used already. You can tune
the cameras at home, simple things like taking pictures of a white wall. Ideally, you want a
color card that was on one of the earlier slides. But with very inexpensive tools, you can do some
pretty basic, an initial start at camera tuning. So if you've got devices and you want to investigate
this, that's a great place to get started. So that is not my work here is done, but it's Yacopos.
That's front and back cameras from running on the device. This is captured using CAM,
which is just a pure test tool in the camera. We have CAM and QCAM. They're not meant for end
users really. It's just for helping us develop the framework. The images probably need more work
on the lens shading and white balance, but that's part of the tuning process that we mentioned.
But users don't want to stop using test tools. So what's also been going on and progressing
nicely is support for application layers on top. And Robert Mader, I met back in Prague and since
then has been also working on this with his device, trying to get the desktop environment to be
suitable of the same experience you get on the desktop to work on mobile. And that's
been building up the camera portal in PipeWire, extending support in GStreamer to handle controls
and mapping that all through the camera portals. So from PipeWire's perspective, this is what the
MediaStack looks like for desktop environments or anything using PipeWire, where PipeWire sits on
top of LibCamera, knows how to look at cameras that are VFRil2 as well. But if it needs LibCamera,
it already has that integration. And then GStreamer and applications can sit on top of PipeWire.
And Robert has been doing quite a lot of work trying to clean up and finish that
integration of that application pipeline, particularly in getting the Nome Camera app
to work all the way through. And I remember when I first saw the Nome Camera app, I saw it and
thought, great, there's a standard design for a desktop. I want this to work on LibCamera. So
this has made me really happy seeing that people have pushed this forward. The Nome Camera is a
design, I've forgotten which team was designing it, but then James Westman took that design and
created an application for it, which can be part of the standard Nome environment,
and also run on mobile devices, which is the key point.
If I could have put that in the slide.
So, yeah, I couldn't be here today, but he did manage to record on his PinePhone Pro with
a screen grab and encode on the device, running Nome Camera on the PinePhone Pro,
running through pipe wire into LibCamera, running LibCamera algorithms and through the ISP. So
this is hardware accelerated camera. He's taken a picture. He, in a moment, will change the camera
to the front camera. There. And one of the things I like is, quite interestingly,
you can see the algorithms kick in. So you can see it starts out green and then it corrects
itself. So you can see that real time action from the algorithms that are in place in the camera.
In consumer devices, that still happens on a UVC webcam, but usually you hide those frames.
In the camera, with these up here, we're just not hiding them, so you can see it.
Excellent.
So we can have real live demos instead of video ones.
Can I get back? So, thank you. That demo was running Nome Camera through the pipe wire camera portal.
And thanks to Robert, if you have a device running pipe wire, you can install this flat
pack, get that application and run it on your device. I believe that will just all work.
The instruction from Robert over there. Okay. I went with what I had.
So that's great. We can now run camera application that's exactly the same on desktop and mobile.
But there's more that we want to do on our phones or with communications nowadays.
So getting browser support is really important there. But now we've got pipe wire integration
and portals. Browsers, which will most of them use WebRTC, would be really helpful if we had
integration of WebRTC that could talk to pipe wire. And Michael from Pengu Tronix has been
working on that. There we are. I want to see you later. Has been working on that tirelessly
for a year or more, I think. And I don't think I can point that far out. But what was fantastic
is last week, it went green and that part is merged. There's still a few more series to get in.
But the core support is now there. So in some months, that's a very wishy number. We should
be able to see browsers able to handle this pipeline and you can make a video call on your
Pinephone Pro, which is fantastic. It's not me. It's other people. I do talk a lot about other
people's work, so I don't want to take credit. Talking to some of the distros, I know that
even once the code is ready, I believe we can start getting early PPA-type packages available
so that we don't have to wait for it to filter through all the upstream processes. But that's
up to the distros, not me. So there's actually, it's quite exciting. There's a lot of development
going on with lib camera at the moment. Lots of application-side development. And another one
that's been being supported lately since the last three or four months is Adam Piggs is working on
Sailfish OS. And he had an application called Harbor Camera that ran there and he's ported
that to run on lib camera. So he's been working on that on the Pinephone and it's QT-based.
So I've been running it on my desktop, my laptop and an Intel device. But even though he's developing
it on the Pinephone, because it's based on lib camera, the platform is abstracted so it will
work on every platform that's supported by lib camera. So the same application is now going to
run just the same as known camera will run on all the supported platforms. I'm modified,
which is brilliant. I've run it on, as I said, on the surface goes and my desktop.
He has already started plumbing in manual controls so you can do things that you expect from your
mobile phone camera app to control the exposure and brightness that you might want to play around
with. And then autofocus and manual focus should be coming up soon.
That is a screen capture from Raphael, who is one of the other lib camera community members,
who is just testing Adam's work on the Pinephone. But that's actually running on MimoLest,
whereas Adam's working directly on Selfish. So again, it's nice to see that the distribution
doesn't matter. It's all about cross-platform. A lot of this work here started because Nicholas
Roth was trying to get way droid support working. He wanted to run way droid to get the Android
applications running on his phone. And so that is a way of running an Android environment in a
contained, in a containerized solution on your device, on a regular Linux system, such as we
can now run on the phones. And I said earlier that lib camera already provides an Android camera
how. So we've already got integration there for telling Android how to talk to the camera,
which is great. So that can be reused. There's still a fair bit of work there, unfortunately.
Nicholas has got it working in way droid. He can capture frames. But due to the format that
the buffers are captured in, he can't display them. So that's going to need some more work in
Mesa. And look at how the buffer management is being handled. There may be some updates from
Excellent. So there's recent developments that may improve this in the near future
with panfrost driver development. But might be an opportunity if anyone wants to get dug into
those layers to work on. Sorry, Millie Pixels is a fork from Dorota, where she's been supporting
the Libre 5. And the interesting part there is she's working on a GPU based ISP. So on devices
where we don't have support for managing the ISP, such as Qualcomm or devices that don't have one,
that work would be really interesting to see extend lib camera to work on those devices.
And Pavel's talk earlier was about sharp photos on a mobile phone who's creating a software,
CPU based implementation, which will help get things started as well.
These are lessons Jakob wanted to highlight that he's learned. But they do apply widely.
Fragmentation, when it's all split with lots of different stacks from vendors,
it's very difficult to use that generically. So lib camera's goal is to try and pull this all
together. And to do that, it needs mainlining. We have to have a single definition of what is true.
And mainlining is difficult. It takes a lot of effort. My friend that I traveled up on the
train with to FOSDEM, he was saying he posted patches to one of the Linux lists. And after
four, six weeks, he had no reply and he found that really demotivating.
So it is important that you consider mainlining from the start. You've got to get in so early.
It takes a lot of time. And it's always slower than when you've got control of your own repositories.
We've definitely learned more lessons from developing lib camera. A lot of us are derived
from kernel developers. So we've been on the other side. And now we're seeing just how important
it is on the user space side that these value or controls and interfaces need to be standardized
and have a reference implementation. Since we've created lib camera, we're finding that the sensor
drivers in Linux are improving rapidly, we hope. We've started saying that controls have to be
defined by the sensors. There's so many missing parts to the drivers that are already upstream
and they need more work to get them supported generically. But doing so means it will all be
more consistent and improve the experience for everyone. Thank you. I think I might have two
minutes. Excellent. So two minutes if you do have any questions.
It already does. I think there may be.
Let's just repeat the question. The question was will lib camera support the original
pine phone? So Pavel is brilliant to answer that. Wait, wait, wait.
And so kernel on original pine phone doesn't have required APIs for lib camera.
So you can either break the lib camera to work anyway or you can fix the kernel
and people are doing both at the moment. It's in development. But Adam Piggs who was doing the
pinhole camera app, he is working on pine phone. So no, on pine phone. So it's one of the active
platforms being used. So I believe so, yes. Any more questions? Earlier you talked about
releasing 0.0.4. What's on your roadmap for an 0.1 release? So what do you want to get done?
I'm sorry. I couldn't hear the question. For lib camera 1.0.
He's ducking. He's getting out of the way. We want to, there's key features that we want to
support in lib camera and it will break the API. So we already know exactly how we want to break
things. I started tagging releases so that we had defined points before Christmas. So
trying to do it every two months at the moment. It is in the plan. There's a big reconfiguration
API that wanted to try and get in first before we go for 1.0. But we need testing and app
to know how to get it right. Once we go 1.0, it feels like we're going to say that's it. But
I think some of it's more psychological. But that's versioning. So we're working on it. I'm
trying to make sure we handle ABI breakages automatically now. So we'll be able to improve
on release management. We kind of always just said use the latest because we were trying to
iterate so fast. And that is hopefully improving. So we're working on it. I'm out of time. Thank
you. Thank you. Thank you for a great talk.