Okay, so hello, I'm Pavel Majek and I'm here to talk about cameras, but you can also talk
to me about clicker train horses, mobile phones, kernel, smartwatch by based on ESP32, Mobian
on my molester. So first thing first, video following is not for cameras, it's for frame
grabbers and they are really very different, which is basically what this talk will be about.
They can do remote controls, but they cannot do autofocus for you and so on. But the interface is
fairly simple, you just open depth video zero, select format and capture. Unfortunately, what you
get is blurry photo, which will be either all white or all all black. This is with autofocus and
auto something. Anyway, they are phones with smart sensors, one such example is pine phone and
those are pretty close to the frame grabbers. They do basically everything in hardware. This
used to be a pretty common design in the past, which made a good sense at that point because
USB had limited bandwidth and you could not push uncompressed data through it. It's easy to
sanitize, but it doesn't make much sense today. If you have like five lens on your phone, you
don't want to have five JPEG encoders there. So we are moving to dump sensors, which basically do
bare minimum. There you set parameters like exposure, gain, select area and so on. And it just passes
the bare data over the fast bus and it usually ends up in your memory. And then you have component
called ESP, which is image signal processor, which will do the JPEG conversion and such stuff.
Unfortunately, in case of the interesting phones, which is official LibreM5, pine phone and pine
phone pro, we either don't have the processor or we don't have drivers for that, so we can't use it.
So this is how the image, this is a photo if you try to take it without the automatics. Can you
recognize what's there? It's a USB connector. It's recognizable, I'd say. So what do we need to do?
Nokia N900 is another example of complex design, which used to be very important historically. And
actually the photos in the presentation are from N900 with open source stack. In real time, you need
to do auto exposure because otherwise you will have black or white frame and you need auto exposure
for autofocus. On most cameras, you really want autofocus too because you can't just focus to
infinity and expect good image. And that's pretty much everything you need to do for
the video recording in the real time. Then you have preview. Preview is a bit less important than
the video recording, but it's also important. You need to convert from Bayer to RGB. And you
need to do gamma connection because the sensors are linear in one side and exponential on the
other side. GPU can help here. And then there are extensive post-processing steps like auto
white balance, lens shading compensation, getting rid of bad pixels and probably many others I
forgot about. Advantage of this is that this can be done after taking a photo or after recording
the video. And there are quite good tools for that, including raw therapy, Euro and so on. So
people were working, unlike the other parts, this got some work done before. So what we are
talking, for example, on the N900, you have LED flash, which is a completely independent device.
You have voice coil support for autofocus, which is again a separate device somewhere on
I2C. Then you have two sensors, front and back camera. You have GPIO switch to select which
camera you want. And then you have ISP, which is quite a complex piece of hardware, which will
not be important for this presentation because we will do without it. So tools to use. There's
great set of tools to use, but they have some limitations. One which looks very nice is G-streamer.
And G-streamer is really great if you have an unlimited CPU. Unfortunately, you don't have
unlimited CPU. If I was willing to hack its C code, it would be very powerful, but there's
some learning curve involved in that too. And at the end, G-streamer might be right to use,
but I found other tools easier. There's FFM pack, which has quite nice and very simple
command line interface. So I used it at the end. I didn't really need much. Just please
take these images and compress me every video from there. There's megapixels. Megapixels
is a very nice application focused on mobile phone, very well optimized, but its origin
is a pine phone, and they don't use live camera there. Then there's live camera. Everybody
says live camera is future of video on Linux. It probably is, but there are still many steps
to get there. And there's megapixels. Millipixels is fork of megapixels, which is supported to
live frame 5 and to live camera more importantly. So in many ways, so megapixels actually currently
looks nicer because it is based on newer GTK. On the other hand, millipixels use live camera,
and that's important stuff. Okay, so this will be a bit of history and reasons and so
on. I started to play with camera on pine phone, and first idea was, hey, Gstreamer is
there to capture video. Let's use Gstreamer, right? Okay. I started capturing raw Bayer
data because that's what should be most portable. I did some shell scripting, media control
to set up the pipelines. That's not fun. And then just use Gstreamer to save the Bayer
images to the disk. And I could do 200 kilopixels, which is not great, but better than no video
at all maybe. And I realized that CPU can compress at 70 kilopixels images in real time,
which is, well, people were doing this, but it's sometime ago. So I tried to improve. There's
IUU format the camera could do, which is the Bayer tent converted to like for better processing.
And I could capture up to 0.9 megapixel video with that. And if you were wanted, you could
take a look there. Maybe it's useful for someone. But, well, that was the reason. The reason was
called colorimetry. And someone in Gstreamer decided to do a regression basically. And all the
Gstreamer stuff stopped working. And I realized that, well, perhaps it wasn't good too great to
start with anyway. So I started looking around. Quickly, I found the camera, which is the future,
right? And, well, it's C++. It didn't work at all on pineforms. So I had to do some quite heavy
patching. I get some help on the mailing list. And I realized it has JPEG support, which is,
well, you avoid a lot of stuff, because if JPEGs are already core space converted and compressed
and so on. And I realized that maybe JPEG is worse having second look. So I did. You can't say
data into megapixel resolution to flash, because the flash is not fast enough. But it was like
almost possible. So, hey, JPEGs are four times smaller. Perhaps this could be adjusted. And
saving sound is easy. So maybe we can, well, maybe we already have everything we need. And this is
why how Unixic camera was born. I realized the second reason. Someone decided that placing
Uncache data to user space is fun. And the camera decided that placing Uncache memory up to the
application is great. I thought someone stole my CPU, because the performance penalty is about
10 times. But not. It's just the way it is. I believe this needs to be fixed. If you fight with
the streamer and the performance seems too bad, this is probably why it's too bad. And I don't
know, talk to your kind of person which can change it. By the way, in the old days, we used to have
a read interface to get data from the camera. This is now deprecated. Of course, it is faster to
read the data than to get Uncache memory, right? That's how badly Uncache memory sucks. Anyway,
so Unixic camera started. Audio is really simple. You just create a small C application to sound,
record sound, split it to chunks so you can have easy processing later and timestamp them,
which is important for synchronization. Live camera with some small hacks can write 35 frames per
second to megapixel this data to the file system. All you need to do is edit timestamp and sim
links so your preview can tell you which is the latest image. Very easy. Control application,
you probably don't want to start your video record from command line, but that's also very easy. You
just take some GTK and Python. It creates timestamps, telling you, hey, start recording it now,
and displays preview, which is the most intensive thing there. And this is basically what runs
during the recording, so this is to be determined a bit optimized. Post processing is not that
important, right? So you just use Python and FFM pack to compress the resulting video stream. Easy.
This is something I was pretty happy about. If you want to deprecate it, you will need some setup
like patching clip camera and so on, but code is out there, and there will be easier method in
future. So I like this solution because I could use multiple languages to do my camera recording,
write language for the job. In the end, this was few hundreds of lines of code total. And it could
do some quite interesting stuff. Like you could take still pictures during recording. You simply
copy the GTK one more time. Easy. In video resolution, but if you are recording it at two
megapixels from phone camera, I'd say this is going to be pretty decent picture anyway. You could
take photos with arbitrary delay. Like you could even take photos before the user asked for them
because you are taking all of them anyway, so you just don't delete them. This was fun.
Then I've got access to LibreM 5, which is different in important ways. It has damp
sensors, so it won't give you JPEG. But it had better support. Let camera work there out of the
box. There was megapixel application, as I explained about before, it with patched megapixels,
but it had no auto exposure, auto white balance, or autofocus support. It couldn't report video.
And there's more issues on LibreM 5. Canon could use some work. It only gives you 8-bit data,
which is not really good enough for good photos. You can select one of these three resolutions,
so megapixel, three megapixels, or 13 megapixels, and for some reason only 23.5 frames per second
work. I don't know why. Hardware has face detection autofocus, which is a very cool
sounding toy, and I have to thank Purism for their hardware and for the great work they did on the
process of Verstek. They are heroes. That's the best photo I got with Nokia N900. Some megapixels,
they are very simple application. There's small development teams, so it's easy to work with,
it's plain C, it's easy to mark patches. It does all the processing on the CPU,
which is great if you want to change the processing. So I started to do auto exposure because that's
the most important part, and I did a very simple one. I prototyped on N900 years ago. So basically,
if you have too much, too white pixels, like overexposed, you need to turn it down to exposure,
right? And if you don't have enough white enough pixels, you need to turn the exposure up,
and this is it, and this works well enough. It takes a few seconds to converge, can be improved,
I don't know how to do that, but this is good enough to take photos. Other thing is auto white
balance. This is not that important because you can do it in post processing. Anyway,
they did have manual white balance, so I felt this is easy enough to do. It will need some
more work. Again, if it's too blue, you make it more red. If it's too red, you make it more blue.
That's it, works well enough. And in a few hundred lines of code, I had simple software only,
auto exposure, and I got that merged. Next step is autofocus. Autofocus is something
which deserves more respect because you really want it tuned, but well, if you want to do it
simply, you just start from the infinity. You compute blurriness of each frame, and you only
need to take a look at part of the image if you want to save your CPU, and you start your sweep,
you start to blink the focus closer, and when the image gets more blurry, well, you stop. You
might want to go a little bit back because of the physical issues of the lens, but this works
well better than manual focus, and I got it merged rather quickly. Next step was video, so I decided
that I like the ideas from Unixi Camera, and simply did 0.8 megapixels recording directly to the
disk. I hacked millipixels to save timestamped frames, and left post-processing after the
user presses the stop button. Easy to do, obvious to their disadvantages, right? You are now limited
by the disk space, and maybe you could say it's not quite nice to the flash to just stream raw
data to it, but hey, the flash is cheap and the phone will die anyway. Post-processing is quite
long, it takes five times slower than recording, or I guess this could be optimized. This is again
my old code, so I'm Python with FFmpeg. Ideally, there is hardware to do the encoding, we should
use it, but I feel that doing that is awful lot of work. Anyway, this is now upstream, so if you
update your LibreM5, you should be able to take video off, and I believe it's important to have
something other than video recording. Next thing I want to talk about, which is very exciting,
is face detection autofocus. You may want to Google it for nice explanations, but basically they
have selected some blue pixels, they are special, and they are special in a way that they only
take light from certain day directions. So you have a lens, and if it's focused, it's okay,
the light comes and meets at the sensor, but if you are autofocus, funny set of happens,
and light from the left of the image ends up at different place on the sensor than the light
from the right part of the lens. But if you block the light from the direction on the chip,
which is easy to do, you can use it for focus. So if you take a line from the sensor, and you
have on the top you will have left special pixels, and on the bottom you have right special
pixels, for example, then you will have this. The tree you will see on the line will be at
different positions on different special pixels. Well, and you can use this to focus, right? You
just compute correlation between the two lines, and it directly tells you how much autofocus you
are, and in which direction you should focus. This was great to play with, it was like hacking.
Unfortunately, it is not too usable on LibreM5. They are two issues for the special pixels are
quite far apart, which they basically have to, because if you made all the pictures special,
you would have, you would lose your resolution, and it only works in the high resolution mode,
and you don't want to run your preview in high resolution mode. So if someone is interested
in fade detection autofocus, I have the code, the code is on the, on the GitLab somewhere. It was
fun experiment, it worked, but I decided, like, for real focus, you would probably have to do
hybrid, like do course focus using the fade detection, and then do contrast detection on the,
on the end. It seemed like a lot of work, and with the driver, which would only give you 23
frames per second, and so on. Well, I decided not to take this much. So I have some wish lists,
and I think I have, like, five minutes left. So five minutes talking, or five minutes questions?
Including everything. Including everything. Okay. So I have a long wish list for all of the world.
I would like to have better media control support in the tools, because it just doesn't work.
A piece changed, and the tools didn't catch up. I would like library to get conversions between
formats, and so on. I would like better than 8B support. I would like multiple applications
accessing the camera at the same time. Better support would be nice, and someone should
re-resolve the caching problem, because that's bad. For lip camera, I shouldn't be really hacking
millipixels. I should be hacking lip camera, but lip camera doesn't really support software ISP,
and I'm not a great C++ hacker, so I could do it, but they will reject the patches if I do. So I
would much prefer them to do the preparation, and then I would fill the code. And that's
pretty much it. So time for questions.
Sorry?
We do want your work to software ISP.
The comment is that they want my work on software ISP, and I guess I will want to cooperate, but
lip camera is not easy to hack for me, because of the C++ stuff. So be patient, and I maybe
it would be better if someone else did it.
Yes, so well, there will be not be much to see. So you know, millipixels could use some work too,
but I can take pictures, trust me. I didn't use autofocus for this, because, yes, I can do it.
So it's now upstream, so you can just update the operating system, and you will get one,
and it should be possible to do just a short video recording too, so now you have all been
recorded, and now the CPU is busy converting that.
Okay, so I guess.