So thank you all for coming to the Fossil Mobile Dev Room.
Happy to see that the room is quite full.
Somebody sees the sign that the room is full.
Please turn it on the outside.
I have no idea how to do that.
And Luca is here.
He just made a master done post about standing on the stage.
And yeah, I guess we are all excited to see Luca's talk from phone hardware to mobile
Linux.
So please give a round of applause for Luca.
Hi, thank you all for coming.
Yeah, this is my talk from phone hardware to mobile Linux at Fossil 2024.
So about me, my name is Luca Weiss.
I am an Android platform engineer at Fairphone.
But I'm also an open source maintainer and contributor for projects like PostMarketers,
which is a Linux distribution for mobile phones.
OpenRazer and open source driver for Razer products, so keyboards and mice, and a bunch
of Linux kind of stuff.
And you can follow me on Masuda if you want.
So what is this presentation about?
So in order for me to really understand how to port Linux to a phone, I need some understanding
of how the hardware works, because otherwise it's really obscure what it's doing, et cetera.
Here I'm trying to go from the PCB level using Schematics, which are public for F4 and F5.
Since it worked, that's quite useful for me.
Also trying to say how Linux communicates with the different chips on the SSE, which
protocols they use together.
And even though you might not have Schematics for the device you are working on, generally
these phone Schematics are quite similar, because they are based on some reference design.
So even though it might be a bit different, the same concepts generally apply.
So let's look a bit at the printed circuit board.
Also kind of on the left side, you can see, I cannot really show you, pointer nicely,
you can see a kind of a phone PCB.
This is from the F2, which I found.
The picture with the red arrow, or the chip with the red arrow, the big one, is both the
SSE, but also the memory, like the RAM.
On this device, on newer devices, the RAM is combined with also the, for example, the
UFS storage, the internal storage.
This is put on top, called package and package, so they are actually both chips on top of
each other.
It saves PCB space, so you can make it a smaller PCB, and also saves some signaling, or makes
better connectivity between the two chips.
And the chips itself, they are kind of these big array of pins.
It's called ball grid array, because they are little solder balls.
And these are like a thousand pins or something, and they all connect to the PCB, and that's
kind of where everything's routed to.
Yes.
So then if we go kind of inside the SSE, so this is the one chip that you saw.
Here you can see a so-called decap chip.
Here the Snapdragon 845.
And you can see only kind of the green area here, is actually the software Linux is running
on.
These are the, in this example, 8 ARM cores, and they are running kind of the Linux you
interact with.
But as you can see, there's a lot of other stuff on the SSE.
Stuff like the modem, is a separate chip.
There's a GPU, there's DSPs, et cetera.
There's a lot of other things also running on the SSE.
Most of them are booted up at runtime, so Linux is actually loading the firmware into RAM,
and then you're booting these chips using some proprietary firmware.
So, yeah, the firmware there is not for Linux, but for the separate chips on the device.
Unfortunately for most devices, at least for the hacker community, unfortunately, it is,
you cannot replace this firmware, because it's signed using private keys that are, where
the public key of that key is burned into the SSE, so you cannot change this firmware.
So only the manufacturer can change firmware, but in this way, kind of the manufacturer
can also ensure that only trusted firmware is running on a device this way.
So, yeah, all of these firmers, they're basically just executable files that get loaded.
In this example here, for the ADSP firmware, it's just in the hexagon architecture, so
it's just a different CPU architecture.
But modem chips is also, I think, there's some firmware for different CPU architectures.
I think 10-silic or something, I've heard, we just used for the video firmware.
So yeah, now we go a bit into the software side.
So kind of you saw that there's a lot of stuff on the SSE, but we also need some way to address
this, because otherwise we can just run CPU instructions and they don't do much as in
communicating with anything else.
And so MMIO gets used for this.
It's called memory mapped IO.
This is kind of, I believe, what most or all ARM chips use, probably also on a lot of other
CPU architectures.
And you can kind of imagine this, so I guess you're probably all familiar with RAM, so
you have some address space and you can write to it and you can read from it again.
But kind of with MMIO, on different addresses, you can talk with different functionality.
So for example, on some address, for example here on the address 0x100000, there's the
GCC, the global clock controller.
So you write to some bits there and then you activate some clock on SSE.
Yeah, yeah, so just by writing to some addresses and reading from some addresses, you can kind
of talk with the rest of the SSE.
On the right side, you can see the same representation in the device tree.
So this is the thing that Linux uses to let the kernel know, or not just Linux, but to
let the kernel know kind of where it can talk with the different components and which drivers
it can use for this.
So you have a bunch of different nodes for the different addresses which kind of defines
like what component is present at this address.
And this needs to be statistically defined in the device tree, which is kind of similar
to the ACPI firmware on x86.
So there it's just kind of a bit differently included for the kernel, but it's included
with the kind of the bias that you get on the phone.
Yeah, just to clarify the picture, like the colorful picture on the left side, just for
illustration, the other two values should match kind of.
Yeah, in the PROC IOM, I'm output you can see, so where you can see the different memory
regions.
You can also see at address 8 with a lot of series to 2.7 FFF.
It's kind of the system RAM, so it's actually the address space for the 8 gigabytes of RAM
that are on the device.
So if you're right there, you can read the same value back just as you would expect from
a RAM.
Don't ask me how this MMIO stuff works.
I attended the Free Silicon conference last year.
They know kind of how this stuff works.
I have no clue.
So talk to them if you want to know more.
So you saw two slides before with the BGA, so the many pins on the device or on the SOC.
Even though it's a lot of pins, so it's over a thousand pins or something on kind of modern
or, yeah, modern SOCs.
Of those, most are used for ground or power, so really just not usable for anything like
more than ground and power.
So on the chipset for the FF5, you have 176 kind of GPOs or useful pins, which you can
use for do-do stuff.
Even though it might sound like a lot, it's really not much.
So if you have, for example, eight I2C buses and eight HPI buses and eight UR buses on
the SOC with separate pins each, you already have 80 pins used.
And you still want to do a lot more with an SOC.
So there's normally a lot of different functionality behind each separate pin on the SOC, which
you can switch to in software.
This enables flexibility from both the SOC manufacturer, because they can produce one
SOC that can be used for a lot of different use cases, but also on the other side for
customers buying this chip, they can use the same chip for also a lot of different use
cases.
So they can, for example, if they actually need eight I2C buses, they can actually use
them and kind of sacrifice some other functionality by switching to it in software.
So this is called pin control in Linux, also called TLMM for Qualcomm devices.
So you can kind of switch between different functions.
So for example, between I2C, SPI and UR, you can switch to a different function.
And you get behind the pin, I would say, there's a hardware controller that can actually talk
this protocol natively and can do it very power efficiently.
On Qualcomm devices, there's up to nine different functions per pin, but most pins really have
something like two to four functions, and this is just a part of the hardware that you
get.
In theory, for most pins, you could actually kind of bit bank a protocol on there.
So for example, using a software driver for I2C can just have a loop that sets the pins
low and high all the time.
This generally works for I2C.
It's also used in some devices, but it is prone to bad timing, for example, or it's also
just using CPU cycles to do something that's really something dedicated hardware component
can do way more efficiently.
On the right side, I grabbed this illustration from the Raspberry Pi documentation.
It kind of shows that on the one GPIO that you see, and there's a lot of different functionality
behind it.
So for example, in software, you can enable a pull-up regulator, which is useful for I2C
buses, or pull-down, or a lot of different interrupt detection mechanisms.
So you can just connect a GPIO to an interrupt line from a different chip and then get an
interrupt in there.
And the I2C can tell you very efficiently how this works.
The bottom left is an excerpt from the device tree.
So we can see here, for example, the GPIO155 is just configured to be in the function GPIO,
so just normal high-low.
So drive strength of two milliampers and bias pull-ups.
Bias pull-up here means, for example, the pull-up regulator gets enabled, and you don't
have to do this in hardware.
You can actually use the one built into the SoC.
For the second example, there's the I2C buses.
So for example, for GPIO24 and 25 on this SoC, for the function called QUP06, which is Qualcomm
thing, Qualcomm talk, you can get an I2C bus there.
And you need to configure this in software because otherwise you just kind of get the
default pin controller set, which is either set by the boot load or just kind of the default
GPIO.
So if you want to talk SPI, you need to set this correctly.
Otherwise, the communication from inside the SoC doesn't go anywhere.
So kind of now we know how the pins on SoC works.
So here's an example of, for example, how the speaker amplifier is connected to the
SoC.
So there's a physical connection between the SoC on GPIO8 and 9.
Nothing with microphones falling down.
Sorry.
Try to fix it again.
OK.
Perfect.
Yeah.
So, and this is connected to the SCL and SDA pins.
So for I2C on this AW88261 chip, which is just a speaker amplifier.
This is used for the speaker connection, so how you output audio on this device.
It is used for controlling this amplifier, so kind of telling it some information that
it needs to know to be able to properly amplify the audio that's coming.
For the audio data, we'll be on the next slide.
So here, also for GPIO8 and 9, you can see in the pin control driver, which is the first
top screen on the left, or with the black background.
Here you can see from the pin control driver, that's for GPIO8 and 9, there's a separate
function defined.
So then in the device tree, you can use this function.
So QUP02, which is, again, Qualcomm Dock for I2C here.
And on the other side, you can see on the I2C pin, or on the I2C bus, there's this on
this AWINNIC chip defined.
So the chip on the hardware has its own node in the device tree, which has, for example,
here the I2C address defined, which you can find in the datasheet of the device, or if
you're plotting downstream Linux, then you find this in the downstream device tree, which
is probably most commonly what you use.
And also, kind of knowing that GPIO8 and 9 here is the thing called I2C2 in this example.
This is also, if you work from just the hardware design and datasheets, and you find in the
datasheet, but these are most of the time confidential, so you probably just look at
the downstream sources if you're plotting main Linux for it.
So the speaker gets controlled via I2C, which is a very common protocol for the actual sound
data that you send from the SoC to the amplifier.
So it gets to the speaker.
You actually use a protocol called I2S, Inter-IC Sound, sometimes also known as M-I2S, which
is multiple I2S, so where you have multiple channels.
And you can see here for GPIO150 to 153 are connected to the I2S pins on the amplifier
chip, and this is how the audio data then gets actually transmitted between the two
different chips.
Again, you need to configure the pins correctly, so you need to select the I2S functions on
these GPIO150 to 153.
The example might be a bit confusing because actually in the device tree you set GPIO6,
7, 8 and 9.
This is a bit also because it's not part of the same ping control thing.
It's a bit complicated.
Some probably going to skip this.
But you also need to, because you've set the I2S output correctly, you still need to
send the audio data to the correct I2S bus because the SoC has, I guess, five I2S buses.
In this case the Qwinary one is used, so primary, secondary, quaternary, etc.
And Qwinary, so the fifth one.
You need to configure this, and then you can really set the sound device tree configuration
to send data from the, here in this, for example, it's called Q6AFE die from the Qwinary basically
to the amplifier.
And then Linux knows that it actually should use the Qwinary connection, so GPIO150 to
153, it's sending it to the amplifier, and then Linux also knows that it needs to configure
the amplifier correctly to be able to output the audio correctly to the speaker.
Here also the I2S interface is actually handled by Linux itself, but by the ADSP, which is
the audio digital signal processor.
It's also a thing on Qualcomm, probably also other SoCs, where the main SoC can go into
a lower power state if it's just playing audio because then the separate chip, which is way
more efficient, can do this correctly.
For microphone it looks completely different because why have the same thing for both audio
output and input.
Here it's going via the WCD9385 chip, which is a Qualcomm chip, which is kind of an audio
collector.
It's connected via sound wire, which is a completely different protocol.
It's a new standard from around 2014.
In comparison, I2S from the previous slide is from 1986, so it's quite an old standard.
It's also a slim bus used on some SoCs, or actually still on the same SoC, but for different
use cases again, which is standard from 2007.
In this case, again, as you can imagine, there will be some connection path between the SoC
and the amplifier chip, and then from the audio codec to the microphone to get microphone
input.
You can see here is a bunch of pins, cheaper, 144 to 158, and sorry, 149, and then 158 also.
They're connected there.
It's a protocol's sound wire, and the other will be just a kind of a physical connection
between the audio codec and the microphone where it can get the data there.
This WCD chip is used for three different microphones that are on the phone itself,
and also for USB-C audio, where you can actually get an analog audio signal via the USB-C connector
out, and here both output and input is handled via this chip.
Yeah, so the microphone also needs to be configured.
It works a bit differently here because it's a sound wire device, so it's different.
But you have these two sound wire devices on the left side.
For some reason, they have one for receiving and one for transmitting the data, and then
you put them together on the right side as the kind of, again, you have a node for the
WCD audio codec because it's a chip on the SoC that you have.
From this, you reference it, and then you also, again, need to configure the audio path
from the, or basically the other way around from the audio codec, so from the WCD chip
back to the SoC so it can receive the microphone data.
USB-C is quite a cool thing.
In the past, micro USB only had kind of the functionality for power, so to charge the
phone data, to transmit a bit of data, some pictures behind it, back and forth, and also
OTG functionality, to, for example, plug in a keyboard, but it was not really used for
much.
USB-C can do a lot more.
On the FFVV, you can use the connector for USB 2.0, so USB high speed, USB 3.0, which
is called super speed, analog audio out via this thing called the audio adapter accessory
mode, which is a standard part of USB Type-C, and also display port out via the display
port alternate mode.
So yeah, on the 24 pins, you can do a lot of different things, but for actually doing
this, you don't just need the SoC and the USB-C connector, but also some other components
that actually switch to different components based on kind of what the user plugs in.
So for example, for actually getting analog audio data out of the USB-C connector, so
you can just connect USB-C headphones that don't have a, where the data is not transmitted
digitally, but just like analog, just like with a headphone jack connector.
So actually the analog signals, they go over the USB 2.0 data pins, so kind of the D plus
and the D minus for the right channel and left channel, and also the two sideband use
pairs, which is a part of the USB-C connector.
They are used for microphone and ground, or ground and microphone, depending which headphones
you are in, because there's two different standards.
And in order that audio data here flows again, you actually need to configure the different
chips to handle the data correctly.
So I've tried to kind of put in a reasonably understandable format.
On the top, you can see the WCD codecs, or the sound wire codec that went before.
It has the headphone jack output and microphone input.
This is routed to a chip called the OCP 960096011, and this is some switching between USB and
audio, depending on the use case, and this send this router to the USB-C connector, so
you can actually get the data out there.
In order to do this, kind of Linux needs to tell this OCP chip, this audio switch, to
actually switch between either USB or the analog audio, depending on what you plug in.
And this is here called the mode switch in Linux.
So in Linux, in the device stream, you need to connect them together with some USB-C nodes.
There's documentation for all of this, which might not be the best, but if you understand
it some fun, you may be understanding.
For this report alternate mode, it's way more complicated, because you can put, there's
a lot of different combinations that you can put on the USB-C connector, so some of the
super speed pins you can reuse for this report, either just one line or two lines or four
lines, and for four lines, you actually cannot do USB 3.0 anymore, because all of the USB
super speed lines are used for this report, but you can still use USB 2.0 for this.
So there's a lot of different combinations that you can use.
The DisplayPort aux channel, which is used for some communication also, runs over some
different wires, which also needs to be switched correctly, depending on which orientation you
plug in the DisplayPort connector, or the USB-C connector, basically.
And everything needs to be configured, and if something is not configured correctly,
no data will flow, so then you have a bad time, because you don't have DisplayPort out.
So kind of device tree, because this talk has been about hardware and device tree, device
tree actually represents hardware.
It doesn't represent software concepts, you're not telling Linux to do something, but you're
providing Linux with the information that the hardware is made in that way, and then
Linux can use this information to actually do the correct things.
So keep this in mind when writing device tree or the bindings or the commit messages.
Don't write about, hey, this is telling Linux to do this, but you're actually telling, like,
you're describing the hardware.
Yeah, if you're adding a new chip, all of the power supplies and GPOs and everything should
be included, because, yeah, it represents the hardware.
Even if Linux doesn't use them yet, you should include them in the bindings, because the bindings
represent the hardware.
Device tree is also operating system independent, which is really cool, because many, many years
ago you wrote kind of really Linux specific codes to make the hardware work, but device
tree is operating system independent, so you can use the same device tree, for example,
UBoot, which is Caleb's talk, which is coming up in a few minutes, but also, for example,
for FreeBSD or whatever.
They should be able to use them, because it represents the hardware, it doesn't represent
Linux specific things.
So, yeah, so, what do you want to say?
Right.
It's a great ending.
Yeah, don't put any specific things in there.
They should work if the other operating systems have driver for it, of course.
And that's it.
Thank you.
Thank you very much, Luca.
We have, like, two minutes for questions, so if somebody has a question, please ask
now.
I will give you the mic.
Thank you for the presentation.
Can you tell us if you pick the Fairphone 5 now, or maybe the 4-1, like, and you actually
rent a vanilla Linux channel on it, what can actually work?
So you spoke about headphones, mic, things like this.
What's the actual state of the Fairphone on vanilla Linux?
And also, next question, obviously, is, like, any plans to make it even more, like, open
in the future?
So, right now, like, completely vanilla Linux.
A good chunk of stuff is working, but some stuff like displays are not working yet.
But I have some patches which, like, for displays, still depends on some hack that actually
Marainas is working on, hopefully.
Yeah, you can use display, GPU, DisplayPort out over USB-C.
You can use Modem.
You can make inferior phone calls without audio, because audio is not working yet.
So basically, the big things that are not working yet is audio is not working yet, camera
is not working yet, and probably quite a few more things that you want from a phone.
But kind of, you can use it as a kind of media machine if you connect Bluetooth headphones,
you can stream via Wi-Fi or 4G or 5G and everything else.
One more question?
Yeah.
Or maybe a quick plan for the future.
I definitely plan on working more on this, but that's also not my full-time job to working
this.
I have one day a week currently for this.
So I can definitely get way more done than without more time.
But yeah, we will see.
Could you please describe in a few words how to debug device 3?
The second?
How do you debug device 3?
I mean, to some degree, I'm not supposed to debug device 3, because it's just a data-formatted
space-based hardware.
Of course, when bringing up hardware, you need to figure out what values to put in there.
This is most in combination with the Linux driver then, because your device is just a
data format to describe the hardware.
So you need something to kind of interpret it and use it that way.
So most of the time, you are debugging Linux drivers and probably fixing Linux drivers
and then at the same time, fixing your device 3.
In the perfect words, you would just write it once and it would be done and then fix
the operating system maybe better.
So that's the time we have.
Another round of applause for Luca, please.
And make sure to check out the stands we have for Linux on mobile in building AW.
And now we have five minutes of break.
Don't forget the mic.
Yeah.
And yeah.
Thanks.
Bye.
Bye.
Bye.
Bye.
Bye.
Bye.
Bye.