Okay, so, last talk of the dev room, Paolo Bonzini is going to be talking about.
Reverse engineering a solar roof panel. Sorry, roof. Yeah, whatever. Well, whatever he said.
Here, yes. So as a quick introduction to myself and to set the expectation straight, I'm not a
hardware guy. I'm not a security guy. This is basically something I did for fun. So I'm a beginner
in these topics. On the other hand, I'm not a total idiot either. I know assembly pretty well.
I work with compilers, kernel stuff, so I know X dumps enough to do this stuff. So anyway,
this all starts almost five years ago when I bought a solar roof for my family. And the installer
asked about having this optional data logging component. And I didn't really want to have
anything cloud related because IoT is short for Internet of Things that shouldn't be on the Internet.
So I didn't want it to be on the Internet, but it's entirely local. So I said, sure,
why not? And this is what I got from them. This is the normal solar roof setup, the stuff that
they want to touch with them football. This is the part that this talk will be about. And
it was already suspicious from the beginning.
But I mean, who knows? Maybe they bought it on sale. I don't know.
But the plot thickened when I by chance had wireshark running and I saw there was
a Raspberry Pi that I didn't know of. I have other Raspberry Pies, but none with this IP address.
So actually, a few years later, when I was not really preparing for this talk,
but I noticed these on their website. So these are the specifications for this.
It has quad core ARM cortex. It even has a microSD inside. Okay, so you know what kind of Raspberry
Pi is going to be in there. But anyway, let's take a step back and go back to 2018 and say,
let's see what this thing does. So it's pretty nice. I mean, you can only see these from outside,
but actually there's extra things around. Here you can see that they power it through the GPIO.
In fact, the power brick, they actually cut out the USB part and they just screwed the
wires to the GPIO here. So what does it do? It basically logs data to the SD card every five
minutes. It lets you plot nice graphs. Unfortunately, I don't have any picture of the graphs because
I don't use the software anymore. But it also has five relays. You can see them here. Some of them
have normally closed, normally open. Some of them only have normally open. And it's based on five
volt inputs, which is a bit weird because usually you use 12 volts or 24 volts DC for communication.
Five volts is a bit weird. And it's not very useful also because you have to actually put
the wires on the wall and it was already hard to find a place for the whole solar roof things
retrofitted into a relatively old house. Another interesting thing is that it has a built-in UPS
because the inverter has like a 10 ampere line that stays up 24-7. If the grid goes away,
it's still battery powered. So this is another incentive to actually reuse the Raspberry Pi for
something else. And also on a slightly lower level, what does it do? It has port 22 open.
Unfortunately, there's no way to upload keys. It has a remote update, but it triggered exactly
once when I plugged it in and then not anymore for four years. The web server is not Apache.
It's not nginx. It's just an embedded web server with a vanilla server header in HTTP.
But it has a nice JSON API. It's easily discoverable with the Firefox developer tools and that's how
I used this thing for some time. So for example, there is the API login. There is the API dash,
which is basically what is used for the dashboard. And it returns the instant data like one minute
old at most with some really cryptic names. Okay, Ver is not cryptic. Temp is not cryptic,
but the other ones are a bit weird. And it also has the possibility to get CSV data with similar
headers for a particular day or average across the whole year. The daily one is the most useful
because if you have a lot of data for the same header, it's easy to figure out what it might be.
So for example, this one, ENPH, okay, V probably stands for voltage, W stands for what, but H
you may not know of hand. But if you look at the number, you can see that pretty clearly that it's
probably the frequency of the grid or something like that because it's about 50. So you get lots
of data. You get voltage, current, power values. You get a few computed fields that are easy to
recognize because the name starts with X. So for example, X home is the current consumption of
the home, independent of whether there is data coming from this, sorry, there is power coming
from the solar panels, or maybe instead it's coming from the battery. The actual data from the
inverter are a bit not suited, for example, to plotting useful graphs, but they have a few
computed fields that put things together. There's also a few dozen flags that are interesting. They
are almost all zero, and so they will be a bit harder to reverse engineer. And also you can see
the five-volt inputs and the relay outputs in the logs. I actually never use these, so they are
always zero, but they're named like in one, in two, in three, in four, in five, so it's pretty
easy to figure out. So what you can do with this is already do simple-minded hacks with curls. So
for example, you can gather all the yearly data and do your own plots. For example, you can see
here is when I do laundry because the weekend and Tuesday is where I consume more power.
And another thing that I did very early on was push data to MQTT to get some nice widgets on
my phone that could give me instant data without opening the web interface. And also for home
automation, at some point I was using these to turn on and off some ZigBee smart plug bugs.
But this was not any reverse engineering at the Raspberry Pi level. It was just looking at JSON
stuff and doing just stuff with curl. But this was already enough to find some interesting bugs.
There are some weird stuff in the logs. So for example, here you can see this probably stands
for day and hour, and this stands for month and seconds, but it's a value that has a decimal
point in it, so it makes no sense. And I will show later what happens. Also, there are some fields
that end with L and H, but sometimes they are swapped. So you can see that L is the one that
stays always the same, and H is the one that keeps going up and down, up and down. And also with
L and H, they didn't really care a lot about them because also if you plot them, the plot looks
like this. So this is the solar production from the total solar production since the day that I
installed it. And this is how much it had produced in January, 2021. This is in December, and it's
really weird. What actually happened is that they put the low as a signed value, but it should have
been unsigned. So whenever it goes from 2 to the 15 minus 1 to the 15, it actually flips back to
minus 32,000, whatever. So you can see that this is exactly 655.36 kilowatt hours. And if you fix it,
it's still wrong because there are some bugs. But I mean, this was not supposed really to be
used by the buyers of the appliance. People were just supposed to look at the nice plots.
Also, this is way more worrisome because there are flags that seem to be passing correctly. If you
look at the logs, it keeps giving these weird names that they keep going on and off like a few times
every day. And the name is very scary. It's battery charge over the current hard limit,
and same for this charge. And the weird thing is that you can see this error in the web interface
all the time, but you never see it on the inverter. The inverter has a little panel with errors,
and you cannot see that any time. So that's weird. And I didn't really like that. I asked
customer support, which were otherwise very nice, but they just said, yeah, don't worry, it's fine.
So all the time while I was doing these things for fun, I had the lingering thought that the
microSD card would die over me because they say diamonds are forever. I don't know if it's true,
but certainly microSD cards are not forever. So even worse, I couldn't just go and say, okay,
it's 10 years warranty, please send me another SD card. Because the newer models, they moved the
storage to cloud, and they only have five years of service, which is suspiciously close to the
time before my own SD card died. So people won't pay for the cloud service. You won't have to repair
the data logger. Anyway, I never did anything about it until last year. Last year, I got the
data logger to shut down on me twice in a week, and I naively thought that it was just a network
problem, and I said, okay, I will just take out the SD card and set up a static address.
Actually, the SD card was dying. I don't know why it really had all these problems in general.
It didn't have any problem until last November, and last November, it really died in a matter
of a week. So I have no idea what happened, but I'm very thankful to the SD card gods for.
So when I open it, this is what it looks like, and this is the behind of Raspberry Pi 3.
It's potato quality, but okay, the first impressions was that it's a standard Raspbian
install, so I guess I can forgive them for not obeying the GPL because they didn't do anything
weird. I mean, it was just Raspbian. And there were two statically linked binaries in OMPI.
One was running on TTY1. Nobody ever noticed that it was running there because there's no
room for an HDMI cable, so it was there, though. The main use of that HomePI, I think, was that
if you plug a keyboard through the USB and you press there, it rebooted, so it probably was
a quick way for them to do some testing. But it had some nice ASCII art of the company logo.
The other one is the one that we care more about. The data is placed in HomePI storage,
and the nice thing is that Strays is installed, so I was already thinking of ways to get some data
from this because with Strays, it's so easy. Anyway, there's basically, again,
in stock install, some files are newer than the others, so that's an easy way to see what's going
on and what they changed. There's a couple system D units. There's an I2C RTC. This is nice because
at the time, I hadn't even looked at the PCB or anything, so I didn't know what was in there.
And knowing that there is a real-time clock on the board is nice to know. They disabled Bluetooth
for no particular reason. Well, there is a reason, but I will say it later. So what I do
is just I copy the binaries to my computer, I add my SSH, and I turn it back on and it works.
So one thing that you need to consider before doing this kind of work is what about the warranty,
especially for something as expensive as the Solar Roof. The thing is the connection to the
inverter is through USB. It's not direct RS485. There is a scooter. This thing is away from the
board, so I didn't touch it. I just removed the screws to take a picture, and that was it.
It's basically RS485 to USB adapter. So also, the inverter has the Baud rate. You can customize it,
and the user manual has basically three choices. Date and time. What was the other one? It was
the total that it reports for the produced energy. So you can reset it to zero if you give it to
another person or for whatever reason you want to change it. And the Baud rate for RS5. So the user
is supposed to know that it is based on RS485 and is supposed to look at RS485 data. I don't think
this is any kind of lawyer analysis, but still I'm pretty sure I didn't break the warranty
on the inverter, which is the really expensive part. So anyway, I got the binaries. The first
thing I do is using strings, and just doing strings by plus is awesome source of information.
And what I noticed by doing quick search, for example, for the API endpoints and for
DevTTY USB, is that all the strings are together, and this means the program is unlikely to be
written in C, because in C they will be null terminated, and they will be all one after another.
So it could be go because, I mean, there's not that many languages that you would use to write
a web server in and that produce a large binary. Certainly, it wouldn't be rushed because it was
four years ago, so it wasn't as fancy as today. And anyway, if you do a read-elf, you can see
some section editors that go sim, tab, go pcl, and tab is basically the go format for the bug
information. So it's almost certainly go, like what? Certainly go. Another thing that you can do
for strings is for GitHub, because why wouldn't people use RPIO libraries from GitHub? And also,
you can find some nice names and things. And this is the name of the model of my inverter,
so I know that what they call it in the source code or in the files can be handy. So anyway,
the thing is, there is running this, I have SSH access, so what I can do is just trace it
and see what happens. And one nice thing is that the T2 opens and closes the DevTTY USB 0 every
minute, so it's pretty easy to also get not just the board rate, but also the parity, the stop bits.
Okay, that's very little, but okay. I will go fast. So with go, the thing is it has an event
loop that can move a subroutine from one thread to another, so you need to track the file descriptor
numbers. So what you get here is something that is basically you can recognize to be Modbus.
This is a read 16-bit register request. This is read one-bit register request. So what I did is
I basically took this from the logs and I put it in a small C program to the code what it was.
I mean, I could probably do something with Wireshark or whatever, but trace gave me already some C
strings, so I put it in a C program. And this is enough, for example, to compare with the CSV files,
at least for the 16-bit registers. So for example, I can see now that these are probably low and high.
So this was the minutes and seconds, the hour, the day, the month, and the year. This is 21 in
hexadecimal. And I can also see that some values are fixed points. So this one is the version
multiplied by 100, and this one is the temperature multiplied by 10. For the discrete inputs,
it's a lot more complicated. I could find in strings some nice names of the fields. So for
example, GRN was no voltage from the grid and so on. It's a bit weird that they put it an alert
that there was a blackout, but they didn't put as an alert that the fan broke, whatever. And this
also doesn't make a lot of sense to average the bulls, but whatever. So anyway, this is already
nice because I have the names corresponding to all the fields, but I don't have the mappings
of the discrete inputs. That's what Modbus calls the one-bit Boolean values to the flag.
So, and I knew that this was the part that was broken in the code. So this was probably not going
to succeed, but actually it was successful. I used radar 2 for this, and I will super, super quickly
go through radar 2. This is what I learned about radar 2 because I had never used it before. So
the commands are one letter per word. So ADF means analyze data in functions. There are some
people here that are old enough to remember Lotus 1, 2, 3. That's the way the menus worked in
Lotus 1, 2, 3. Basically, you had like one letter per word in the command that you wanted to execute.
And the main ones are seek, print, and slash for search. And another interesting thing to know
is that the state of your work is saved in a project, which is actually a single file in the
Git repository with thousands of commands in it that say all the nice things about your binary.
You can get some information from the debug info that I showed earlier. There's a nice command to
do all the analysis that is possible, but it doesn't really work for a static binary-linked
binary. So I use these ones instead. It starts giving some nice information. So for example,
in the project file, after I do the analyze strings commands, I can see that it has these
CS commands. And now when I do the disassembly, it actually prints these as a string and not
as instructions. Likewise, when I do AXD, I can see that it loads these. And it also says what is
the data that is loaded from here. This is probably in the constant pool. So for example,
this instruction is the one that loads the address of dev.tty.usb0. One thing that you can do is also
you can write your own commands and add them to the file. So for example, here, I know that this
location is a data operand for this instruction. And I can tell Rader 2, for everything that is
in here, make it dumped as bytes, not instructions. So after I add all these things to the project
file, it will not be dumped as a rubbish ARM instruction. It will actually print the word.
Then you can also search keys. For example, if I search for the two flags that went up and
down, I can see that they are here. And here, they are closed. So I decided to search for this
address. You have to put it backwards, because it's little Indian. So this is the first byte and
this is the last one. But then I found that they were found relatively close, like 68 bytes
apart. So you dump them and it's very helpful that it even tells you where the hits were from
the previous searches. And here, you can see some nice things. It seems to repeat every
68 bytes. And it's always pointed to the string followed by the length. And also, there's these
nice numbers, which might be maybe the numbers of the discrete inputs, who knows.
And if you go back and back, you can see using the seek command, I can go back 68 bytes at the
time. And sooner or later, I get to a point where the format changes. 68 bytes before,
there's nothing like what was afterwards. So this was the beginning of the array. And now, I know
exactly which discrete input was with which name and so on. I can also tell to print data. So,
for example, this was a floating point number. And if I print it, it's 0.1. So the guess is that
this would also be something related to the fixed point values. And now, it's time to actually find
the pointer to this. I do, I search for this address here. And I find it here. It's also
address followed by the number of entries. So it's probably some kind of array descriptor for go.
And then, I search for the address of the descriptor. And then, if I go here, I see that there is
a reference from the query function of my model of the inverter. So I guess we have a winner.
And in fact, what I did then was rewrite the software using Modbus. It outputs the logs
in exactly the same format. So I still have a continuous log from the date of the installation,
except I didn't fix, I didn't leave it exactly the same. I fixed the bugs. So,
no averages of dates. The scary flags are not logged anymore. I can see now whether it's using
the battery or not. And I even got the grid non-flag during a blackout. So I guess that's
full confirmation that it works. It also does the same thing that I was doing before with curl.
Now, I do it natively. So every minute, I export it to MQTT with Home Assistant. I don't have the
plot functionality, but I can get it from Home Assistant. So that's fine. This is the source
code. And sooner or later, I will try to put an Ansible playbook so that if the SD card dies
once more, I will just have a very quick way to deploy it. As a bonus, for the last minute
of the talk, I have a picture of the PCB because at some point, I wanted to update the DBN11.
The nickname changed, so it didn't get the network anymore. I had to really connect the Raspberry
Pi to the keyboard and monitor. So here's the PCB. It's a work of art. It's all through whole
components. Don't ask me why. The inputs are voltage dividers, so learning that the blue resistors
are the more precise one finally paid off after, like, 40 years of my life. Here's the battery-backed
RTC. There's a power LED and an alert LED connected to the GPIO. This is nice. This is a driver I see
for the relays and also for the LEDs because these are powered in five volts, not 3.3. And there's
also, this is nice if somebody wanted to hack further on it. These are test pins and they are
connected to eight more GPIOs on the Raspberry Pi thing. But there's something really weird here
because this part, these Q-terminals are not exposed. They're unused. And this is a bias
resistor. This is a terminal resistor. This is another RS485 transducer because remember that
it was connected through USB. And it actually works. I have no idea why it's there. But if you
look at the website, there is probably an older version of the board where you can actually
read here common A and B. So at some point they wanted to use it. And then they didn't so
on the brochure picture, this is not used on the website picture. They still have the older version.
And that's it.