Okay, now we have Charles Brozegard and he's going to speak to us of binary pattern matching
in Elixir and Erlang, one of the cool features of Erlang and Elixir.
So if you speak binary to me, give it up for Charles.
Thank you, don't worry, I'm not going to speak binary, but yes, this is a talk about speaking
binary to other devices.
And this is me and you can find me around the web at ARTRARBR, yes.
I work as a software developer in Denmark in a company called Indelab and part of my job
is to work on innovations in the Internet of Things realm.
So things like smart buildings, smart cities, and smart factories.
In practice, this involves building gateways that link many different kinds of stupid things
together and then when we have a network of things, they can exchange information and
we can embed the smartest into the system.
And by many different stupid things, I mean things like remote terminal units which are
used in the grid, electric grid and all the utilities, PLCs which are used heavily in factories
for automation, solar inverters, heat pumps, thermostats, all kinds of smart home equipment.
And at the lowest level, we have simple sensors and actuators.
And the thing that all these things have in common is that I had to speak binary to them.
They don't speak JSON, they don't know XML, they don't know protocol even.
They had their own custom binary dialects depending on the protocol that they are using.
And later in this talk I show an example of a simple binary dialect so that we all know
what it is.
When I'm building an integration for a system like this, I always reach for Elixir first.
This is because Elixir has some special affordances that makes it extremely good at this kind
of work.
This is also the case for Erlang and LFE and Beam and Gleam and the other languages on the
Erlang virtual machine, but Elixir happens to be my happy space.
Some of the Beam's known strong points in this area are fault tolerance, state machines
and concurrency.
And then there's business acts and that's what I'll be talking about today.
This is a beginner level talk.
I don't assume you know anything about Elixir or the previously mentioned systems, but my
hope is that if you should find yourself in a situation where you need to speak binary
to something, this talk will help you get started.
So binaries.
Computers today work by manipulating electric signals and a signal can be either a logic
high or a logic low.
Because a signal is known as a bit and a sequence of 8 bits is a byte.
And this is also how computers communicate.
No matter if it's Ethernet or Wi-Fi or whatever, it's about transferring a logic signal of
high and low bits.
There are different ways we can write down the binary signals.
The first notation that I highlighted here is in binary where we just take every bit
in the sequence and we write it down as a 0 or a 1.
And then it prefix it with a 0B so that it's easier to tell from other numbers.
And in this case, the sequence of 8 bits, which is a byte, can also be turned into a
decimal number because in a byte we can have 256 different combinations.
So it's a number from 0 to 255.
And in decimal notation, this sequence of bits is 75.
There's also Hex notation, where we use the characters 0 to 9 and 8 to F.
This means we can describe the constant of a byte with just two characters.
So this is very convenient when we are dealing with binary numbers.
Just a common example of where we write the bytes as decimals.
We do that with IPv4 addresses because an IPv4 address is just four bytes.
But then for human consumption, we write it as a binary number, or sorry, a decimal number.
And MAC addresses use Hex notation instead of the binary notation.
While all the operations in our computers are processing bits and bytes, we rarely think
about them when we are programming.
We think in terms of integers, floats, strings, lists, maps, and all the structures.
But binary data is just a flat stream of bits.
There's nothing inherent in it to tell one field from the other, or any kind of structure.
It's all down to how we interpret those bits.
This means that when we need to speak binary to something from our programs, we need to
write a translation layer that can take whatever list of maps we have in our program and turn
it into a binary sequence.
I prefer calling this translation layer a codec because it's short to write.
And you can say that a codec encodes our data structures in bytes for sending.
And then when we receive data, the codec decodes that data into our high-level data structures.
And sometimes we're lucky we can find a library that will do that for us, but sometimes we
need to do it ourselves.
And that's where bit syntax comes in handy.
So let's take a look at that.
Let's say I need to send a sequence of free bytes to some other system.
They need to have values 10, 20, and 30.
To do that, I use distance x, so I use double ankle brackets to start the binary sequence.
And then I write the bytes that I need in sequence and separate it by a comma.
So that's pretty simple.
Yes.
And when I need to receive that data, like if I'm receiving the same sequence of bytes,
I need to decode that into free variables, a, b, and c.
Then I use this syntax, again, double brackets.
Then I put the variables instead of the numbers that I want to extract.
So it's pretty simple.
But of course, we are not really done yet.
It's very few domains where we only work with integers in the sequence of 0 to 255.
We need larger numbers.
We need negative numbers.
We need floats and strings.
But many languages will just give you a byte stream, and then you need to sort of do a
lot of strange computations to turn that into lists and strings.
But in Elixir and on the Beam, we have the bits and sacks, and we can do more.
Specifically with bits and sacks, we have the option or the ability to specify modifiers.
And the modifiers can specify the type, the sign, the size, unit, and indianess of the
sequence of bits we want to extract from the binary.
So here you see that the type is integer.
It is unsigned, so it's positive numbers only.
It has a size of eight units, and one unit is said to be one bit long.
And it's big indian, which I will talk about later.
So these are all equivalent 10, 20, 30, they are encoded the same way.
It's just different syntaxes.
So you can see if you don't specify any modifiers, these are the defaults that I used instead.
And the second line I used, I omitted size, I just wrote eight.
That's something you can do when you know it has a constant at compile time.
If the size is variable, you will need to use the full size modifier.
And the modifiers can be combined in any order, so you can do whatever you like.
And when we decode it, we use exactly the same syntax.
We can say the same things, like grab the first byte, tell it to compile it that it's
an integer, and then it will extract it like this.
And instead of just going through all the different modifiers and the combinations, I move on
to showcasing some examples.
But before we do that, I want to mention where the bits and text came from.
Bits and text comes from a place of pain.
These two guys, Clairs, Wichström, and Tony Rockwell, were working at the computer science
laboratory at AXN on implementing networking protocols for Erlang, and it was painful.
And so they sat down, since they were so close to the makers of the language, they could
invent a new syntax for use in Erlang.
And this paper, which is published in 1998, describes the first version of the syntax
as it was implemented in an experimental version of Erlang.
I think a few months later, it was released with slightly different syntax, but with all
the same concepts.
And that paper also explains what Indianness is, and it's actually just a fun word for
byte order, because if you have a 16-bit integer that you need to send to some other system,
that's two bytes, right?
And you have to figure out, the one byte is A2, and the other byte is C1.
And you have to figure out which byte do you send first.
Some systems will send the most significant byte first, so that's A2.
But other systems will send the least significant byte first, that's C1.
And so this obviously has consequences, because you need to know the byte order that the system
you're talking to, what it expects, otherwise it just gets confused.
And yes, the byte ordering is said to be big Indian when you start with the most significant
byte, and it's said to be little Indian if you start with the least significant byte.
And this is kind of a thing, I've been working with this for years, but I didn't really know
what Indian means, because it's a sort of weird name, right?
But the paper by Claes and Tony hinted me in the direction of finding that by pointing
me to this Internet experiment note from 1980, which is, I think, the first sort of place
where Indian and byte order was used together on holy wars of the plea for peace.
And that sort of shows that this is an important topic, sort of like Vim vs. T-Max, I guess.
And it's actually just, Indian is just a pop culture reference to a book called Gulliver's
Travels, where a seagull travels out into the world and meets the people of Lilliput
and Plefusco, I think, and they are in conflict, because the emperor of Lilliput has commanded
that X must be broken at the little end when you eat them for breakfast or whatever.
So that's obviously absurd, and so they wage a war.
So big Indian means we send the big end of the number first, and the little end means
we send the little end of the number.
So examples of bits and texts.
For the purpose of this talk, I have invented the T-Box, which is a very simple device.
It has a name, and it can measure the temperature, and it can tell you if there's an error in
the time stamp or the measurement.
It has a binary dialect or protocol, which I also invented, and this sort of mirrors
what you will find in a real protocol description for some kind of device.
A client can connect to a T-Box and can send requests to the T-Box, and the T-Box will
respond with a reply.
Every message that is sent includes a header, which is one byte long, and replies from the
T-Box will also contain a value.
The header starts with four bits of magic, which is a constant value that is always there
and is used to sort of make sure this is the beginning of a message.
Then there's a direction bit, which tells whether this message is a request or a reply,
and there's the attribute, three bits, which are used to tell if this is a name or temperature
message.
There are extra, we only use one bit in the attribute, but that's just because they expect
to expand the protocol someday.
This is an example of a sequence request-reply.
First we send the request with the header, with the magic bits first, then it's a zero
because the direction is a request, and then we're requesting one, which means we're requesting
the temperature, and then the reply has almost the same header at the beginning, it's just
one in place of the direction, and then the bytes with the value after that.
If you're requesting the name of the T-Box, then it will respond with 12 bytes, and it's
always 12 bytes.
If the name is shorter than 12 bytes, then the rest of the bytes are just null bytes,
and that looks like this.
If the name of the box is fustum, then you have six bytes of actual characters, and then
null bytes for the rest.
The temperature is a little more complicated.
It has three fields.
There's the time, which is a 32-bit integer, counting the number of seconds since 1st
of January, 1970.
Then there's the temperature, which is a 16-bit float, and then there's the quality byte,
which tells you whether there's an error in some of the measurements.
It's the two last bits in the Q-byte that are used.
The second to last bit tells you there's an error in the clock, and the last bit tells
you there's an error in the temperature measurement.
It's important to note that the numbers are little in the end.
This is what a temperature value looks like.
First, it has a timestamp, which is a couple days ago.
Then the temperature, which is 32 degrees about that, and then both of the error bits
are high, so you should not trust the sample.
This syntax, we want to send a request to the T-box to get the name.
I use the double anchor brackets again.
It's all integers, and it's all unsigned, so the only thing I have to specify is the
size.
Here I specify the magic is four bits, then I have my direction, one bit, and the attribute
which is zero for getting the name.
This shows how we can use the bits and tags to encode, easily encode things which are
smaller than bytes.
The reply that comes back looks like this.
When we receive the goodbye, we do like this.
First I want to assert that the message I get back is what I expect.
In place of the header, I assert that the values are what I expect.
I assert that I get the magic bits first, that the direction bit is high so that I know
it's a reply, and that the attribute is zero so that I know it's the name.
This is all true.
I know that the rest of the message is 12 bytes long, and I want to assign that to the
variable name.
Here you can see I will use the bytes modifier.
This changes the type, and it changes the size, or it changes the unit property.
Before with integers, I would specify, like in the header you can see it's two colon
colon four for the magic.
That means four bits, but when I specify bytes, that the type of name is bytes, then
I also say that the 12 means 12 bytes long and not 12 bits long.
That's just to say that the bytes is a type and has different defaults than integers.
When I get the Tbox temperature, I request it like this, pretty much the same as before,
just with a one for the attribute, and for the reply, I again assert on the header that
I get back what I expect.
Then I have the timestamp, which was the 32-bit integer, but little indian, so let's put that
in there, and the temperature, which is a float, 16 bits, also little indian.
Then I discard six bits from the cube byte because they were not used, and then I just
plug the second to last and the last bit out as clock error and temperature error.
That's the basics of bits and tacks.
There's so much more to cover with writing whole applications or libraries to do this
kind of stuff, like what do we do when you don't receive the entire message at once,
you have to frame messages when you're streaming them or receiving them as a stream.
There's generators, there's a special, well, it looks like just a normal forward generator,
but it actually has some special optimizations for working with binaries when you need to
generate a binary.
We could talk about performance tuning, but as I said, I've been working with this for
years and I've never had to really do any performance tuning.
It's pretty performant as is.
That's also one of the points in the paper by Claes and Tony is that this is performant.
Yeah, there are many other tools for working with binaries that help us when we're looking
at this data and not understanding why it's not doing what we expect.
Wireshark is definitely one of them.
I recommend checking that out.
I also recommend if you want to explore more depth, that you check out Protohackers, which
is a sort of advent of code challenge thing about their networking protocols, and Andrea
Lopardi from the Elixir core team has a live stream on YouTube or has streams on YouTube
where he sort of goes through the problems one by one, so that's very good for learning
that.
Andrea has also started writing a book about this kind of stuff.
So that's it.
Thank you.
Thank you for your thoughts, Rolf.
Do you have any questions?
Hi.
Thanks.
So I've done, I've implemented a few binary protocols like network protocols like HTTP
2 or other stuff like that.
I'd love to know how you, if you have done any streaming of data and passing of messages
coming from streaming data and generators also would be interesting just to know how
you approached it because I know I did something but I don't know how we did it.
I think I've implemented a few protocols that use, for example, TCP as the underlying
transport and then that's like a stream.
And then there are a few patterns for how you want to handle that.
I think Frank Honloth and the nerve team wrote a sort of framing behavior, which way
you have a couple of callbacks that you need to implement in order to handle a stream so
that they give you bytes and then you return back messages when you see a full message.
It's actually a very good sort of guideline for how to do that.
Another approach is taken by Andrea Leopardi in his library for Redis where it's like
you call, when you call the decoding function and it returns a result, the result will be
whatever messages was in that binary you gave it and then a continuation function which
you call next time with more bytes so that it continues to return new messages, yeah.
Any question?
If I would buy a T-box from you, do I still get support in about 16 years from now?
Maybe five minutes.
I actually have a question for you if there are no other questions.
Do you have a library you suggest to see?
Because I implemented a library to decode the QOEI which is the quite okay image format
which is a new image format just to get my irons dirty with binary pattern matching in
Elixir but it gets very unwieldy very fast.
So I saw like in JSON they have some macros that generate binary pattern matching.
Do you have any libraries you recommend to check out?
I think the Redis library is pretty nice.
I also have a KNX library which is like a smart home protocol which I don't know if
I would recommend but when I wrote it I thought it was made sense, yeah.
Thank you.
Any other questions?
The last one I guess.
Maybe a word about exceptions when patterns don't match.
Yeah.
You didn't talk about that if I'm correct.
That's true.
I mean sometimes we say we should just let it crash in this community.
That's not always the case.
Sometimes the protocol will say that if you're unable to decode a message you must ignore
it and just continue going.
In that case what I usually do is I define functions where I match on the data I received
and then I always need to have a fallback clause in case it didn't match anything and
then just lock an error and continue.
But I mean probably the proper thing to do in Erlang is to try to just die.
There's also you could when you receive a message you could have a special process that
is only for decoding so you start a task, decode a message and get the data structure
back but I think it depends on the use case how you want to handle bad data.
Okay, thank you very much.