All right. So, it's two minutes early,
but I have a tendency to speak and not stop,
so we better get the start in here.
So, I am so tired of doing introductory talks to poke.
So, I have decided that this is the last one I'm going to do.
So, but let's see.
Is anyone here familiar with this program somehow?
Yes? Okay.
So, yeah, we need to do an introductory talk.
So, GNU-Poke is an Extensible Editor for Structure Binary Data.
So, it is a program that you can actually expand.
We will see how, and you can expand it actually quite to a very high degree.
And we will see that it's a program that is used to poke,
or to mess, to edit data which is encoded in binary.
We will see what is this thing about binary.
Everything is binary at the end of the day, right?
So, first, I'm going to do a very small introduction,
an abstract one, right?
I mean, why would you use such a program?
Then, we will see how we are actually integrating poke in
other programs via a shared object, a library.
Then, I will basically talk a little bit about the project itself,
the current status.
We just made a major release that we are very happy about it,
and we are very busy because the project is getting a lot of,
we have a lot of things to do.
It's getting very fun actually.
And then, finally, an invitation to you to actually join us in,
you know, in hacking this program.
So, let's start with a very small introduction.
I mean, when it comes to edit binary data, well, what do you use?
You use a binary editor, right?
If you go around and then you look around, you know,
in internet about binary editors, you have, you know,
the simplest kind, what I call like simple binary editors,
which is, you know, your garden, garden variety, you know,
programs which show you a byte dump.
And then, a lot of them also show you the ASCII representation
of the same bytes.
What do you see in the screen?
How many of those programs are around?
Quite a lot of them.
And they are nice.
I mean, they are nice, they are useful, they are small,
they are very easy to use.
There is nothing mysterious about them.
They are interactive, usually.
The ones that allow you to actually change the value
of those bytes in the files.
You just go to the byte you want to change,
you put in, you know, the new value for the bytes, ta-da.
If you want to change, you know, based on some string,
you go to the ASCII column there,
you go to the position you want to edit, you change it,
ba, you know, the file gets updated.
Very nice, they are interactive.
They support immediate editing.
Like you go, you change here, and it immediately gets reflected
in the file that you are editing, for example.
What do they let you edit?
In terms of what?
Well, it depends.
But on those simple binary editors,
they let you operate in terms of bytes,
in terms of strings, like we just mentioned.
And sometimes, this is not often,
but sometimes in terms of bits as well.
Some of those basic binary editors,
they support, you know, down to the bit level too.
And that's it.
That's the thing, right?
You can edit in terms of those entities, of those concepts,
bytes, bits, strings sometimes, sometimes numbers.
Then the editor is not so simple anymore, but you see.
But it's always, you know, like a fixed list of abstractions
that you can edit.
You manipulate your data using those abstractions, right?
And then they give you, you know,
byte dumps, the ASCII views, and they have fixed capabilities,
depending on how nice the particular editor is.
Some of them allow you to search for patterns
from others, you know, to search and replace,
to make byte divs, right?
That kind of things.
Measure entropy to search for frame buffers,
that kind of things, right?
Simple binary editors, they are useful.
But then sometimes you have to edit,
or you want to edit your data in terms not just of bytes
or bits, but in terms of more abstract entities,
like MP3 headers, or else relocations,
or list of numbers, or whatever.
And then for that, you have traditionally
specialized binary editors, which are editors,
like the two that you can see here,
that know about some particular format.
Like some of them, the one on the left knows about the L
format, so you can see a tree view, which is quite nice.
You know, with the different sections in the L.
And the one at the right, which is small, but it's nice,
is a very small MP3 editor to edit MP3 files.
Actually, the metadata associated with the MP3,
like the title of the song, the name of the singer,
or whatever, right?
Those specialized binary editors, they are nice too.
They are useful.
But they are not extensible, usually.
I mean, you cannot go and say, oh, well, you know,
I mean, in the ELF program here, you know,
you know that ELF sections can contain arbitrary data, right?
So you say, well, I want to use the same editor
to edit the contents of one of the ELF sections.
Usually, they don't let you do that.
So they are not extensible.
They also, they are not quite good at dealing
with incorrect data.
So if you get a corrupted ELF file,
or some MP3 file that has problems in it,
those editors probably are going to refuse
to actually open them, open those files.
And if they do, they are probably
going to show you garbage, right?
So they are not good at that.
And you know, it's the typical situation.
Exactly what you need is what they don't implement, right?
It's always like that.
So those are the specialized binary editors.
And then we have K-Ti extract and friends,
which is they implement this paradigm of, as you know,
we got a nice presentation before.
The first, you decode the data, like with one of the parsers
of K-Ti extract, for example.
Then you do your computation with the data.
And then maybe you use an encoder, right,
to write back the modified data.
This is what I call the code, compute and code.
Programs like these are also useful.
Those are extensible, like we have seen with K-Ti extract.
You can define your own structures.
They use, usually, you know, some sort of declarative way
of describing the layout of those data structures.
They generate code in several target languages.
I don't know why K-Ti extract does not generate C. I mean,
to me, I'm so puzzled about that, but OK.
Usually, those are non-interactive.
They are not interactive.
Like, you generate a parser that generates, you know,
that parser in some programming language that then you
incorporate in your program, and then you run, right?
Usually, they are not that good to dealing with incorrect data
either, right?
Because the parser that they generate expects correct data.
And they are either bit oriented, which is not common.
I don't know if K-Ti extract can deal a little bit level.
Good.
Also, not aligned stuff.
And all byte oriented.
And often, there are no encoders.
I know that K-Ti extract now they are starting to add support
for actually writing data back to the file.
This is nice, too.
And then you have the poke approach, which is circular,
right?
What we wanted with this was the following.
We wanted the immediate aspect of the simple binary editors.
Like, OK, I go to this byte, and then I change it.
Now, you know?
I mean, now change it immediately.
But also, we wanted the extensibility and the ability
of working with higher abstractions
like you have with the parser generator,
like K-Ti extract, for example.
We wanted everything together.
And that is what poke is in few words.
Basically, you describe your data structures,
like in abstract type, for example,
and you can immediately poke at it.
You can immediately update it, edit it, write to it.
And if you are not satisfied with that,
you can, on the fly, using the same program in the prompt,
you can redefine your data structure and do it again.
This is good for also discovering the format of what
you are editing, like in reverse engineering and whatnot.
And when you are developing a new format,
you know, that kind of use cases.
So it is interactive with the poke application.
It allows immediate editing.
It allows data integrity.
You can define your own complex structure,
quite complex ones.
And then it supports a very powerful and, to the point,
domain-specific language.
I'm a big fan of domain-specific languages
because we have the ability and the brains to actually,
you know, talk in several languages
and write in several programming languages.
And it is so great when a tool actually gives you,
you know, the way of expressing things
that are most suitable for the task at hand.
The DSL is called poke as well, like the program,
but with a big P to distinguish the programming language
from the program itself.
It is interactive, aesthetically typed, in purpose.
It is garbage collected.
It has some very interesting features, this programming
language, because it's designed to, you know,
to the point, to the task at hand.
So for example, it's not bit-oriented
and it's not byte-oriented.
It is unit-oriented.
So in poke, when you start talking about offset sizes
in memory and so on, you don't talk in bytes or in bits.
You talk in terms of arbitrary units that you can define.
Right?
I'm sorry I cannot get in detail because this is,
you know, like a fast pitch, but you have all the information,
you know, in internet and so on.
And also, it works in bit-addressable IOS spaces.
It can work within correct data because you
can do non-strict mapping, saying, OK,
I want to disable the constraints,
that integrity constraints, so you can actually discover
the, you know, what you have in front of you
and adapt your own definitions to it and so on.
You can define several versions of the same structure
very easily to be more strict, less strict.
And it is extensible.
We will see that notebook is not just a binary editor.
It is a full infrastructure to write binary utilities as well.
Right?
And then, similarly to what KITI extract
is aspiring to do, for example, when
it comes to document formats, you
can use poke also to document formats and protocols
in a functional way because your same documentation,
you can use it, you know, to actually operate with the data.
To do prototyping, to write binary utilities,
to implement filters, and so on.
And then, to integrate in other programs, which is very cool.
I will show you in five minutes one example with the debugger,
with GDB.
Now, poke operates in, it can operate some files, memory
buffers, you know, the memory of running processes.
There is a collection of IOD devices, right,
which are what you are editing.
But what a poke program or what you have access
from the command line is to a bit addressable IOS space.
We call those IOS spaces in which you can actually map
or, you know, manipulate different kind of entities,
which are integers and same integers are strings, right,
from the poke language.
OK.
We all know what bytes are, right?
You will be surprised many people don't.
They are just little numbers in a certain range, right,
from 0 to 255.
So this is the way, you know, in poke,
you have to refer to bytes.
But what I wanted to show you, because I think
it's the interesting part, is that it's the way that you go,
that in poke you have from going from bytes
to actually encode in integers, right?
So you see here, I don't know if you can see,
you see here the IOD bytes there, which is the underlying
device that you are editing, like the file, right?
And then it has bytes, which are little numbers, right,
from 0 to 255 in the range.
And then the IOS space, which is on top of each byte of the bytes,
is the bit addressable IOS space that your poke programs
actually see.
But we all know that bits are actually
a very interesting thing.
Bits exist usually at the hardware level,
then they disappear until you will recreate them virtually
on top of those byte numbers, right?
It's very interesting.
But so from poke what you see is the bits that are conceptually
on top of the bytes.
And that's the poke type, for example.
This is a very, this is an unsigned 16 bits integer
mapped in the IOD device at the first byte.
But this is a boring example, it gets even more interesting.
Like what we call wired integers, weird integers, right?
So for example, in poke you can operate with 12 bits
unsigned integers, as naturally as you
will do with the 32 bits unsigned integer.
This is quite cool actually, and useful, believe it or not.
Then we have some conventions to refer to the bits and everything.
But this integer actually occupies one full byte
and then half of the next one.
But also you can go to less than one byte, right?
Like in this case.
So you can actually operate with an unsigned integer of five
bits, and then it doesn't feel like a complete byte.
Obviously, since everything that there is in a computer is actually
bytes, the drivers level, hardware level, you know,
and everywhere, this is an artifact.
But it's a useful one.
And poke also has full support for unaligned stuff too, right?
So you can work with actually a 16 unsigned integer,
shift to bits in the IOD space.
Can Kaitai extract do that?
Yes, we will have to see.
So yeah, so you could also skate your file just by shifting
that three bits to the right, for example.
Why not?
Maybe fun.
I included this here not to impress the cat.
But to give you an impression, you know the impression that
actually poke whether it's a serious problem.
I mean, it's not just a stupid problem that poking at bytes
here and there.
We actually take it very seriously.
And you can do this kind of stuff.
And believe it or not, people need this kind of stuff.
The other day in the IRC, we met the multicians, which is a
community of people who are dealing with multics.
And you will not believe what they need.
Really, right?
Like nine-bit bytes, you know how it's unbelievable.
And we are struggling to actually give that people what
they need, because they have rights too.
The multics people have rights too.
Anyway, so the poker sphere, poke is growing and growing.
This started when simple.
It was always a little bit special, but a little program.
But it's getting out of hands at the moment.
And in the sense that we have leap poke, which is a
sered object.
Obviously, first I made it in a prong, but then dodgy in one
called an asshole, put it in a library.
So I did.
And then leap poke is a sered object that has the poke
incremental compiler, the IOS page support, and everything.
We will see now with GDBS an example how you can actually
make use of it in your own programs.
Then poke is the command line application, which uses leap
poke.
But it's just a program, a very small program, with a
prompt.
Like being a less load elf, you can dump bytes, right?
Oh, sorry, here.
And bar elf, elf64 file at serobytes, stuff like this.
The command line application.
But all of the logic is actually in the sered object.
Then you have other applications like GDB, which
is not upstream yet.
It's in a branch upstream, but not in master.
But they can actually use leap poke to give you poke
capabilities.
I will show you now in a two-minute little demo.
There is a poke demon 2 that Mohammed will talk about.
They have 10 minutes.
Utilities, the pickles, which are the poke programs that
give you the support for some particular format or for
a particular domain.
Then there is an emacs interface, of course.
Then there is an emacs mode for the poke program, Veeam,
for unholy people, for editing the poke code in Veeam, in
VI, and something called pokelets, and so on, and so
on, and so on.
So it's getting fun.
And then the integration.
I'm very like this.
You know, there is one problem when people write a
program, they want it to do everything very well.
And that does not work.
So for example, in poke, we have some support for editing
the memory of running processes.
We do.
Well, I could show you, but I don't have time for that.
Because we have an IOD, which is you can specify the process
ID of a process, and then you can edit the running memory of
the memory of the running process.
But poke is not a debugger.
It's not.
I mean, there is a dwarf pickle, but you
know, not to the same extent as a debugger.
You cannot set breakpoints.
It cannot use ptrace to command a process.
But gdb is a debugger.
So gdb is good at debugging, at running processes, and so on.
And poke is good at poking the data.
So let's put them together.
So then the combination is good at both.
And this can be achieved by using live poke.
So I can show you very fast.
So I have this C file.
You see the C file?
I have a C extract from with an intact.
Can you see it properly there?
Yes?
Which one?
No, I'm not going to address this.
Trust me.
OK.
So there is this extract type, and then there is this
buffer, there is this db global variable here, and so on,
and so on, right?
And an int main, int main.
So then I compile that into an aout, and then I do home.
This is the poke capable gdb, a dot out, break main, run.
Now I am in main, right here.
And then I can use gdb to look at the database here.
You know this db global variable, and so on.
Now this gdb is extended with poke.
So you have a poke command where you can execute any poke
code, like this, like 2 plus 2 equals 4.
Brilliant.
But here you can do anything that you can do with poke.
So for example, you could say, and poke, in this case, it
has access to the memory of the inferior that you are debugging
with gdb.
So even if it's multi-process or multi-thread, you
switch in gdb, and then poke has access to the memory of
the inferior, right?
So what kind of things you can do?
Well, anything.
I mean, what is the address in gdb of this global variable
here?
Sorry.
That is the address, right?
So with poke, you could access to that address.
But you have a command which is poke at types, right?
Which is you are telling gdb to make poke aware of all the
types known to gdb at this point in time, right?
So the type char, the char type int, the struct
prop that we saw before, and then this translated into poke
type definitions for the keyvalent gdb types.
This means that you can go from btf, from ctf, from
door, from any debugging format that gdb understands to
poke types, what you are using gdb in this way.
Now, poke now has access to the memory of the inferior.
So we could do, for example, these, and those are poke
expressions, right?
Struct prop at where?
And you can use something we call alien tokens, which is
the address of the gdb symbol db, which is the variable,
right?
And this is the poke struct.
This is not a gdb value.
This is a poke struct, which is poking at inferior memory.
And you, of course, can write.
You can do whatever you want.
You can load pickles.
Why is this useful for?
Well, imagine you are debugging a program that is some
sort of router for TCP.
So the program itself, you know, you have TCP packets and
some buffers in the program.
But the program itself doesn't need to understand the
payload of what it is transporting.
But imagine that you want to take a look at what is going
on there, right?
So then, you don't have the dwarf definitions of the
structures that are in the buffer, in the payload.
But with poke, you just load your pickle or you
write your own, and you can poke at it from gdb in the
buffer of the running process, for example, right?
So this is one example of application.
This is 400 lines of C, the integration, using live poke.
It's very nice, and, you know, it's easy.
Another example, which is work in progress, and it can't
wait to get this finished, is basically to add to the
assembler a.poke directive.
Because you know those.war,.bite, and so on directives in
the assembler?
They are not portable.
And this is a very, very, very, very, very, very, very,
pain in the ass.
Because when you have to, actually, for example, test
door for tests, you know, that kind of things, then they
are not portable.
I know it's amazing, but they are not portable.
So see, for example, that.
This is a real example of something I found.
I will not tell where, of some people who are actually
embedding some sort of executables inside other
sections of some of the stuff.
This is Sony computer, this is video games, you know, kids.
They do these kind of things.
And that is a struct in theory of what, of some header, right,
for the PlayStation or whatever.
Well, compare.
You see?
So once we have the integration in the assembler, then you
will be able to load the PSXX pickle.
It will be a small one where you define the structure of
the PSXX executable, and then you could poke at assembly
time, you know, like that, too.
And accessing gas assembler symbols, you know, using
alien tokens, like we saw you could do with GBB as well.
So this is another example of integration.
Yeah.
So and we are on these kind of things now.
We are looking into parasiting other programs, you know,
and incorporating live poking them.
So they do all the boring work, and we do the fun one.
OK.
So what's the current status of the program?
We just released the POC 3.0 a few days ago.
Up to now, we were doing one major release every year.
And we had a maintenance branch.
But people are not happy.
Why?
Because it was too long, and the difference between POC 2
and POC 3, it was too big, it was too much.
And actually, we released the POC 2.0, for example, and in
two weeks, we have forgotten about it already.
Because we are so happy, you know, and so excited with the
main branch.
So now we are committing to release two big, major
releases every year.
Let's see if we can actually do that.
So the development, we are old, new peace pods, right?
So we don't use GitHub or anything like that.
So we use a mailing list, and you send your patch to the
mailing list in Unity format, and so on, and so on, right?
And well, that is the website of the project.
We have a Git repository, obviously.
We have a mailing list, a development mailing list.
We have a very nice build box at Sourceware.
Thank you very much to the Sourceware Oversers.
They are doing a great job maintaining the
infrastructure of many new pack programs, including POC and
also the Toolchain, GCC and Ellipse, and so on, for many
years.
And also, we have a pipeline hosted at GitLab that Bruno
Hibel maintains, and I have no idea how it works.
It's not clear to me what even a paper line is in that
context, but it's green.
So I guess it's good.
We have a community website.
It's called Pocology, and we try to get practical
information there, like how can you write your pickles, and
so on.
And also, I have a blog in my website where I sometimes
publish small articles with practical stuff.
So how can you do, for example, implementing Spark stables
using POC, or accessing them stuff?
We want to be friendly to users.
And now, starting now, in POC, we had in the POC
source distribution, GNU Hacks POC, we had a pickles
directory with a lot of pickles.
Like for P, for L, for DOR, for BTF, for this, for that.
Some instructions, set RISC-5, BPF, but this is getting
crowded.
And actually, some pickles are big and complex enough to
actually need their own releases, so you can have
several versions that work with the same version of POC.
So we are basically putting some of the pickles in
separated packages.
And that's the case of the elf pickle and the
dwarf pickle.
So for example, the elf pickles, now they are distributed
like separately, have not made the first release yet, but
they are in the Git repository.
They have their own manual, and so on.
And the dwarf pickles, as well.
Those are the first ones.
We want to get the peak of support, because it's a huge
fat monster, that one.
We want to put it also in its own package.
And also, with nice manuals, I have to show you this,
because it is such a pain to do it that you
have to brag about it.
This is the POC manual, the new POC manual.
And then, when you install the pickles packages, like POC
dwarf and POC elf, and here, you have a nicely
documented, you know, the pickles and everything.
And the source distribution, sorry?
Use man pages, not the menu.
Well, well.
Oh, well.
We are generating man pages from the info.
Then, actually, the idea is to use the elf pickle, because
we are new writing pickles, because POC is sort of new.
So we are trying to discover our way forward, but the POC
elf pickles is sort of the canonical example.
We are using it that way.
We are writing it very carefully.
So if you want to write a complex pickle, you can look at
it, but, well.
And this is it.
I am so sorry that they could not give you, you know, an
actually taste of how this program is.
But there is no time for that.
And there are other videos in the internet that they have
done already.
But, and then, if you want to join the development, please
read the hacking file.
Because we took the effort of writing it.
It's huge.
It has a lot of good information.
And absolutely, no one reads it.
So thank you very much.
Thanks.