Hi everybody, welcome to my presentation about Kaitai struct.
I am Peter Pucil and I have a question.
How many of you have any experience with Kaitai?
Okay, there are a few of you.
What is Kaitai struct?
It's a tool for dealing with binary formats, especially parsing.
It is based on a declarative language, Kaitai struct YAML, that can be used to specify arbitrary
binary formats.
It works as a parse generator and it currently supports 11 target programming languages.
Parsing means to convert the binary data you see above to the structure data and object
tree so that you can work with it later.
Today I will also introduce a new functionality, which is serialization.
I've been working on it for the last six months and it currently works in Java.
Serialization means, I didn't mention that, that basically the inverse process.
You want to create a binary file from an object tree.
Something about this story.
So the author of Kaitai struct is Michael Action and the project started in 2014.
In 2016, Michael decided to release the project as open source and at that time the only supported
languages were Java and TrueWe.
In 2017, Michael presented Kaitai struct at FOSDEM and by then it already supported eight
languages and had over 400 stars on GitHub.
Michael also wanted to come today but unfortunately he couldn't.
But if there is some chat or something, I think he should be there so you can ask him
some questions or whatever.
And how is it today?
So we have 11 target languages and over 3,000 stars on GitHub and Kaitai is used in more
than 500 GitHub projects.
So let me share how I discovered Kaitai struct.
This was in 2019 and I was playing electronic keyboard with a band and I wanted to create
a MIDI editor so that I could record the songs on the keyboard and edit them on the computer.
And I wanted the user to be able to upload a sound bank in the sound font to binary format
so that they could control how the song could sound.
And I wanted a web-based MIDI editor so I searched for a JavaScript parsing library
of the.sf2 format but I couldn't find one that would work for me.
So I started writing my own parser but it was really hard and a lot of debugging had
to be done and it was just not fun.
And when I finished I came across Kaitai struct and I found that my two months of work I spent
on this could be done in just one day with Kaitai.
So Kaitai impressed me with its concept, simplicity and versatility and I started contributing
a lot.
And Kaitai also helped me my personal development because until then I'd only programmed in
JavaScript, PHP and a little bit of Python and within a few months I was able to work
in 14 programming languages that were used in Kaitai.
And in 2020 I accepted an offer from Michael to become an administrator of the project.
So in my story I showed what options there are to get a parser.
So the most convenient way that you are probably familiar with is to use a dedicated format
library in the given language.
So it will probably have a user-friendly API and can be optimized for a format.
But sometimes it may be of poor quality and incomplete and it may be difficult to debug
and fix it.
And also for the most common formats like JPEG, L for zip, you can find even several
libraries and you can choose, but for less common formats, some obscure ones, there will
simply be no library in your language.
So we need to look into other options and another option is to simply write your own
parser.
But in my experience this is the worst option because it takes a lot of time and you need
to do a lot of debugging using some debug brains and dumps and it's just not fun.
But it's what most people do, often because they just don't know any better.
So that's why I'm here today.
And well, the problem is that if you have already written a parser for your format in
Python, for example, and then after some time you are asked to create a Java parser for the
same format, you basically need to start again.
So a bit better way is to use a parser combinator, which means that you are essentially still
writing your own parser, but you are using some building blocks from a library.
And a parser combinator typically allows you to declaratively define some sub structures,
but still in the code and like in CU can define structs for the fixed size pieces of the format
and then you can directly interpret some block of bytes with that struct.
And there are many parser combinators, perhaps dozens in popular languages, but as with the
two previous options, you have still the disadvantage that the parser you get this way is still
bound to the particular language.
And it may be even bound to an application.
For example, if it was developed for a graphical editor, so it may be difficult to separate
just a parser from that application to use it somewhere else.
And the fourth option is to use a parser generator, which means that you are not writing the parsing
code directly in the programming language, but instead you describe it in a domain, describe
the format structure in a domain-specific language, and this description can be then
automatically translated into a parser.
So, KeitaStruck falls into this category and it is the KeitaStruck language is designed
so that it's independent of both the application and the programming language.
Here I'll show you how to work with Keita.
The first stage is compilation.
So you take this KeitaStruck specification of the format and in this case, this is a format
that has one byte because this U1 type means unsigned integer of one byte.
And you take this KeitaStruck specification and you compile it using the KeitaStruck compiler,
which is a command line tool.
And as output, you get the source code of the parser, in this case in Python.
The main stage is parsing.
You take, you give the input binary file to the generated parser.
You get in the first step and you give the input binary file to the parser as input and
you get parsed data as output, so an object tree.
And in case of KeitaStruck, the generated parser works with the runtime library so you
need to include it also into your application.
Why use Keita?
What are the advantages?
So as I already mentioned, the advantage is that you write the KSY specification once
and you can use it everywhere.
It standardizes the way we describe binary formats and there are already many formats
described in the Keita format gallery and any described format can be visualized automatically
in a Gravis diagram and the KeitaStruck language is simple, you will see.
There are also several visualization and dumping tool available in KeitaStruck.
So the write once use everywhere feature means that you get parses in 11 programming languages
for free from a single KSY specification.
So in this case, I've had the compiler generate Java, Python and Ruby parsers from a simple
KSY specification you see on the left.
When you look for specifications of binary formats, you will find that each one looks
different and there is no single standard to how to document formats and Keita is used
or intended primarily for creating parses but some people write KSY specification just
to document a format in an easy to understand way because you don't even have to be a programmer
to understand a KSY specification and it's often easier than to read these long PDF documents.
And the Keita project includes an extensive gallery of described formats.
At the moment, there are 181 formats described by 76 contributors and there are also several
hundreds more format specifications in various Keita projects.
And so the Keita format gallery contains formats of various kinds, for example, as you see
archive files, for example, executables, file systems, game data files, multimedia files
and network protocols, you can go to this page and I took it from there.
And this suggests the wide applicability of Keita.
And it offers an idea to create an international database of formats where various obscure
and historical formats would be documented in a uniform way for future preservation.
And this would guarantee that we could basically, we could read the binary files we write now
in like 100 or 200 years from now.
The fact that the Keita extract language is declarative makes it possible to automatically
visualize it, visualize the described format in a Gravis diagram.
The Keita extract language is simple but powerful.
You can describe pretty much any binary format with it.
And a case one specification starts with the meta section and this sets the little end
in byte order as default.
The SEQ section is a sequence of attributes.
The attribute name is in the ID key.
The type U4 means that in this case num underscore files will be an unsigned for byte integer.
You can define your own types in the type section.
A field can also be repeated.
So in this case the files attribute will be a list or an array of base type file.
In the instances section you can define attributes that start at an arbitrary byte position.
You can also use a powerful expression language in many places.
And there is another built-in type is a character string in a certain encoding.
And if you omit the type and only specify the size, the result is a byte array.
There are several visualization and dumping tools available for inspecting files.
And this can be useful for, for example, for finding errors, forensic analysis, or debugging.
And the visualizers allow us to view the structured data parts from the input file based on a
kitesh track specification, so something like this.
And you can use the console visualizer or also the command line to case dump is available,
which can give you the same structured data as you can see in JSON format.
And this can be useful for automation.
But the most popular visualization tool is the Web IDE.
You can check it out on this URL.
And at the top right is a hex-dump of the input binary file.
So in this case I selected this.png file in the file tree on the left.
And at the top left is the kitesh track specification editor, so a KSY spec editor.
And according to the kitesh track specification, the input file is parsed and the result is
the structured data that you see in the object tree at the bottom, bottom left.
And when you edit the kitesh track specification, the input file is automatically parsed again
and the object tree is updated.
Serialization is a new feature in kitesh track and it's being developed thanks to the financial
support of the NLNET Foundation.
While parsing allows you to read binary data to an object, serialization is all about
the inverse process.
So we want to write an object to binary data.
And currently in kitesh track, the serialization for support for Java is fully working and
C-sharp and Python are in development.
There are basically two use cases of serialization.
You can edit an existing file or you can create a new file from scratch.
And the support for serialization greatly extends the use of all written format specifications
because now you can use them not only for parsing but also for serialization.
And this has many uses, for example, you can convert one format into another or it can
be used for fuzzing or video games modding and so on.
This serialization process in kitesh track can be divided into four phases.
First you need to create a ks object and then you fill it with data.
So you set its individual fields or attributes using setters.
Then you should call the underscore check method to check the consistency of the data
with the format constraints.
Finally, we can call underscore write and pass the stream where to write.
And you can actually check out more details of how to use serialization in Java on this
page.
Currently, the serialization support in kitesh track is designed for the general case so
that it works for every conceivable format specification.
While a simple solution would work for perhaps most specifications, well, the solution that
works for all of them was chosen.
Even at the cost of delegating some task to the user.
In the future, I would like to automate these tasks that need to be done manually at the
moment so that it's more convenient for the user.
The basic idea is that the user sets everything, including lengths of sets, magic signatures
and kitesh track checks for consistency.
Also, only fixed length streams are considered.
So once you create a stream, you cannot resize it.
Finally, I would like to talk about the plans for the future.
Design for C-sharp and Python is in development and they should be ready in two months.
There is also interest in adding Rust, C and Julia as target languages.
And I would also like to see Wireshark desectors as a target because the concept of kitesh
is not limited to programming languages.
A target can be anything, for example, we already have a target for construct, which
is a Python library for parsing and serialization of binary data.
Thanks for listening.
Now it's time for our questions.
Yes, there is a dock key for which you can use on attributes and types in many places
and you can write some documentation of the specific element and in some languages, but
it doesn't work like 100% of the time, but the idea is that these documentation should
translate to the generated parser as dock blocks and then the IDs and tools for development
should autocomplete usually this documentation.
Do you support in DNS when generating source code, depending on the target machine?
Yes, there is a feature for calculated NDNS, it is called and you can switch the NDNS or
the default NDNS based on the value of an arbitrary expression basically, so this can
...
But do you support host NDNS and target NDNS?
Well, not really, but it's not that of a limitation because you can, for example, you can use
parameters, for example, to pass it from your application basically because I don't
know if I can... I don't know if it's a good idea, but another feature of KiteStruck
is that you can define that types can have parameters and even the top level... Yeah,
I should probably at least... Never mind, yeah, and you can define parameters and you
can easily pass a parameter from your application that will somehow change the behavior of the
specification over, yeah, so it's possible.
With KSI, you seem to aim to define specification for certain languages or formats, but for
languages and formats that already have a specification, how can you ensure that these two specs are
actually the same and that you're not passing differently than other parts of it?
I don't... Well, you mean that there is already an implementation of some...
For example, someone's passing ZIP files out there, how do you guarantee that KiteStruck
will pass ZIP files the same way?
You don't basically, but from this point of view, it's just another implementation... Well,
if you compare it to other parsers, for example, so there is, for example, a ZIP parser in
every language, yeah, so ZIP parser library and this KiteStruck specification, it's just
another implementation, so, well, it needs to be developed carefully so that it works
well or... Yeah.
I guess you would need a way to translate from a written specification to the KataI structure
or the other way around to validate that what you wrote as the script actually corresponds
to the actual specification, for example, if a specification is already matched in machine
written, which is readable, I mean, it's not here, we should have a tool to convert from
one to the other, so that would ensure that the passing is correct.
But it doesn't help, because the implementation is done by humans, it's impossible, it's
impossible.
It's just an introduction.
You have to run all those things.
Why?
I'm wondering if it would be possible to add some functionality to that, not only parsing
but some very common functionality, do you think you can add that in the highest form?
Common functionalities, so... Like, for example, there's a binary format and there's
very common functionality everybody uses on that, let's say, like, I don't know, cutting
a part of it or getting, calculating some, I don't know, value, magic value or hash value,
could you add some extra functionality other than parsing in there?
Well, so the question was that if you can, if we can add some common functionality in
addition to the format specification, and the answer is that, well, you can do this
to a certain extent, because there are, I didn't mention them or talk about them, but
there are value instances, and you can prepare some, you can, this is like a calculated attribute,
so you can write an arbitrary expression to it, and this can calculate, for example,
some, like, I wrote a, I wrote a BMP specification or I extended it, and I used this, for example,
to, well, in the BMP format, there are like color masks in different places, depending
on the head version, and I used a value instance to get it from, so, depending on
the version, so either get it from here on here, or if it's a fixed, fixed, I don't
know if it's fixed core palette, or what is it called, so, yeah, we can do this to
a certain extent, but some common functionality, like, I don't know some, well, if it would
require, like, a programming language or something like that, so this would be infeasible, basically,
because then, then we should, we would have, we would have to some, some programming languages,
something language that translates to all targets, which is basically impossible, I
think, yeah.
There is some different type of learning, like service, you know, testing, you are
this tool set to write a comprehensive diff tools
that explains the differences between two binaries
and that can leverage the existing descriptions
to explain what the difference became to find.
Yes, so I think you can compute some diff.
Basically, I would do it, I showed the ksdump tool here.
So I think you could generate the gson dumps of the two files
and compare them, but when I did this,
it was usually very, very massive,
but you can probably improve that somehow, I don't know.
But it's, yeah, okay, so thanks and...