Yes, so I'm Mathias Valdman. I'm the author of this FQ tool. Now I'm going to do, I'm
not going to be able to show all slides, I think. I want to do a lot of demo, because
I think it's a good tool to demo. Sorry, can you talk a bit louder? I can try.
What is FQ? It's a tool and the language and the group of set of decoders to work
with data. More or less it used to be binary data, but now it's also text data,
but I guess it's binary also. In a sense, so it's heavily inspired by JQ, both as
it is JQ, the language, but it's also the CLI tool. It's inspired by how
the arguments work and everything. And it's a tool for querying and displaying
data, exactly like JQ. But it also has an interactive weapon, so you can work
with it in a more interactive way, and it has auto completion and a lot of other
bells and whistles to make it very nice to work with. And it's available for a
lot of operator systems. So I like to call it like, it's like a debugger for files.
That's how I use it. So why JQ? It's a very CLI friendly language. You don't
need any new lines. It's like you, it's like the syntax is very, very tears. You
can do a lot with a little syntax. And it's very composable. You have this
like pipe, more or less like shell pipes. But it has these generators. You can
even do loops and iterate and recurs with even this, using these pipes. You see
these square brackets like that to iterate and dot, dot is to recursive,
recursively through a tree of values, you can say. And it's also kind of like DSL
for like selecting and transforming JSON. It is, you can call it JSON. It's
not, it just happens to have JSON as an input output kind of. Internally, you can
call it like JQ values. So it has arrays and objects and numbers and strings
and boots. So it's kind of like a super set of JSON. And I have an example here
that you, you build a new object that has a key A, that is an array, that is one,
plus two, plus three and empty, and you see that it becomes the object with just
A and array one, five. And empty was just a function that does nothing. It outputs
no, no value at all. And you can also select values from the input. So here we
have the object with A and B and then you create a new object that has this, the
sum key. And then that's the sum of A plus B. So then it's three. And it's a
purely function language based on generators and backtracking. But it has
conditioners, it has function, it has bindings, it has all, all things you need.
And it's, the default is that you run, you have one JQ filter or program that you
run individually on each input file. But then it has, it has functions to tell
it to not behave like that. So you can run one filter on a group of files, for
example. And FQ has support currently for 113 formats. And it has most of them, I
guess, half of them are media related because I work with media. So just like
MP3, MP4, or Flack. And they also have support for like demuxing some of these
forms like you, some of these containers they have support for segmenting and
things. So you can kind of recombine it and then decode the, the demuxed sample
and things. But they have been, other people have added other things. So
executable formats, archiving, networking. So you can do PP cap nowadays, you can do
TCP reassembly even, and even decode the TCP stream. So it's a support for
RTMP, for example. And it's also a support for some serialization for like
message pack and the ASM1 beer and the seat, seabor and those. And it's also a
support for some text format. Some of them you can even decode and encode. So
you can decode it into a JQ or JSON value, transform it with JQ and then
encode it back to some other text format. You can't do it with the binary
formats. I will see if I get to that to explain why that is, that is not easy to
do. So what does it mean when you decode? What does it mean that you FQ
support the format? It means that it can, there's, there is some code is written
in Go. So there's like a kind of a DSL for writing decoders. And that produces a
structure that is like JSON compatible. But each value in this structure also
have no which bit range they come from. And they also have optionally can have
like symbolic mappings. Like you can map the number to a string or string to a
string or boolean to a, so you can in binary format you usually encode some
number that means something. Like this is the upliver if you get that. And for
media this usually means that it's, you decode everything except the pixels or
the audio because it's like, yeah, then you buy, then you can use ffmpeg or whatever
you want. There are some format that actually decodes to the samples. Flack for
example we have support for actually coding. So there is a full flack decoder in
Flackube but you can't, you can't listen to the sound. And some format can use
other formats. That's how they are. They're like hierarchy over that they use. So you
can even type in pfmpeg users, the internet decoder using IPV. And then you
can even end up with the loops here that they're like zip files in the zip file.
And yeah. And there's also support for formats to pass. You can even return
value. You can return like values from a decoder into another decoder. You don't
see this from the outside. But then for example mp4 has some boxes that have
like information about how the samples should be decoded. Like how long are
some fields or things. So then before and before the decoder can pass that
information down to another sample decoder. So how do I use? I use it because
of this. Because I work with media so all more or less the whole mp4 file is
metadata about how to play this. How to seek, how to how to sync, how to yeah
everything. And I guess Derek had a good talk about why this multimedia is
basically endless pain. And I guess FQ can't really fix the pain but it can
locate the pain I guess. So this is what I use it for. I debug the query assist
when I work with media files. And it's used for someone usually at work there's
a media file that has broken that someone it doesn't transcode or it's not in sync
or and they say what what is wrong with this file. And we have to figure out
like what is it? Is it the decoder problem? The maxi problem? Is it the encoder? Is it
is it corrupt? Is it whatever? So FQ is very useful to quickly triage a lot of
broken media files. And I can just short what is the time? Short like what can it
not do? Or what is not good? It's not good for encoding like to change things.
You can change things but it's more about slicing binaries into and then
concatenate them together and then write them out to a new file kind of. So there's
no like you can't just change the value in some JSON structure and just realize
it back because it's like for example I gave an example here with the mp4
maxi. What would it mean to add or remove a sample in mp4 file? Then you
would have to change all the boxes that describe how big the samples are and it's
just cascades away into yeah and you can see that mp4 ffmpegs mp4
implementation is 17,000 lines of c-code very dense c-code so it's use ffmpeg
if you want to do those kind of things. And I think you see and Peter will talk
about more about encoding I guess and why why this is complicated. And you can
repair media files with fq but you you probably have to be more or less an
expert in the format that you're fixing. I usually do like I have you you can
fix things by kind of testing the code the configurations or you use some kind
of you encode something and borrow from another media file and yeah
somewhere stitch together to see what it is or so there is I have some fq code
to build like ADTS headers and whatever if you want if you find an AC frame
somewhere you don't even know what because an AC frame has to have metadata
about like what how many channels it has and what profile it is and things. And it
doesn't do any decoders in runtime at the moment you can't write the decoder
you have to write in go now and then compile it so we'll see maybe in the
future it's going to have cacti support. I have to have a prototype for cacti but
it is complicated also it has an expression language I have to write
the parser for that. Maybe I will talk to you about this and see. There is more
slides here but I will you can read them. I want to do a demo instead. Is it big
enough? So fq is just a CLI tool. Work like this and you can do if you want to
list all the formats that it supports. So if you run fq and let's see it has pcap
if you do so the first argument is the filter that you want to run and dot in
jq is just an identity function that is just it gives you what you put into get
the root kind of so here we see that it's a pcap it has a head there it has
packets and some TCP or TCP connections and things. So you can do dash I actually
I want to do I want to do a crash course in the jq. I don't know how many people
know how jq works so I can do a short version how just to show the
particularities of jq. So now I started jq with dash I which gives you
just a null input if you get one input that is just null so it's just a way
of you can at least just execute jq values because it needs to have an input
somehow always. So now you can just write strings or whatever you want to do
and if you do dot there you just yes null and I guess the most special thing
about jq is the comma operator and that outputs a value so you can do 1 comma
1 2 3 it gives you 1 2 3 but then in jq you can there are some special forms
like this collect which sounds this looks very familiar to us as an array
that then collects those values into something but then in jq you could you
could just write the expressions here or whatever you want so let's say we can
define a function for example so and then you can just collect that function or
you can call it two times or you can map those values yeah so you see how it
was too fast to but you see how you can define function you can have binding
so it's a full functional language that's it's very I like it a lot how it
works so let's back to that pcap file so let's see you can do for example if you
want to see look at the first packet you can do this you can pipe it to D which is
shows more recursively all the if you if you just give it when you do this you
actually run D also but it also show it only showed one level so if you do D it
it will show all of it so here you can see like you can write on jq
expression here it is like show me the first and the last packet and it will do
this and then you can say both of them yeah so you can do things like that so
let's see we can go into the TCP connection and we take the first one we
can do D and we see that this is seems to be HDP request someone has downloaded
the file so let's see we can go to the server stream let's see and there is a
there's one thing about jq is that the jq don't have doesn't have binary support
it only has string so you so fq has to to support binary you can there is some
special functions in fq to tell it that this this string is actually binary or
I want it as binary if it's possible so then you can say like this stream for
example if you if I would just do type it just a string but if I do two bytes you
will actually get the raw bytes and then I can say like I wanted to maybe 400
first and DD is something that shows the whole it doesn't truncate the output so
here we see that this is some kind of HDP request
so let's say we want to get the bytes for this okay plus so here I think I have
the body of the HDP request so in jq you can more than do for example this that
all all the all the coders in fq are jq functions so you can do this now and it
will just decode it as an mp4 file so now you can start as a sub-repell for
example now you're inside the mp4 inside the the pcap and now you can do and
here you have the whole box 3 for the mp4 file and and you can for example go
into the samples I think this is some kind of subtitles for mp4 file that I
found somewhere and here you have the tracks it has samples so this is like
the raw the raw bytes for that sample and you see it's some it's some kind of
weird XML subtitle and fq has support for XML so you can do this and then you
get a JSON version of the XML see we can see almost there is some let's see I
can write something that takes out all the this is probably not how you write
TTML subtitle parser but we can we can do a quick quick version of one and there
is some functions like grep by that recursive look for some condition so here
for example you can look for
did not work
why
aha
so so now we it recursive defined all those text the objects that had the text
field and then just take took that text field so now you can for example take
this expression go out to the you have the wrap up here
and you can go out to the prompt again and then remove the interactive and then
do this instead or we can do this if you only want text and we can even say
we want all all samples so here is all that that's the thing you can do with
with all these decoders that the codes and blah blah and then you can
iteratively do all this it's like yeah it's nice so let's see and I also want
to show that you can you can actually you don't have to write this all this
when after a while you maybe with your expression starts to get very long so you
want to have more structures I can show you I have some helpers for mp4 files
for example maybe you know so I spend a lot of times in mp4 files because that's
like what is used everywhere nowadays so here is some helpers for example that
can be written in a more structured way that with the indentation and things so
you mean you don't go crazy and I can show here for example this is a bit
long but for example here is an expression that loads this mp4.jq and
it finds the video track and then it uses some jq code to produce a
GNU plot output and it uses a GNU plot and then I have my weird tool I have for
producing images in the terminal so so here's the bit rate for the video bit
rate per bits per second in this video and you see that there's probably this
top sear up over the iframe in the mp4 file maybe we can take questions then yes
yeah you can use that's a hex editor also maybe it's not very convenient as a
hex editor but you can for example you can do dash d bytes or bits then you get
like it's just a dummy decoder kind of so you can do bits a lot be kept so so
now you'll just get it so now you can do so and then you can concatenate this
into you can the kind that you can build this like a binary arrays like one they
are like iolists in Erdogan if you have used that that you can like you it's
just an array with with things that can become binary and then it can and you
can kind of pipe this to two bytes and then you get back so you can kind of
do this yeah so you see that if you can you can build this that's how you can how
you can repair things with fq that you take the samples and then add on some
head there and then you concatenate the bytes and you can even decode it with
the jq again so you can i mean you could you could try decode this as an mp3
file for example but it yeah it won't work yes i would like to have some more
round timey version i can't show how there is one of the slideshow show
actually how the dsl how the how the go along dsl looks like yeah here here is
how kind of how the go dsl works that you there's like a d function of this
kind of like that the the context for that keeps tracks of where you are and
then you create like new structs and fields and and you have these mappers
that say like this is a binary for made up binary format for live by for open
source licenses that I was going to be a moment there so that's how how most of
the code looks like so if you can do you can write any go code i mean i prefer to
write them to make them look very declarative like so don't use too many
yeah too much weird code in keep them simple is the and other questions yes
it's i would say that it's probably hard to of course to write the code to the i
think it's it creates a lot of options how what what does the user mean when
they change something like do they want yeah it's also like do you want the
checksums to be recalculated or not do you want what happens if you change
something in a if you have the max something and then you change the size
so now the segmenting becomes different so then it cascades to change the whole
file so it's like do you want that to happen it's also like there are
encodings that are there are like many ways to encode the same integer for
example var ints can be encoding many many ways so you would encode and
normalize all that or should it behave should remember try to remember how how
the original thing was decoded yeah so it's it's complicated any other questions