Okay so we start again. Our next talk is about liquid soap. Please welcome Romain.
Hello and good morning. I'm really happy to be here after not two but three actually.
And we gave a talk here before the whole thing happened that I don't want to talk about.
A lot of things happened and I want to go back to it and talk about what we did both in the code
and with the community and the kind of things that happened. So yeah first of all for those
who weren't there at the first talk, what is liquid soap? It's a programming language.
Technically it's a scripting language, scripted language. It's statically typed with inferred
type. So if you're familiar with type script, it's type script but you don't have to write the
type so everything here is a string, an integer, something. What does the language do? It allows
you to create online stream and it's not a low level tool. So we delegate that and I'll talk a
lot about it. What we want is to empower the user to rezone about the tool. That's what the language
really does well. Programming, logic, business logic. I want to play that song. I want to switch
to a live source. So that's an example of a full code that we can use to run two outputs on
IceCast based on playlists, a file, a list of requests, an input from HTTP, all sort of stuff.
Yeah. So what has happened with liquid soap since first time 2020? We worked with Radio France.
That's the reason we came in 2020 and it was that starting point of a lot of new work because it
really created a new cycle of work interest. We had a lot of community growth that reflect back
on it in the first part of this talk. Then we do a lot of new features because guess what? We also
had a lot of time during that time and we worked a lot on it. And I want to finish the talk by
talking about future development and challenges that we foresee for the future. So first of all,
what happened with the community? So I started looking back at the stats over the past three
years and I looked at the GitHub stars. It's nice, but it was growing pretty steadily except for
that little bump here when we did the 2.0 release. And I looked at the Slack channel,
which I want to move out of this platform. But anyway, it was also growing pretty linearly.
I was like, doesn't seem really anything happened over the past three years. Then I looked at the
issues. And that's where I was like, yeah, I remember that. So what happened is when the whole
shutdown happened, because we are a project that enables people to communicate online,
well, it was one of the places that people wanted to go to communicate with people and to try to
reach out, create link, maintain link. So we had a lot of people suddenly who were like,
I want to do an online stream. I want to put music for my friend to listen to. I want to do all
these things. And the second effect, we had a lot of people with time. So we're kind of one of
those projects where you're like, oh, we like it. I like it. I'd like to look at it at some point,
but I'm too busy and suddenly I have time. So people starting using it, submitting a lot of
issues and well, we got busy. One of the things that we deal with so is because we didn't have
that kind of in-person meeting, someone who's the other code developer of the project decided
that we should do an online workshop, which was the thing to do. And it was really good because
what it really allowed us is to get to meet our community. And I think one of the things that's
really nice with this project is that it's both a technical project, but a lot of our users are
actually not technical people. They're like people who have Burning Man Radio at the Burning Man
thing. Some of the cute project we had was like, the top one was a network of community radios
in Barcelona where they literally have trucks that they bring to different neighborhoods to
do online reporting of what happens in the city. And the lower one was another Hungarian
community radio. So that was really good to get to meet that. And I think we valued that a lot.
We have, of course, we have industrial users, but this is also one of the core motivations for
the project for us. And then, eventually, another thing we did, again, by product of having a lot
of time, which was mainly some working on it, was to write a programming book specialized in
audio stream, media stream, and how to use liquid soap for it. And it was very useful on many
levels. One of them is that it forced us to rethink our API and reorganize it. So you say,
can we do that? What's a good example to do that? Well, actually, it's not clear. Let's make a nice
module that can do that easily, document it. And also, it enabled the users a lot to be more
confident because, as I said, most of our users are not programmers. And so they come to this,
and they're like, whoa, never touch a line of code. How can I do? So that book was a really good
starting point to get people interested and more confident with the project. So what did we do
for the new features, though? That's the part that the gist of the work we did. So first of all,
we did a lot of language changes because for a long time, our focus was more on features. We only
did a lot of output, a lot of input, a lot of different interesting operators. But when you
start to want to implement things that are more complex, you also need powerful language extension
or toolbox. And also, we did a lot of FFMP integration. So those are the two things that I'm
going to talk next. And first, the language change. So yeah, more expressivity. You want to enable
your users to write code that is nice, readable, that they can understand, that is powerful,
but it doesn't have to have a million lines to do a simple thing. So simple things like that.
These are like spreads, you know, to split out a list. You have a list, you want to get the
first element, last element, and whatever else is left as a remainder. We could do that before,
but we had to use functions and it was complicated. This really helps users just visualize the code
and be like, all right, understand what this is doing, and have much less lines to write it.
We implemented a lot of types that were missing. So for instance, a null variable. A lot of that
will remain a sense of, of course, dynamic scripting languages like JavaScript or others. So we
didn't have a null value. We did a null value and we added a nice operator that says if you have a
null on X, if X is not null, it's a string, then you get X. If it's null, you get the string full.
Stuff like that. But when you start writing actual usable code, you need these things and they're
very useful. Yeah, you can write function with optional argument and you can exactly say the
user didn't pass an argument. That's just basic programming language features. The other thing
we did that was very useful is a new module syntax. That basically means that any value in the
language can be attached methods or attributes to it. So here you have a string and that string
has two methods. Well, attributes here, duration, floating number, and BPM. It's basically an object
oriented ID, but not really an object. More like JavaScript hash, I guess. And that means that you
can query the song.duration and get the floating point duration or you can print the song title
as a string using the underlying value. That was really useful because when you want to create
something that's easy to use for user, you need to structure your language. You need to have modules
and you need to have functions in those modules. Also, another typical case is that now you can
specialize things. So you have sources that have different functions. So all sources have a skip
function. It means that you can get to the next track in your stream, but you may have sources
that have specific function. So for instance, you can have a source that can insert metadata
that's created with this operator. It will be added a new function that's called insert metadata,
and now you can use it. So instead of having a billion of general calls that are API based,
you can start really attaching specific use to specific variables and it makes things more compact,
more specialized, and very useful for cleaning up your API. Another thing we did after that
with the module is that now we were able to describe high level things. So one of those
things that's really painful in static ID type language is parsing JSON. Why? Because JSON can
be anything, and we really want to know that name is a string. We really want to know that name is
a version. So in language like OKMOR you have to parse an object, iterate through the keys,
validate that is a string, and every branch of that you need to think of what you're doing.
And our users, again, not programmers. So what do we do? We try to find a primitive that's
readable and easy to use. So what are you reading here? You're reading a parsing statement that
says I want to parse and I want to get a module that's going to have a name, a version, and another
script attribute that is itself a module that has a test function. It's what you get in a package
that JSON for node modules. And this is a type annotation that says I want the name to be a
string, the version to be a string, and the script to be a module with a test that's a string.
And at runtime we're going to take all this information, we're going to try to parse this,
and we're going to do two things. If we have what we need in the JSON, you can go on with your
script. If we don't, we raise an error, you can cache the error and reason with it. But that
really makes parsing of JSON more easier on the user, which is important because, again,
when you want to connect a lot of interconnected systems for streaming, you want to be able to
talk to JSON API. So pretty useful. And we did the same for Yemo recently. So yeah,
you can do settings now. Yeah, those are the new features. There's more, but I don't want to spend
too much time on it because the other part that I want to talk about that's exciting is the FFMPG
integration. And that really started after Radio France because they had a strong need for it,
and we started looking at the API from FFMPG. And the thing with the API from FFMPG is that
it's really good. It's amazingly good. It's all very low level in C. So for us, because I didn't
mention, but LiquidSup is implemented in OCaml, so we need to speak to low level implementation
to really be efficient. We can talk to Pearl or to whatever, Python. And this C, it's really
simple for us. But it's also extremely abstract. And that really helps us because, again, we want
to do what we do well, which is the programming language side, the logic, the typing, the functions.
But we're not specialists in multimedia implementation. We don't want to do that. We want
to find people who do it better than us and interface with it. So that's the API for a FFMPG
packet, which is a little tiny bit of encoded data. It contains, I don't know, an MP3 frame,
a video, A or B frame, all the abstract things I don't want to know about. But it's going to
tell me two things. It's going to tell me this is your data. This is your pointer to a presentation
timestamp. And that thing here tells me this is the time at which you want to insert this packet
in your stream. This is the data. That's what I need to know. Because then we can build a stream
with it. We can pass the packet around to our different operators, not even knowing what they
contain. The other thing that FFMPG really, yeah, so that's why we started doing, we started
implementing a new encoder that was basically reflecting everything that you see as a parameter
in the FFMPG command line. We think that we support it as an option in those encoders.
And why do we think that is because another thing that FFMPG really does really well
is describing their API. So I'm sorry, that's not very readable. But that's a C structure that has
all the parameters name here for H264 encoders, description, type, somewhere, and minimum,
maximum value, everything we need to basically write an interface to it. It also does it for filters.
Again, not very readable. Sorry about that. But it's basically a programmatic interface
to everything you need to know about parameters for FFMPG. That's also not readable. Great.
So then what we can do is this is an FFMPG filter implemented in liquid soap. And if you could
read well, you would see that every parameter is in the filter, like speed, is a floating point,
optional. It has no value default. They saw no value default. Sometimes they don't.
We get this information from the FFMPG C API, and we can plug into it immediately and be very
confident that we're using it well. And so one of the things it allows us to do is to actually
have scripted manipulation of FFMPG primitive like filters. So we take a source and we want to
define a filter that is a flanger followed by a high pass and then output it. So you need a graph.
That's just part of the C API. You need a source. You create an audio input from it. That's the
FFMPG side. That's what they call an audio input. You pass it to the filter with the parameters.
Everything is typed here, so we can check. That's the right value. And then you create an output.
Run that. You have a high level description of your filter. I don't know if you all have manipulated
FFMPG filters, but when you want to do complex filters, they have a description graph that's
pretty hard to read. I'm used to that, so I'm biased. I can read these things easier, but also
it's kind of more descriptive. It's typed, too. So this is another filter. It takes an audio, splits
it into two sources. One of them is going to go through a flanger. The other one is a high pass.
This is some conversion that was required. I don't know why. And then you merge them back.
So we're describing now a graph that branches out, do two filtering and comes back together.
This is a simple one, but you could use that to do, I don't know,
a multi-band compressor, for instance. You can do that in FFMPG. It's just a little bit more
structured and readable and also type safe. So next, I want to talk about how we implemented that.
Yeah, I still have some time. This is the timeline for us, infinite time. We started here,
and we go all the way here. If you use your imagination, all these little horizontal dots
are audio packets that came from FFMPG. Vertical ones are video frame. This is your stream of data
that you're sending to an ice castle, anything. What we do is that we find the lowest common
denominator between the video rate and the audio rate. We need to find a little chunk of time that
will contain the same amount of audio and data. Most of the time, with the parameters we have
internally, it would be 0.04 seconds. That's what we call our frame. And then, the idea of the
streaming loop, once you've parsed and prepared all your outputs, is to just recreate that frame
every 0.04 seconds, infinitely many times. That just creates your stream. And so,
here's an example. We have a simple script. It has two outputs. You want to save to a file and
send to an ice castle server. Fullback is an operator that will take the first source available.
So, the first one is an input. It's a hardware. So, it's one of the operators we have that can
receive ice cast clients. So, let's say you want a DJ to connect to your radio. You can direct them
to this input, and it starts streaming. The fullback will stream that data immediately. If it's not
available, we have a queue of requests that you can. So, let's say you want to send a track to
be played immediately after the following one. Every now and then, you can send a request here.
Otherwise, we have a playlist of just files in the directory. And if that is not available,
we have some kind of fullback. Just in case everything fails, something is going to say,
I don't know. We're having technical issues. Please come back. So, now we're going to run the
streaming algorithm and see how we do that. So, output.file starts. It always goes back from
the output down to the inputs. And the reason is because all of that is dynamic. So, there's no
reason to start asking these people here to prepare data. They might not be used because,
up in the graph, the fullback might choose to use just one of them. So, we have to start from the
output. Bring it down. So, I have an empty frame, a cycle, 23 sudden streaming cycle. And I want
to fill it up with 0.04 seconds of data. I go to the fullback and say, hey, can you fill up this
frame? Fullback is like, sure, let me ask first, input a hardware? Not available. But request.q
has something in the queue, actually. Let me pass it down, pass it down to this operator.
Request.q partially fills the frame. It added a little bit of audio and one video frame.
What you can think about it is that it was just finishing a track. Remember, request.q takes
files when you want to play them. So, I don't know. It's just done playing the jingle or the
commercial that you wanted to finish. That's it. I'm done. So, it's a partial fill of the frame,
in which case it comes back to the fullback that says, I need more. Playlist is not available.
For some reason, the directory is empty. So, we go back to the single and say, hey, single,
can you fill this frame? One of the things we do when we start the script is that we actually
double check that we have at least one operator here that's always going to be available. So,
that's what happens here. The fullback is being used. And single is, sure, I got a file. I prepared
it. I can decode it. Boom. Finish filling up the frame. And then it goes all the way up the tree.
We're ready to encode that, save it in the file. Now comes the second one,
which is the iSCAS output. Same thing. It's like, hey, I need to send data to my iSCAS server.
It goes back to the fullback. But then what we do here, of course, we cache stuff. So,
we know that fullback has already produced a whole frame. We have saved it. We can just fill it up
here, send it back to iSCAS, again, encode it if needed, send it back. So, the thing that's really
nice with this algorithm is that, again, we don't really care about what's in the frame. We just
care that we know that we can fill it up and we know how much is filled. And then we can pass it
down. So, these things can actually be FFMpeg encoded packet. No problem. They can be raw PCM.
Then we have to encode them. So, it's really, again, we are just looking at things from a
high-level perspective. So, if the time for that whole cycle was t, we have two possibilities.
If we generated the 0.04 second in less than 0.04 second for the computer, it means that we run
faster than the real-time. We can sleep a little bit because we're generating things in real-time.
If not, we have a problem. We need to catch up. So, we need to run the loop again as fast as
possible. Basically, if you're in the red, you have a problem. It means that, I don't know,
encoding takes too long. Something happened in your script. You cannot produce in real-time. What
we want is to be in the green all the time so that we know we can deliver the content at a
real-time rate all the time. So, that's how it works. Now, because I told you all that we now
can use encoded content, we're having a little bit of problems that I won't have time to describe
here. But essentially, sometimes we have to do a lower-level understanding of what's
happening in the bitstream. So, basically, in ffmpeg, you have things that's called extra data.
So, if you take an mp4 media, everything that is needed to decode like fman-table is in the header,
it's communicated first, and then all the packets, the frames after that, don't have it. But if you
take mpeg-ts, because mpeg-ts doesn't have a global header, every frame is going to have that data.
And that was a problem because now we do things that ffmpeg doesn't do, which is a live switch
between two different bitstreams. This is a RTMP input, this is a file, mp4. And when we live
switch, if we started with this one, for instance, the muxer from ffmpeg might say, oh, you know what,
I already have the, no, if we started with this one, the muxer will say, I know that all the frames
are going to have the private data that I need, I can go on with it. And then we live switch to that,
and suddenly we start receiving frames that don't have it. And the muxer is like, whoops,
I cannot do it. So we have to insert those filters here to make sure that they're always present.
Which is a problem for us because it means that the user has to think about low-level stuff.
And that's part of one of the questions that I was wondering if ffmpeg might find
some of the beautiful abstraction they have to alleviate that kind of problem.
All right, almost done. I'm going to finish real quick. We have a 2.x release with tracks so you
can manipulate and remix different tracks. Let's say you do an MKV with French, English, and Italian
soundtrack and encode it in different format. We want to rewrite the whole clocking system
and streaming because Okamera 5.0 does concurrency. It's pretty exciting. We want to think about
JavaScript and YSM because we can do it and why not. And I'm very interested to see what
VLC does because that's the next part and why they do it. And long-term development though
is what we are wondering about because we have grown a lot of community, but most of our users
are not programmers and our programming language is OKMO, which is still a niche language.
And we're two developers, someone and myself. So one of the questions, you know, the project is
20 years. We're 40-year-old. At some point, we need to think about moving on. So what we have
done so far is do a lot of automation, which is very powerful. It allows us to focus on the code
and have less to think to do. Like, we have automated release. We do CI to run all the tests
every time. We have augmented the number of automated tests we have so that we catch everything
very quickly instead of relying on a lot of manual testing. But it's far as a challenge to
think about how we can bring in more developers that like OKMO, like radio, is a pretty intersection
of two niche things. And that's it for me. And thank you very much for your time. And maybe there are questions.
So we have time for a couple of questions, yes?
Yeah, so once you go from the debumster to the coder, it's expected to format the entity.
Then the same thing on the mob suicide. So if the expected output format is something else,
so it will actually tell you that you're expected to insert a bit stream filter again to get it
forward. Yeah, it was actually an answer. There are APIs in FFMPEG to inform the user of the
C API whether or not they need to insert those filters. And I see that there is automatic
insertion of those filters in the code. I guess I have to have another pass at making
sure I understand when and how. And I regretfully for this presentation I didn't dive again
and I remember that sometimes it becomes a little bit intricate. But thank you very much.
I will definitely have a good look.
Is it possible to control the running script using an external automation, especially like editing running playlists?
Yeah, absolutely. So are there any tools to control a running script from the external user?
So we have a lot of different options. The traditional one, the old one was a TerNet connection.
But we have a fully featured HTTP server, so you can write your own API endpoints.
And basically because every source has their own methods now that are attached to it,
and you can run scripts, you can basically take a source and say,
okay, there's a function that inserts metadata, plug that into running everything else.
We're programming language. So yeah, we have a lot of different options for that for sure.
Can you use LiquidSoup to do static operations, not to compare files,
and not with the goal of doing streaming, just to apply operations on many...
Yes. The answer is yes, because the clock doesn't have to be...
Oh yeah. Can you use LiquidSoup to do offline processing faster than real-time and not just real-time?
The answer is yes, because you can run a clock that says,
I don't want you to be real-time, I want you to run as fast as possible.
But the sub-answer is that it's not a common case, it's not commonly tested.
And that's part of the thing that I want to bring on more when we write the streaming.
Because for instance, a request needs to resolve a file which needs to make a network request,
download the file test that you can decode that.
Most of the time we expect that to take a while,
but it's not been tested a lot when you want to run very, very fast.
And you run into race conditions that are very different.
Okay. Thank you very much.
Thank you.