Alright, okay. My name is Swim Taimans, I work for Red Hat and I started writing pipewires
some seven years ago, I don't know anymore, way too long. I gave a talk about pipewires
last year, so basically it's a follow-up on that, a little bit of things that happened in the last year.
For those who don't know what pipewires is, it's basically a multimedia sharing and processing engine.
So pipewire was originally built to send video frames from Wayland to applications because screen sharing
in Wayland was completely unimplemented in anything, so there needed to be some way of funneling those frames around.
It went to a whole bunch of iterations to make that happen. It started with G-streamers, some custom implementation,
and then version 0.2, which is something that sort of worked, and then it sort of devolved into an audio framework
because people think pipewire is for audio, but it's actually more for video.
So it devolved into an audio framework and here we are now.
So basically the core of pipewire is to link applications and hardware into a graph.
It's very similar to what G-streamers does, you make a graph of processing elements.
In pipewire's case, this is distributed, so it's an IPC mechanism to funnel multimedia around between apps, devices, and so on.
So there's a whole bunch of multimedia that you can funnel around, cameras, screen sharing, but also audio.
So pipewire tries to implement all of the APIs to make that possible.
So there is support for Video for Linux, there is support for Bluetooth, there is a compatibility server for Pulse Audio apps,
for audio, a compatibility library for Jack applications.
So you get all of these things here, all sides covered, and you can also run Jack next to it, but in essence, it funnels data around.
It's built in the same principle as G-streamers, so it doesn't exactly know what data is, it just funnels it around.
And it does so very efficiently or try to.
So that's basically where it is now.
We managed to build a whole bunch of stuff on it and replace Pulse Audio and the Jack Demon in most test-tops now with pipewire.
So 1.0 was released last year, so that was a major milestone.
Very happy about that.
So for that to happen, I wanted to have at least as good latency as Jack server so that we could actually replace pro-audio use cases with pipewire
without having to sacrifice latency or performance.
So that took a while, but it eventually worked and now we are on par with Jack regarding latency,
and it's using quite a bit less CPU for large buffers, and it's getting almost a little bit better than Jack for very small buffers.
So that's pretty good.
One of the reasons for that is Jack is more efficient even at lower buffer sizes,
but pipewire is more optimized in its conversion and funneling samples around.
So that's the compromise, I guess.
So compared to last year as well, we have now support for NETCHAC with Opus.
I think it was a question last year, why don't you have that?
Well, now it is there.
So you can actually NETCHAC between Jack and pipewire.
They're compatible.
One thing that doesn't exactly work very well is firewire devices.
The problem is that I don't have a firewire device.
You can't really buy them anymore, so somebody needs to send me something.
They are like €1,000, you can maybe buy, I don't know.
It's also professional audio, so you need cables to connect.
It's just not a plug and so on.
So that's a little bit of a back home yet.
What else are we working on right now for AES?
It's basically RTP.
It's used for various hardware, professional hardware, that does audio over TCP and IP.
So you can interface with Dolmete devices and so on.
It requires like a shared clock with PTP and all of that.
So we have worked that in pipewire.
You can run the graphs with PTP clocks, it syncs and all of that.
So people are testing that.
Very specialized hardware, I don't really have any of these things.
On the other end, we are now past the audio stage and now we are going back to the video.
Because last year, some things fell into place to make that possible.
For example, video modifier support was added.
It requires a multi-step negotiation.
I have these modifiers, do you support that?
No, I do.
I do all but then what video formats and what resolutions.
And you need to go back and forth to arrive at the video format that the compositor in this case and Gstreamer, for example,
or any other application like OBS, to get the most efficient video frames negotiated.
We also added support for compressed audio formats.
So for Bluetooth, we are still tracking, it's a draft, low energy audio.
There is development in Blues, which is the D-Bus service that runs and there are all the connections with devices.
And it exposes a D-Bus API that an audio server such as PipeWire could use to talk to the Bluetooth devices.
So there is development there and we are trying to track that and match it to make that work.
Some small things that were added that we don't actually know what to use it for.
Interesting things that are happening is the video support.
So I hope this year this will continue going forward.
So we added video support in Firefox, so that means that instead of Firefox going directly to the video for Linux device with IOCTLs,
which is not so nice in sandboxes, but which also doesn't work with newer cameras,
because newer cameras, they need much more setup.
They need to control setup media controls and all of that.
So there is a new library called Lib Camera that also handles these new kind of cameras that you are supposed to use.
So instead of porting Firefox over to Lib Camera, it's better to port it over to PipeWire,
because then you get all these cameras that are new, but you can also do some other things like send video frames between applications into Firefox.
I was going to try to demonstrate that, but the video support in OBS is still a pending patch to make that happen, maybe next year.
So there is also camera support there.
So in OBS, it's an application for making screencasts and YouTube videos and stuff like that.
So you can compose some things and try to demonstrate that.
There is also a thing called virtual camera.
So OBS can export its scene, and it looks like a camera that PipeWire makes,
and then you can actually consume that feed into Firefox, and you can start chaining just like you would chain audio processing elements, but then with video.
So that's hopefully something that we will try to make work this year.
There's some more work needed to get that going.
So we are bug fixing small improvements, because there is nothing really to be done on audio that we know it should work.
And all the remaining problems are, in my opinion, I don't know yet, driver issues, timers that don't work so well, unpredictable delays in drivers.
So I think the work needs to be done somewhere else.
No immediate plans to fix there.
So all the work goes into the video side of things.
So video routing.
So we're working on video converters so that we can convert between formats.
Like if you want to implement certain shaders that work on one format and not on others, this should be made possible.
Also processing filters with Vulkan shaders or processors.
So here, so now that Firefox and OBS use pipe wire for the cameras, we need to start thinking, okay, this is now going to work in flat packs without having to open the whole socket.
But then we can also start adding security, like the pop-ups, do you want to allow this camera, yes or no, or take away access to the camera if you don't want it anymore.
So there's some talks about that to make that better.
This is in-planet currently with the portal.
But there are other use cases, like for example, we don't have any access control for audio in browsers at all.
But that is something that we'll hopefully flesh out this year.
Another thing, explicit sync support.
Again, if you do the video processing, it's better to delay or like to queue up as much work in the GPU as you can and then have GPU itself synchronize all the buffers waiting for rendering and stuff like that.
So explicit sync would transfer buffers and also file descriptor with it that you can use to wait for completion of the buffer data actually.
So that's also something that we want to try to do.
And then tooling and docs, the things we continue doing.
So I was going to show you a little bit what it looks like the video.
Everybody knows the audio.
Also a little bit of tools here, I don't know if you know any of these things.
So there's like a top thing.
This is interesting.
It doesn't do anything because there's nothing going on.
But you can also get these things like a draw view.
I'm showing how now because then you can see the cameras as well as a device.
So if you, I don't know, let's see if this is going to do anything.
Probably does, but there's no, it's going to the HDMI.
Anyway, you can see, maybe it comes on the feet.
I don't know.
So you can have like a little look what's going on here.
In this is a tree view of the graph basically.
So you have like the audio driver is iterating and pulling in samples from another tool, PA Play.
You can also see this as a graph view.
And all of these things, you can link them together to other things.
So right, so each of these devices and nodes are in a graph.
You can visualize the graph.
You can change the links between these things and do all these things.
So for example, for OBS, this kind of what it is.
Well, I made a very stupid scene, but you can make some interesting things.
I don't know.
You can put some backgrounds there and place yourself there.
So this is using a screen sharing from one of my windows.
I think the terminal, but it could be anything using pipe wire and also the capture, which is a new thing using pipe wire.
And also these things here, the microphones, they are still a bit pulsed audio.
You can look in the graph here.
That's becoming a bit more complicated.
But you can see these green, these yellow boxes here light up.
So you'll see that hopefully a bit more.
So you know Shell, that's the screencast stream that sends video to this one.
That's the camera from OBS.
Yeah.
So I was going to show some Firefox things, but there is no export button here.
So normally in OBS, you can now do so start streaming and send all of that to, I don't know, one of the hundreds destinations that are supported.
But you can also start a new camera, a virtual camera, and then you could consume that camera or this composition in other pipe wire apps.
So if we enable all these pipe wire apps and we make them as efficient as possible with all of the video modifiers and all of the tools that we get from Vulkan.
Yeah.
We should be a step closer to the ultimate goal.
Yep.
Some other thing that's interesting.
I haven't shown that yet, which is basically called filter chain.
So you can do this.
You can make a small little file.
Wait, let me see where I put that again.
Yeah, this one.
Yeah, it's a conflict file.
It's not very easily, but I can imagine GUIs that generate these things, but nobody has written any of them.
But you can basically make a little graph of lots of plugins and LV2 plugins.
And you can link them together and then you can tell pipe wire to make a new sync of that.
That's the input for applications to use.
And then that is the output of this filter.
So this is something that does again.
And you can just then I'll use some debug here.
Okay.
You can run this graph.
And if all goes well, you should also see new sync here.
So this is this new thing that appears.
So you can just stop this program again and take away the sync.
So this is interesting.
And did I quit?
I can do it again.
And so here is this new volume sync.
So you can just on the fly create and remove devices if you want.
It's a bit like pulse audio with loading modules.
But in pipe wires case, you don't actually all need to load them into one demon.
You can have separate programs starting and stopping them as they go.
So for this filter chain, for example, that's used on for like implementing like sound correction for speakers and all of that.
We haven't done any of these things on desktop yet.
Also, maybe something we can do.
Like for example, on Apple, you get the sound of of apples are so great because they do a lot of filtering to make the frequency match speakers and all of that.
So if you don't have that, it sounds very thin and a lot of laptops, they need some extra processing to make them sound great.
Sometimes why they sound a lot better on Windows.
We don't do any of these things yet.
So that's also something that we can do with these filters.
All right.
Something else that that I don't mention here because it's actually another project, which is the session manager.
I've shown this.
One big component in all of this is a session manager.
We use wire plumber normally.
So that one is kind of orchestrating all of the things that happen in the graph, the devices that appear.
If a player comes where it's going to be linked, how it's going to be linked, is it going to do a mixing down mixing or is it going to need some filters before it does that.
So all of these rules are external to pipe wire in in a session manager.
So a lot of work is also happening there.
It's a separate project.
But yeah, there's, for example, a version five coming out where all of the conflict files are rewritten in a different way.
So that's also a change or interesting things that are going to happen.
For the pipe wire demon itself, I think it's kind of that's what it is.
No new plans.
Okay.
Yep.
So the usual.
Yeah, we worked a lot on our documentation too.
There's a lot more stuff there.
Also the weekend as a whole lot of stuff.
It's a bit difficult to organize all of these things, of course, and it's why am I.
This is weird.
I didn't start the browser.
Well, I could do that.
I guess.
We got tons of information on the week.
All of the stuff should normally be documented somewhere or another.
So a few.
Problem is that it's so, so much configuration and so much options that people get lost.
I tried to do some simple guides.
How do I enable multiple sample rates and you literally have make this file put that in it.
That's it.
So.
All right.
And get up.
That's where we are.
So yeah.
Questions.
Yes.
Speaking of docs, I was looking at them just the other week.
I assume you have the ability to use your own event loop manager rather than the basic tutorial, which says create this one of pipe wires.
Yeah.
So the question is, can you use your own event manager or do you have to use the pipe wire one?
You can use your own one.
The pipe wire one, you can make it and then you can get the file descriptor from it and add that to your own, to your own loop.
So for example, no shell does that.
It uses the G main loop.
KD as well.
Is there something or do you know some project which hooks in speech recognition into the audio part and creates subtitles on fly?
What they're in the stream.
The question is, is there an application that hooks in the audio stream and generates subtitles on the fly?
No, but it's a great project.
I think.
Yeah.
There's also the case, for example, of keywords.
So listening for keywords like, hey, Google, okay, Google or something like that, or I don't know.
Hello, Gnom.
Yes.
When you talked about consuming the virtual camera, would you be able to send those sources to multiple destinations?
Yes, so the question is, does a camera can be sent to multiple destinations?
Yes.
So there can be multiple consumers from one camera in pipe wire.
I can actually show that.
Just to show what's going on.
How am I going to do that?
I can, for example, start OBS.
So that's one using the camera.
And there's also, let's say, I think there's an example here.
It's in build.
Build.
Examples.
I think it's called video play.
Other way around.
No.
The thing is, of course, the second one has to have the same resolution of the first one.
There's no conversion going on immediately.
There's a way to reorganize the negotiation and all of that, but that is a GAM policy for wire plumber, I think.
So that's,
I think,
it's not immediately implemented.
Yeah.
I was curious about the,
the work capacity.
I know that there is an AES-627 plan.
And I also was wondering if it was the same thing for video, maybe a SIMPTP or an NDI or things like that.
So the question is, RTP or network support for video?
Completely unimplemented.
At all.
So only done for audio.
Yep.
No.
The current stage, like, is it just, do you have, you've been involved with people who are using the AES-727 communication?
Yeah, I know people are testing it.
There's an issues page about the state of it.
So I'll have to look it up what it is exactly.
But you can find it if you look for AES in the issues.
You find it on all of the hardware that people test with things they have, the tweets they have to do, and then we try to all, so that's ongoing.
I have to go over here.
Yeah.
Thank you for making it because I switched to hardware like two years ago and it was just a very pleasant experience because it just worked.
Yeah.
And I've also been using it in music related stuff and all the places Jack for me as well.
Yeah, it's great.
Cool.
That was the plan.
It wasn't the, if I have to repeat the question, it wasn't the question, it was just praise.
Yeah, we have more questions.
I have two questions.
The first one is for the wire plumber.
Does it have a ground using the place or it's just a direct command line?
Command line.
So the question is wire plumber.
Does it have a GUI?
No.
No GUI.
So you can, for example, have several applications and all the sources you can.