So, now we have Haley Thompson and we're going to talk about the distributed music programming
with Gleam, Beam and the Web Audio API.
Give it up.
Okay, so, hello everyone, yeah, today I'm going to be talking about a little web app
I've been making using Beam, Gleam and the Web Audio API.
Just before I get into that, maybe a little bit about who I am.
My name is Haley.
I'm a front-end Elm developer, actually, so I don't really do any back-end stuff.
I'm totally new to Beam, Erlang and Elixir.
I've been doing Elm professionally, almost exclusively, for about three years now and
kind of personally, for four or maybe five, and also a PhD student.
I'm writing up my thesis at the moment on programming language design and particularly
how it relates to sound and music computing.
And finally, I am a Gleam community person.
If you've ever dropped into the Gleam Discord, you've probably seen me spending way too much
of my own time there.
So distributed audio, what the heck am I talking about?
What am I going to be making?
This nondescript-looking box is called a mono, and one of the things it can be is a step
sequencer.
And so what that means is each of these buttons represents a note that can be played, and
the columns are steps in time, and the rows are different notes, different frequencies.
And what I'd like to make is one of these in software, and I want to supercharge that
basically by making it networked and collaborative.
So we want everyone to be working on the same instrument, you know, on different computers
over the web.
The way I structured this talk, I'm not going to be going into too many technical details
about Gleam or the app itself.
If you were here earlier this morning, Harry's talk would have done a really good job of
introducing you to Gleam, and if you missed that, the language docs are a much better
start than what I could give you.
So instead, I'm first going to go over some of the languages I could have chosen and didn't,
and then briefly explain why I picked Gleam.
And then I'm going to give you a very, very abridged tour of the codebase by basically
building the thing from the ground up.
So why not your favorite language?
Why not JavaScript?
Well, I've been doing Elm, as I said, for three, four, five years now.
I've been in this great statically-typed, pure, functional fantasy land, and the idea
of going back to a mutable, dynamically-typed, object-oriented thing terrifies me.
I just don't want to do that at all.
So okay, why not Elm then?
If I'm so used to it, why would I not use that?
Well, I actually maintain a package for doing Web Audio things in Elm, but if you've ever
used Elm before, you probably know it has a rather interesting take on foreign function
interfaces and interrupt with JavaScript, and I just don't want to deal with that for
this particular project.
And then it also leaves the question open on what to choose for the back end, and really
add, like, just one language for the entire stack.
And finally, why not Elixir?
Well, I don't know it for a start.
As I understand, I'm still going to need to use a lot of JavaScript for the audio side
of things, even if I use something like LiveView.
And I'm a bit of a type nerd, so the dynamic typing kind of puts me off a bit.
For me, I think Gleam conveniently addresses all of these things.
So I get to use the same language across the entire stack.
Gleam targets both Arlang and JavaScript, and I get to share types across the stack
as well.
So my audio code and my messaging and stuff, this can all be well typed across kind of
the network boundary.
It also got a really good interop story.
The FFI in Gleam is very simple, very, very easy to use.
And so if I need to dip into JavaScript or Arlang or Elixir, that can be quite easy.
And also, it's a very simple language.
So for someone like me that's very new to back end programming, this is a great kind
of soft introduction to Beam and OTP and that sort of thing.
Well, I didn't go to that slide, but that's the slide I just did.
The first thing I want to do is make some sounds.
And to do that, we need to have a bit of an understanding of the web audio API.
And so a super, super quick primer on that is it's a lowish level browser API for making
sounds on the web.
You create audio nodes, so they might be sound sources like an oscillator or some signal
processing like a filter or a delay, and you connect those into a graph in JavaScript.
But all the signal processing happens in native code that we don't write and we don't control.
So this is just a very brief example of what that looks like in JavaScript.
I don't know about any of you, but to me, this is really, really clunky.
We create a bunch of nodes, then we set a bunch of properties, then we have to remember
to connect them up, and then we have to remember to start some of them, and then at the end
hopefully we get some sound.
Instead, what I'd like to do is get a really nice declarative API for this, something that
we might be used to for doing like view code.
And for that, I'm going to model that with these two types in Glean.
So we have a node type with a filled T, which stands for type, and so that says whether
it's an oscillator or a delay or a filter.
And we have a list of parameters that we want set on that node, and then a list of connections.
And then we end up with something like this.
So this is the same audio graph that we just saw with a, in my opinion, a much, much nicer
API.
You kind of get implicit connections based on how nested things are, kind of like a DOM
tray or HTML or something.
What I'd need to do then is write a little bit of JavaScript to turn those Glean values
into some Web Audio code, and we're not going to go into any detail on that here.
It took me about 50 lines of JavaScript to do that, and that is the only not Glean code
that I wrote in this whole app.
So assuming that all works, the next thing we want to do is render something onto a page.
For that, we're going to use a framework that I made called Luster.
I've said maybe like 50 times now that I'm a big Elm fan, and so Luster takes a lot of
the ideas from Elm, particularly its ModelView update or the Elm architecture, and it basically
applies it on top of React.
So we actually have a wrapper for React, and we can use React components and all that sort
of thing with this nice kind of unidirectional stake flow.
So we start off with a model, and this is what we're going to derive both user interface
and audio code from.
And so here, I don't have the type up on the screen, but where we've got rows, a row has
the note, so the frequency to play, and then an array of steps that either indicate whether
it's on or off, and we take that model and we render it into something.
Now Gleam doesn't have macros, it doesn't have a templating engine, or really anything
like JSX or anything like that.
What we have is just functions.
So here, we're calling element.dev, and we're setting a class on it, and then inside we're
rendering a button, and we have this message, this update step message, and basically that's
going to be fired whenever the button is clicked on, and that goes through the runtime into
our update function.
We change some rows, update some program state, and the cycle continues.
So the state changes, our UI changes, more interactions, blah, blah, blah.
If all goes well, we end up with something that looks like this.
And what we have here is just a simple client web app.
This is the sequence that I've been talking about.
This only runs on the client, so anyone that loads this up is going to get their own thing.
And so far, we haven't spoken about back-end, so I'm assuming you're serving this on GitHub
pages or your own server or whatever.
So what we want to do next is serve this with some Gleam code, and to do that, we're
going to use two more packages.
One is called GLSEN.
This is a fairly low-level package that sets up a supervisor and manages a pool of connections
that can manage things like TCP connections and sockets and this sort of thing.
And on top of that, another package called mist, which is a web server written in Gleam
that provides a kind of dead simple HTTP server that you can then configure to accept web
socket connections or do SSL connections, these sorts of things.
So far, I've been heavily abridging the code.
This is pretty much all you need to start serving some static files using mist and
GLSEN.
The magic kind of happens just in this very simple serve static asset function, which
takes a path.
Ideally we'd do some finalization on the path, but I've left that out to be brief.
Read the file if the file exists.
We just respond and we make sure we set the right headers, and that's it.
Now we can host our little web app statically with more Gleam code.
The final piece of the puzzle then is client server communication.
How do we make this distributed?
How do we have everyone connected to the same instance?
So for that, we need to set up web sockets and mist makes this dead simple as well.
You just set up an upgrade handler on any particular path that you want here.
It's just the web socket path, and that code looks like this.
You set up some event listeners on when the socket opens or closes, and then also how
you want to handle messages.
On WS message here, essentially just Jason decodes the message into something well typed
and sends that off to our app's main process.
On the front end, we need to hook up web sockets as well.
There's a package for that called LusterWebSocket.
This isn't made by me.
Someone else has very gratefully made this.
For that, we just need to call WS.init in our app's init function, and that will set
up everything that we need, so it will do all the plumbing into the runtime to make
sure the events are dispatched and end up in our update function.
So here, we pass in this WebSocket message constructor, and then whenever we get an event
on the WebSocket that goes into our update function, we can change our state, do whatever
we need to do, and that will affect the app and renders and so on.
Now, I mess, that is the wrong text, but oh well.
I mentioned earlier that one of the great things about DREAM is that we can share types
across the front and the back end.
And so, what we can start to do is have to type messages between client and server.
So here, we have a to back end message type, so this is what the clients will send to
the back end to ask it to update some state change.
So for example, start the sequence, stop it, toggle a step on or off, update some parameters,
and then we'd handle that in our apps main update function on the back end.
So here, we're updating some shared state, and this is the state that is shared across
all clients, and then we're broadcasting that state back to clients.
And we do that with a to front end message, and so this is the same kind of idea in reverse.
This will tell the client to update a particular part of its model.
That looks like this.
Again, we decode the JSON that we're getting from the web socket, and then we can just
branch off of that, and this would be called in our update function.
And so what we end up is this really neat, tidy kind of loop where the server sends
a message to the client with some state to render, then user interaction happens, an
event is emitted from there, and instead of updating the state locally, we send a message
back to the back end, that updates the state on the back end, and then that state is broadcast
back to the clients, and we have the same kind of event loop that we had just on the
client, but now across the network.
Now I've waffled on for a bit, I think it would be cool to maybe see a demo.
I'm not sure we can get the sound.
I'm going to check the sound of the video guys, let's try to do what you want to do.
What would you like me to do?
I'll try to play audio, and I will see if I can.
Yeah, we are trying to play audio with the mini jack.
I can just play out the speaker, it's fine.
It's not a very big room.
The mini jack audio is not coming off.
Okay, well while they're dealing with that, I'll just explain what's happening, I think
it's kind of clear.
So we have two clients open here.
Okay, that's important, no problem.
Maybe it was me that was having no sound.
If it was muted, maybe it was the plug in there, let's try.
No.
Okay, cool.
It wasn't user error, it was okay.
So we have two instances going on here, for some reason that one isn't going, there we
go.
So I can change the parameters on this side, you can see they're reflected on the other,
add steps or whatever.
Yes, and so this is all totally networked, conceptually you could run this on the web
and have, I mean this is just running locally but I would have hoped that people could open
up here.
So just a recap, we've got a full stack GLEAM app, we have an ATP server on the back end,
we have a React app on the front end, both written in pure GLEAM, both sharing types,
and we have this live view style of communication, but specifically or kind of crucially, this
communication is well typed and so we know all the messages that we're supposed to be
handling on both the front end and the back end.
And this is just a quick kind of look at how many lines of code we're in this code base,
and so you can see 85 lines of JavaScript was all that was needed and everything else
is pure GLEAM.
Which I think is pretty cool, it's pretty exciting that you can do that today.
So yeah, thank you for listening.
Thank you, are there any questions, yep.
Thank you for sharing, maybe it was apparent from your presentation but I just wanted to
check how are the different clients synchronized.
Yeah, okay, so let me go back.
We had this model and when I introduced that each client had their own model and so basically
the server has its own version of this now and it's broadcasting, every time the sequence
resets, it broadcasts the entire model to make sure everything stays in sync and then
whenever one client changes something it broadcasts a message to tell the client to update their
local version.
So it depends on how the client gets this new information and that's more or less okay
enough for synchronization.
Yeah it seems to be kind of fine, I guess if one person is in Australia and one is over
here there's going to be some noticeable ping but then you wouldn't be stupid enough to
do that.
Thank you.
So I don't know much about the Gleam front end stuff, what was necessary to write in
JavaScript that you couldn't write in Gleam?
Yeah, the JavaScript is just the part that actually renders the Web Audio stuff.
So that's the APIs that are available in Gleam?
Well so Gleam doesn't really have any browser API bindings at the moment, I could have FFI'd
the whole thing and probably taken a bit more into Gleam but for that particular bit I've
done that JavaScript myself quite a few times and so it was just quicker to just keep that
little bit in JavaScript.
Thank you, thanks.
Any other question?
In the beginning you presented an API for connecting audio nodes by using nesting, my
question is how would that work with more complex graphs that have forks and merges
or feedbacks?
So you're talking about this, right?
Yeah, actually presented a kind of Striptown version of the actual API and there we have
like keyed nodes so you can assign like an ID to a node and then there's like Reth nodes
as well so you can refer to other nodes in the graph outside of the tree and so that
way you can keep this kind of tree-like structure but jump out and refer to anything you want
and have loops or whatever.
And so actually that's what's happening in this app so we've got that delay that's going
on in the background and that's the feedback loop and then it's going, yeah, does that
make sense?
Cool.
Any other question?
Hello, sorry, I didn't see the full presentation, I arrived in the middle and maybe I will ask
something that you already shared but I would like to know if can we apply this environment
for live coding, improvise the performance, it's mainly dedicated for building clients
and applications?
Yeah, I think you could totally transfer these ideas to live coding or performance, I mean
ultimately it just comes down to sending messages right and so here we're sending like user
interaction events but you could do conceptually the same thing with code snippets or some other
kind of data transfer, yeah.
Any other question?
Hi, Redsorg, I was wondering you said it was compatible with React and so will it be compatible
with other frameworks like Vue or the future?
Yeah, at the moment it's just React but it's been on my to-do list for a while now to kind
of factor out the state management that Lustre does away from the actual renderer that you
choose so right now just React, some nebulous time in the future, it could be Vue or Morphdome
or whatever.
Okay, I think there's time for one more question if there is one.
Okay.
Thanks for talk but if someone want to use some hardware devices to connect, does Glim
support some other wrappers over Web API to speak with some hardware parts like the USB
serial port, etc.?
Right.
Do you mean from the browser side or yeah, so like I said there aren't really any official
bindings at the moment but as I also said the FFI story is very simple so it's actually
quite easy to create bindings for these browsers yourself which is pretty much the situation
where we're at today.
I mean the biggest thing maybe just holding Glim back at the moment is the ecosystem is
just very, very young and so we don't have many packages or bindings for a lot of stuff.
Okay, thank you again for your talk.
Thank you.