Check. Can you hear me?
Yeah.
Service announcements. If you have glasses that look like this and you don't have them anymore, I found them.
I checked they are not with cameras.
Hi everyone.
Check. Check. Is it loud? Can you hear me in the back?
The HDMI audio doesn't work so I don't have any warm-up music for you.
But thank you everyone for coming. Hi. I'll start in time.
No, perfect. I'm just standing here. I'm not being angry at you.
Who has the lights?
Is there a lights control that we can turn off?
Can you put in the slides and see?
In the back. Can you see the slides if not raise your hands?
Perfect.
You're awake now, right?
We have fire alarms in this building.
All right. I'm going to get going.
Still two minutes.
Two minutes. Oh, all right.
For those who need to follow them, we have to start on time because of video recording.
So that's why we are punctual every time, every day.
I got an assistant just for picking out the best stickers.
That's the one thing you need at follow them is an assistant just to get you stickers while you're working.
It keeps us sane.
The Go community is 50% for the amazing language, 20% for the amazing performance and 100% for the stickers.
All right. Hi, everyone.
Hello. Can I quickly introduce you before you start? I know you have much to say. Sorry.
Sure.
I'd like to welcome James to the stage.
James got this talk because he's every single bus word on my bingo, which is full stack provisioning, DSL and single binaries.
And he will explain all what that means right now.
Go ahead.
Hi, everyone.
So I'm James. I'm going to talk pretty quickly.
And so don't be afraid and turn this down.
Can we turn this down slightly?
Maybe.
We'll see how it goes.
Yeah. So I'm going to talk really fast because I want to show you some real live demos and I'm going to be basically at 30 minutes.
So it's going to go really fast, but all the stuff I talk about is online.
And if you have questions, I'll probably not have any time, but maybe at the end.
So I work on this tool called MGMT. This is some stuff I won't show you.
I'm a hacker. I write a blog called the technical blog of James.
Who's seen my blog? Just raise your hand.
If not, just raise my hand. Your hand. So I seem really popular.
All right. Thank you.
And like, do we want bad information?
Do we? Come on. Louder.
I don't know about you, but I really don't. This is my nope guy.
You might have seen this. It gives me a chance to breathe as I'm getting my talk.
I'm going to be typing, so I'm going to be sitting just out of the way.
So I have this project called MGMT. Do you want to see a quick demo?
What?
All right. All right. Okay.
Really quickly because I don't have a time to give you a full explanation,
but basically I'm running it here over on the left and MGMT is running.
And then over here on the right, I've just asked it to create one file.
So I can cat hello world, right? This file that it has created.
But the cool thing is MGMT is watching in real time.
So if I remove the file and cat the file comes right back, right?
It's always watching in real time, doing that kind of thing.
In fact, it's so fast, I can even remove the file.
It's kind of my old demo. Oops, if I could type and cat the file.
And it always is that fast that it just instantly can fix the file.
And you can even do crazy things like this.
And basically ask it to watch as fast as you can.
And it will even, you can see MGMT on the left here is spinning really fast, fixing the file.
And that's kind of boring because we can do much more.
This is just the base.
What we really do is we have this language called an FRP.
It's a reactive language that has streams of values that go into the resources.
And when anything changes, it kind of rebuilds the world.
And I want to show you some real things it can do.
So no more abstract stuff.
So I want to actually, I want to provision, I brought this little kind of fat raspberry pi kind of computer.
And I'm going to show you a building and then I'm going to go back in my slides while it's provisioning
and show you a little bit about the tool and how it works.
Does that sound fair?
All right. Thank you.
So here's my demo and just just a quick thing to show you.
So I'm just actually run this here first.
So I'm just going to start it up here.
It's not perfect.
Like my type unification algorithm implementation is really slow.
So you'll see it.
It takes a moment.
But basically what happens is suppose you had this, this code at the top printf math dot 42.
It's just a function that returns the value of 42.
We actually build these graphs.
You can see that the values come in and the data flows along these graphs and build these big trees.
And in fact, for even small programs, they'll look like this or like this, these enormous graphs and they all just track
what's being sent and what's not and so on.
And so I'm not going to show you this demo.
I'm going to show you this demo.
So let's say you want to just provision a machine from scratch.
In my language that I've built called MCL, you just put the following code now.
It's all that you need is a single binary that runs this, this code and has everything you need.
You basically just set like your interface on your laptop, the IP of your laptop.
You put a repo.
So his definition for a fedora repo and what mirror to get.
And then you just define the actual machine.
So host name one, the MAC address and you're done.
And so let's go over here.
So MGMT is just about done running here.
It's just starting up some stuff.
It's actually, I've pre-cached a whole mirror on my laptop.
So that's just like the entire fedora repo.
So you have the packages there.
So we're going to have to wait for the internet.
And then the machine here, just so you can see what's going on.
I actually have a little HDMI capture card and right now the machine is off because I haven't plugged it in yet,
but it will just show you on this screen what's going on.
Does that make sense?
Do you want to see this?
It's going to be a little bit boring to watch a machine provision, but I kind of like this kind of thing.
So anyway, so this is starting up and just, I'm just going to kill these slides for a second.
What I'm going to do now, so it's starting up an inside MGMT.
There's resources which embed a TFTP, DHCP and all these other servers.
So let's find the power part here and plug this in.
So that's plugged in.
And I just forgot one thing for my bag.
Some notes.
Yeah, so here's the machine and it's actually starting up now.
You'll see here.
And on the left, we're just going to see what happens.
So as events come in, so you can see right away the DHCP server said,
Oh, I see someone talking to me on DHCP.
Can I have an IP address?
I'm not touching anything.
The machine is now automatically requesting the like the stage one UFI stuff.
This is a pixie boot.
So it's a network boot type of thing.
There's a TFTP server that passes those files.
And in a second, it's going to start doing that.
And you can see right now it's pulling down this file.
Do you recognize these two?
What are those?
It's basically Linux and the initial root file system boot this stuff off.
And MGMT is doing that.
I'm not doing anything at all.
And you'll see it's going to slowly kind of boot.
And just to get a little bit more information about what's going on,
the script I wrote in MCL has a little bit of extra information that it hides.
So, see if I can, oh, there we go.
Nope.
Okay, so right here, I have a little file.
Let me just fix one thing.
I forget which is the right one.
There we go.
So I just have a little thing that just shows what the state is.
So the computer is set to be provisioned and the state is not provisioned, right?
Because we haven't provisioned it yet.
And if you see it starting up, it actually is just loaded the kickstart file.
Does anyone know what kickstart files?
Anyone doesn't?
Raise your hand.
Don't feel silly.
A kickstart file is just a definition file that's built into RPM and like Red Hat, Fedora,
Flavor kind of OSs that just says what should we do?
Like how should I provision the disks?
All that kind of stuff.
And so it's doing that thing.
And I can actually even zoom in here if you want to see what's going on.
Oops, now it's gone.
Where'd it go?
There it is.
Just move over.
Oops.
What's going on with my little media player today?
Where is it?
There we go.
So you can kind of see what's going on.
So this is actually Anaconda itself running.
It's checking the storage configuration.
And in a moment, it'll do some stuff, which we'll see.
And then while I'm doing that, I can even show you.
So this is just me doing a random W get to the web server that's built into this single
Golang binary.
And so if you want to see what happens, it actually is just generated.
It has a template built into the script.
All this stuff is just generated.
And what it's going to do is it's going to do the basic install, install a few packages,
whatever you want.
And at the very end, there's a post hook, which is part of kickstart.
So at the very end, when it's done, just before the machine reboots, it will actually run some
command.
And in our case, it runs a little flag, a little W get that talks back to the web server to
send a secret message that says I'm done.
And so that's going to just do its thing.
And storage is a little slow.
It's just a little low budget machine.
Just to show you what's going on, I'll do, oh, here we go.
So there's storage.
It's running its thing.
All automatic.
If anyone has their laptop, by the way, that wants to be provisioned, I can set up your
laptop for you.
It'll wipe it, but it'll be good.
And you'll have a really good fedora machine.
I kind of like fedora.
It works.
It's very modern, upstream.
Easy to build these kinds of things in.
And let's scroll down here a little bit here.
You can see what it's doing.
So here it's building the partition.
And right now, if you look over here, this is MGMT.
All those packages are downloading.
So if you're ever curious about exactly what packages are requested over the network when
you install an OS, you have exact logs.
In fact, it was requesting all these files I'd never heard of.
And it's kind of interesting to see how machines work.
So with this embedded TFTP, DHCP server, all these pieces, you know exactly what's going on.
And you can also see weird hardware requesting weird things.
So there's the standard like what a pixie boot is supposed to request over the network.
And then when you look, it actually does some weird things that you might not know about.
Probably because some customer had some weird setup that wanted some firmware thing.
I don't know exactly.
So it's installing all the files.
Hi, come on in.
You're missing all the fun.
And yeah, let's just go look at a few slides while it's doing its thing.
In fact, if you want to be really nerdy and look at some code.
I never know how well this goes.
Is that too small for you to see?
So this in so just I don't want to give you specifics about the code because that's not important.
But the entire provisioner, this entire provisioning tool with the lines and comments and whatever is like 500 lines of MCL.
And so the realization I had some time back when I was doing puppet stuff was that if we had the right language,
and we had the right engine and runtime all implemented in Golang,
we would actually have a new way of building tools.
Now I'm not here trying to convince you that we should use my provisioning tool.
I think you should.
I think it's better than most provisioning tools out there.
But the real thing I want to teach you about is that this is a new way to build whatever tool you want.
And because it has all these real time events, it's very easy to glue all sorts of useful pieces together,
whether it's automating a distributed Cef cluster, whether it's single provisioning.
But like for me, like how many times have you all installed a Linux distro somewhere with like the USB key?
Like be honest, everyone here, right?
But the idea now you download this one binary, Golang makes that super easy.
You run it, you just give it a Mac address and you just plug the ethernet cable from your laptop port into the machine you're provisioning.
Or if you want, you could plug it into a switch and plug in a bunch of machines.
I think that's pretty easy, no?
I don't know. I think so.
But yeah.
So it's just installing all those packages.
And let's just go back here and see.
So yeah, so this is all the code it takes.
Now here's the cool thing.
So as I was saying kind of briefly at the beginning, I'll have to look online if you want a more fleshed out definition.
Variables are streams of data.
And so this base class here, you basically include this code as hostname one.
And then there's a hostname one variable and it has fields.
So provisioned.
So what actually happens is this bottom chunk of code is sitting there just waiting.
It's waiting until the provisioning happens.
And then when the provisioning is done, this variable, which was previously false, turns to true.
And that means that this block of code will declaratively execute.
So you can do these kind of stream things that are very easy.
And in this little file that we're pulling here, we just pulled this file and it's going to basically change the contents from that to provisioned.
This is just some screenshots for fun.
It works on all sorts of sketchy old machines like this one and so on.
Just a few thoughts while we're waiting for this to run.
These FRP languages, these reactive languages, you probably have only seen them in the web.
Like all these web UI people use these languages because you just change something and then the elements update.
I don't really know much about them, but that's really where they're used almost everywhere.
In the 70s, they used to actually be used or were done as a proof of concept to control helicopters.
So they can be very, very fast.
And for me, realizing that FRP was a solution to a totally different problem that's not UI based was exciting for me.
And there's so many cool things that you might not realize.
All the variables have events.
So if you're debugging something, you can just point to a variable, have it be displayed in a file,
and you can watch it change as your program executes.
Real-time live debugging of variables.
Very cool.
I use this all the time.
No, I don't use it all the time.
I should use it all the time, but yeah.
This demo has a local mirror, but you know, you could do this off the internet too.
All sorts of other cool things.
500 lines of code.
Just for curiosity, what provisioning tools do you all use today?
Pulumi.
Like all these tools, like how long did it take you to set them up?
You need a DHP server, you need a TFTP server, you need an HTTP server to do pixie booting.
Like I know those people that just, you know, use the cloud to provision their thing,
but like if we want to really control our stack and control our provisioning,
I think it's kind of important.
A few things.
The tool is not finished.
MGMT, my tool, is not finished.
The provisioning tool, which is one tool I've built with MGMT, is just one of those things.
If you're good at programming and Golang, I desperately need more contributors who are smarter than me.
I think I have the overall design, but some of the nitty-gritty little things to get them better and perfect.
My type unification performance is really bad.
It's like a, I'm sure, a very suboptimal, like, end to the power of end kind of algorithm.
It's not that bad, but it feels bad.
I still make, like, Golang concurrency bugs.
I guess we all do, right?
And I'm slowly killing one at a time, but if you're good at that,
if you're good at lexing and parsing and, like, error messages to make my parser not be so hard to understand,
that would be great.
And writing new tools and all sorts of other cool things like that.
I'm just a guy doing this because I believe in this project.
I've been doing it for a while.
It's kind of getting ready to be production ready.
I think now it's finally really usable, but I don't know how to do this.
Like, I don't want to make it proprietary.
And so my next latest idea is the MGMT partner program.
So you sign up for free.
I have, like, a Google form, and I'm going to send you newsletters if you're interested.
And if you have a company and you want to sponsor, that would be kind of really cool.
And, yeah, if you want to go to this link, you can do that.
Bitly MGMT-partner-program, like tell your companies, pay $100 per year,
and I'll send you stuff and send you new tools, or free, if you really want.
And, yeah, we'll come back to that.
So let's just go back to this provisioning thing.
So it's still coming along here.
I'm going to just make this a little bit nicer to see.
So that's, it's just finishing the package installation right now.
The good news is I'll probably have another time for a live demo if you want another one.
And we can see the machine is still not provisioned, still running away here.
Quickly, while this is happening, does anyone have any quick questions, like really good questions?
Scream it out, and I'll repeat it.
I'm not sure if I see any questions, but I see that you were using just Dora for serving the CD-card.
Tell me.
I'm not familiar with the tool, so probably some of the documentation is something less.
Since you were here, I can ask you, does it support other distros?
Yeah, so at the moment, so I've built this provisioning tool.
Can you repeat the question?
Yeah, yep, I'm going to.
So I've built this tool that just supports Fedora at the moment.
And yes, you could definitely, in this MCL code, add like, if, if this, whatever.
I haven't done all those if statements because, bless you, I'm just trying to test the minimally viable thing.
But basically the goal is to add more distros when there's other people that want to do the work, right?
I don't want to do the work for everyone else.
Supposed to collaborate on it.
So yeah, and actually right now here, it's just finishing off the configuring.
The whole demo takes about 20 minutes.
So I think we're about four or five minutes away.
The last thing it does is checking all these packages.
It's installing the bootloader.
Let's just go over here.
Look at that.
Creating the users.
This will ramfs.
And then you'll see here in a moment, MGMT is going to wake up.
And tell everyone what's going on.
I was kind of a system in the original life.
And I would look at machines doing this a lot.
I was an early cobbler user.
I don't know.
Does anyone ever use cobbler?
Like back in the day.
And the thing is cobbler took so long to set up.
I had to set up all these different pieces and get all these templates right.
And it was kind of hard to do.
And so after cobbler, I got into puppet.
So I only use puppet.
I did a lot of puppet.
I have like old blog posts about all the fancy puppet stuff I was doing.
And I was doing all these crazy puppet hacks.
And then I realized one day that like all of these hacks should just be built into one tool, but not hacks.
And then I started working on MGMT.
Because the puppet folks just didn't think they were going to re-architect their core engine and language.
And it was literally all of those lessons that probably happened over 10 years ago that taught me what I wanted to build.
And I've been waiting to build the provisioning tool as one of the first examples of what MGMT could really do for a long time.
And the real question I have, okay, so look at that.
So I just ran the post-installation scripts.
You can see here it got this URL, so action done, blah, blah, blah.
Got flag equals true.
And if you watch this here, this text file, I'm just pulling the text file.
Why the text file?
Because we're just declaratively putting some state into a text file.
And MGMT will know when that changes.
And why do we want to actually catch the state?
Because when we know that it's done, MGMT is going to change a variable.
The code will automatically update.
And then when the machine reboots, it's not going to boot back into the provisioner again.
Does that make sense?
If it's working.
Should be working.
Maybe it's shy.
Good point.
Maybe it's not actually pulling that through.
It's booted up.
The flag should propagate.
Let's see.
It should boot up in a sec.
So the annoying thing dealing with all these machines is they have all these weird kind of firmware,
buggy things and timeouts and things like that.
And here's one of them.
You have to wait for quite a while for the pixie boots to timeout.
Obviously this is a...
So there's Fedora booting.
And if we go here, in a moment, we'll be able to log into this machine.
Not yet.
Still loading.
Oh, I'm plugged in that work.
For a sec.
Did I ruin it?
There we go.
So logged in and the password is password.
Don't tell anybody.
And now we can do something destructive.
Who wants to mess up this machine?
True?
Is this bad?
Is that necessary?
I don't know.
We can delete DevNull.
That's not good.
What do you want to run?
No one's adventurous.
Anyways, you get the idea.
So that's the provisioning tool.
Yeah, do you want to see another quick demo or two?
Only some of you do.
Do the rest of you want to see another quick demo or two?
Yeah.
I'll just do a kind of a classic demo here.
Oops.
Just to show you, I'm going to over here start up MGMT.
And the code that I'm running, is this working?
I think it should be working.
Just to show you what's going on, I promised about streams of stuff.
I'll plug this.
Can you see this OK in the back?
How's that?
So everything is a stream of values.
So I have this function datetime.now.
And it is literally the number of seconds since the epoch.
And so it turns out that this is a function which happens to update every second.
Because time is always moving forward.
Other things like this multiplication are static.
And you have all these different event streams.
System load.
Here's another one.
System load down here.
And for fun, I even added a view meter, which is actually sampling my actual laptop microphone.
And so you just take all these values, combine them into this big template down here.
And over here, I'm printing those out.
So if I actually pull this file, see if this is working.
You can see, I'm just pulling the file over here on your right.
And if you look, you can see the time's changing, because it just reruns the graph every time something updates.
The system load itself on my laptop, you'll see changes.
And my little view meter, if I make noise, you can see it goes up.
If my microphone is up loud enough.
You get the idea, right?
And all this is open source.
You can all try this online.
And the point is that you can now kind of start thinking about building your infrastructure in a way that you just describe the state.
Today's state, the error scenarios, basically write a load balancer in software all very easily in this simple language.
And then run it, and it will take care of things.
So there's more material about this online, but that was just roughly to show you how it works.
I'll kill that.
Really quickly up to the top, because I got a few minutes left.
So we have the core engine itself has all these resources.
Those are the things that do work, like files, virtual machines, DHCP servers, so on.
Those are all built in.
And then there's this language, which is that FRP of values that creates graphs of resources and runs them.
We run in parallel, we run in event driven.
I didn't talk so much about some of the Golang nitty gritties, but Golang has been an absolutely great language for this.
Because we do all this stuff very quickly in parallel, very, very fast, very easy to build this one binary with everything contained.
And there's a lot of great libraries, bless you, that make it possible.
So we leverage etcd, for example, which is also built in Golang and so on.
We've got all sorts of different resources for managing things.
Bless you.
And yeah, lots of stuff about this is online already.
Just quickly down at the bottom here.
Future work.
Who's like a really skilled or adventurous Golang programmer that wants to help out?
Like if you've got skills, don't be shy.
Ping me.
This silly partner program thing, if you want to start using this, your company played around or just on your personal projects.
Let's just recap a few things.
He's just recapping his pen.
That's the best joke I've got for the end here.
I started a matrix channel because IRC is kind of dying and no one's there.
We don't have a mailing list anymore because Red Hat used to host it and then they killed all the mailing lists.
So if you are someone who hosts a reliable mailing list and wants to host our project, that would be great.
There's a bunch of links online.
So I'm purple idea on the internet all over places.
So don't be shy.
Feel free to ping me.
If you're going to put in the time and you want to improve your Golang skills and so on, don't be shy.
Ask me and I'll help you review your patches and so on.
Yes, I'm someone's probably annoyed that I keep saying Golang.
I just find it so confusing to say go all the time.
It confuses my brain and when other people say it,
it confuses me to hear what they're saying.
So I know the language is called go.
I just try to be less ambiguous.
It turns out I'm not the only one, but my apologies if I've upset someone.
There's a great conference in Ghent, which is right after FOSSTEM.
And I'm going to be giving a slightly longer talk and a workshop,
a talk on Monday workshop on Wednesday.
So if you want to come by, it's free.
Although we do ask you to sign up.
It's a big six, seven, eight hundred person conference.
So not huge like FOSSTEM, but still pretty big.
And I have got some stickers if anyone would like a sticker
and they're actually going to use it.
So, yeah.
Does anyone have a question or two?
Yeah, go ahead.
Scream it out.
We're a lot about provisioning, but what about configuration management
after something is provisioned?
Absolutely.
So the question is like what about config management after provisioning?
And we absolutely do config management.
My realization was that the standard way we talk about config management
meant only this narrow thing.
And I really believe that config management, the way I see it,
is actually a more broad topic.
And with the right tool, you can actually squeeze forward
and do provisioning as part of your config management process.
Do the actual configuring of the machine.
So at that machine at the very end, we definitely can kick off MGMT
and run itself and keep doing stuff there.
That absolutely can happen.
And then further, some people talk about orchestration.
I don't like orchestration because orchestration is centralized.
But we do what some people are calling, I think it's a lame word,
but choreography.
So this MCL language allows you to write distributed algorithms
that run on more than one machine at the same time
and they coordinate for various checkpoints and other things.
It's really very cool, but definitely out of the scope for today's talk.
So yeah, good question.
Yeah, go ahead.
I see an article about the distribution of topology.
I'm not sure what's the...
Yeah, the question is how does the distributed topology work?
It's a longer story, but long story short, you write the MCL code.
It gets pushed to every host in the cluster with EtsyD
and they all run that separate algorithm, but some variables are different.
So host name variable will be different per machine and so on.
And they can use those slight differences to run slightly different codebaths
in a way that they work together.
And it's pretty cool, but I'm, you know, again, biased.
Yeah, any more quick questions?
I got about 30 seconds.
Yes, no. Yeah, go ahead.
Do you have a live version of MGMT doing post-provisioning of business
or do you have a system?
Yeah, so I use it personally for stuff.
I run a few low-budget system-in jobs for some local businesses and stuff like that.
But yeah, it's not publicly documented.
It's just sort of, I use it.
I've started using it very recently.
But yeah, good question.
And I think, so my goal right now is to have more people start using it
and being early testers of real-world stuff.
So if you're interested, please ping me and hopefully get you along with MGMT.
Yeah?
Thank you very much.
Round of applause.
Thank you.