No, just rock and roll.
All right, test, test.
All right.
Welcome to FOSDEM 2024, Nick's Dev Room.
So I might have been a little bit early.
Everyone's still kind of trickling in from the real hard parting last night, so that's fine.
All right, I was just going to get started and kind of take it away.
So my name's Tom, Tom Burrick.
I work at a place called Flock's.
I work on a bunch of things called Knicks.
And I want to talk today about the different kind of units of composition we have in the Knicks ecosystem.
Some ideas about what it is that we could maybe do to improve some of these things and just some issues.
And also just a little bit of explanation of some of these units of composition that are maybe less or known or hard to understand.
And maybe just start some conversations about it.
All right, so I'll go over kind of a few ideas, trying to present a concept, figure out like, yeah, what is this thing?
Why we care about it?
What are some of the problems we have?
I have some proposals, maybe some examples.
And hopefully along the way we learn something and come up with some ideas.
As a disclaimer, a little bit of Knicks understanding is expected here.
But if you got here at 9 a.m., I assume that's not a horrible thought.
All right, so basically here's where the experience might come in.
Hey, like what do we call this thing?
Feel free to kind of shout out.
Like what is this thing we're looking at right here?
A package.
A package, a function.
What else?
A derivation.
What else?
Sorry?
An expression.
Okay, so that's a problem.
This is used everywhere, and I just got how many different names for what this thing is.
Right, this is something we don't, I think this is an issue.
So all these names we used are not quite right, we don't know what it is.
So the main idea is like, one I'm going to explain how this thing works, why it works,
why it is the way it is, and then be like, hey, let's actually name this thing.
Let's make this a first class concept.
Let's somehow make this thing a bit more core to the system,
rather than getting seven different answers for what this thing is.
All right, so we have some whole bunch of issues,
tried to reference some of these things, but like, hey, we tried to define what this is.
But we just used package, I don't know, four or five times in the definition of what a package is.
I don't think this is going to work too well.
Right, that might be a problem.
We have another example of Robert Hensing having an idea of let's formally define what a package is
and make kind of NICS understand this, because it's funny, we don't really have a formal definition of what a package is,
yet we often call NICS a package manager.
Well, is it a package?
It's a little bit confusing.
So he's got some ideas on that, which I really like the progress on that.
There's another issue that came from John Erickson.
I also really like this one, like, what are these functions?
We call package on them, but saying this is a call packageable function just doesn't roll off the tongue very well.
And also, like, this also has issues with when you're trying to do things like cross-compilation
or if you're having like your look at your memory usage, having everything be operated after the fixed point operation NICS packages doesn't quite work.
So there's memory implications or performance implications.
I want to address all this, or at least kind of come up with some ideas.
All right.
Why do names matter?
Well, let us communicate.
I could tell a beginner, I could teach someone, hey, here's a thing, here's its name, here's its properties,
here's the form of what it looks like, here's why it's the way it is, and having a name helps.
If every single time you have to kind of subtly give them a not quite right word for the thing, they're going to get confused.
Because you just said this was a derivation, but then when you call it, then it becomes a derivation, but it's not quite,
it's really, again, extremely confusing, especially if you're a beginner.
Right?
Like, we might kind of understand and look at it and recognize something like this, but look at it from the lens of someone who's new.
They're not going to.
So let's define this a little bit more precisely.
We use this thing everywhere, and you encounter it really, really in the process.
As soon as you want something like, oh, I got my own thing I want to put together, or I want to compose an environment,
or compose anything, we almost always use this abstraction.
So let's use it.
We want this to be understandable.
We also put lots of abstractions on top of this notion.
Things like overlays are built on top of it, things like package set.
NICS packages is built on top of this notion, but we don't even have a concrete understanding what that notion is under the hood.
Alright, here's stuff you hear all the time from people, like bring you, you know, it's their first week, second week of understanding what NICS is,
and you ask questions, like, I created this thing, like, now how do I build it?
How do I add it to NICS packages?
Oh, I built, I put this thing, but where do I put it?
How do I now bring it into NICS packages?
That's non-tribal, you have to kind of understand how to do this.
What am I, I'm trying to add a package, but I'm trying to do this in a NICS OS system.
And it starts showing up in packages.whatever, why?
Or it can't find it, again.
It's because, you know, they don't know how to compose these things, and it's hard, it's complicated, and we don't teach them well.
What's an overlay?
Oh, no, if they ask that question on their second day, you're done.
It's never going to, you're never going to be able to get past them.
Like, what's a fixed point?
Oh, here, it's difficult.
What's called package do?
It calls the package.
So is package the function, or what, even the naming of this is really odd.
Oh, I want to do flakes.
Okay, cool.
How do I add my package?
Oh, well, we've got to figure out what it is you're trying to do, how to compose it again, all sorts of issues.
Alright, so I'm going to go quickly through a few of these things as to, like, what they are, try to explain them,
and there's not going to be a full rendition of all these concepts, but just enough to get people to kind of,
somewhat of a common understanding, and maybe serve as a starting point for other research.
So call package, what is this thing?
It's a function itself.
It's going to call your definition with all the correct arguments from a scope.
What's a scope?
I'll go into that in a little more detail, but just consider this is the scope of things that it's available to it.
And it gives you some extra usability benefits.
We use this thing to kind of make things a little easier, to kind of organize mixed packages as a common way of doing it.
You don't have to use this approach, but this is what mixed packages uses.
This is kind of the idiom that we have.
There are some people experimenting with a few other idioms, specifically with, like, the module system,
but this is currently, like, the one that's in use.
A pretty good explanation of this is located at this URL.
All right.
This is kind of a roughly a way this works.
So we define a few functions to kind of, a few helper functions that's like how it works.
We start defining this call package for our package set, and then we go use this thing and make things.
I'll kind of break this down a little bit.
So we have a helper.
This helper function takes three arguments.
We give it a scope, some f, some functions.
This is the thing we were trying to name earlier.
I'm kind of leaving it a little bit unnamed for now.
And some extra stuff.
There's a little bit of NYX magic here to say, hey, go grab the correct arguments from your scope, okay, and then just go call the thing.
And then override it with some extras that are helpful.
It just calls a function with arguments from your scope.
Okay.
So now that we have that defined, now we give it a scope.
Here's a pretty simple scope, just, you know, a few values, not even packages.
Just some values.
Or this could be something massive, like all of NYX packages.
The entire scope might be available to you.
All right, so now we have a call package that's hopefully usable.
Except, I forgot one little piece.
We also extend this definition scope with something we haven't even defined yet.
Something that is the value of all the packages you're going to end up with once you're done.
This part's mind-boggling.
It has to do with the fixed point, lazy language.
It's not a very complicated thing, but this makes it complicated.
This is probably what makes it hard to understand.
But that's the idea is that it has access to everything once you are done.
So it captures some closure, namely the scope, and then it extends it with more things that you want to define.
All right.
So now you've got these two helpers, and now you say, hey, I want to extend it with these extra stuff.
Right, and now we have this reference packages, the packages, that whole lazy fixed point is done,
and we have extra things that are now brought in.
Cool, so now I should have a, b, c, and d.
Well, this notion of I want to add a few packages to the scope, you're going to do that a lot.
You're going to say, hey, I want to add these few things,
and so we give this notion of adding a few packages or modifying them or kind of having access to them.
We should call this thing an overlay.
And for various reasons, we start implementing now overlays.
Except this becomes almost impossible to understand, especially through a beginner.
Right, you get these arguments, final prev.
What are these things?
I have to use the final call package, if I use the prev, I get confused or mixed up,
or some things won't be available, or some things will be, but they'll be a previous version.
Again, we throw this diagram in front of people, and then they go, great.
This was, now I understand.
Right, no.
Again, that doesn't quite work, I find, in practice.
So overlays are powerful, though.
They are useful.
They solve a really good problem if you need to do something kind of deep in the dependency chain
and manage these sorts of things.
And when you know what you're doing, it is powerful.
But with power comes responsibility, and you can mess yourself up really easily.
You know, you've got nested sets, you've got infinite recursions,
like you can get confused, and very often this is what happens.
But most users don't need this, especially in the beginning.
Most users just say, I want to add just one more package, like the one I just started with,
my toy example from the tutorial, or the project I'm working on right now,
and I don't care about all this complexity, I don't need all that power.
So there's a few other issues.
It's hard to kind of, from an overlay, get that original function, that original thing.
You can't really get it, because the way we normally define this, right, is we call it.
So this thing is kind of hidden away, it's tucked away.
I mean, you can probably figure out a way to extract it out, but it's not trivial, it's not easy.
And we've muddled some of these concepts.
All right.
But overlays are essentially the correct way.
Like this is the way you're supposed to use the composition with mixed packages
for everything to work out right.
A lot of the machinery kind of expects you to kind of go down this route.
You don't have to, but again, a lot of the instructions, a lot of the idioms,
a lot of the tooling will expect it.
But it's hard to use.
All right.
There's another concept that we put in there.
We have this concept of a package set.
I don't even know how well this is even documented, but it's this idea of having a set of packages,
a set of things, and we extend it with a few other attributes.
And those attributes are the things that allow you to then further extend upon it.
That's useful, except everywhere where this is used, it's implemented slightly differently.
So that's another issue.
So we use this all over the place, right?
All the X packages, whatever packages stuff, is all the example of one of these package sets.
And it's useful, again, super handy, but it also can cause confusion,
because we don't explain this to people very well.
Try to override something in Python packages.
If someone could do that without reference or copy and pasting, I'll get you a sticker.
So one proposal is, hey, how about we actually look at all the different package sets that we have in X packages
and just make sure they're standardized.
Right now they're slightly different.
Some things are named slightly different things, or they're implemented in a kind of subtly different way.
Let's perhaps try standardize this.
And that way, if it's standardized, we can document it, we can talk about it.
Another notion that if you dig all the way into the internals is there,
but I don't think, I haven't really seen this written up.
Not really anywhere.
This is notion of a scope.
What is a scope, right?
I actually referred to this earlier on.
So a scope is basically saying, hey, take your package set.
I want to be able to extend upon it, and then later extract the portion that I extended it with.
I mean, this is really helpful when you do something along the lines of,
I want to add 10 packages to NICs packages and then pull out the 10 I just defined.
That's basically the trick that this thing allows you to do.
I'm not going to go into a lot of detail here, but it's a really nice trick,
it's a really nice concept because otherwise it's very manual.
Otherwise you have to always remember, oh, I then have to go extract back out of something.
I have to go inherit or grab those attributes.
Whereas with a scope, you could just say, make a scope, make a bunch of changes,
and then go essentially grab the def.
Go grab all the changes I just made and expose them now,
and only those things, not all 80,000 remaining packages in your scope.
So kind of it's a little bit like a right barrier that you could use.
But probably have time to go more into detail on that one, but it's a useful notion.
I don't think we talked about it enough.
It's yet another thing, another abstraction we've built on top of this original thing.
So let's talk about some ideas, some proposals of what we can do.
And some of these stuff we could actually do today,
it's just a matter of either making a decision or like thinking about it,
or maybe some discussions.
So first off, let's give this a name.
I don't actually care all that much what name we give it.
I do have some thoughts on that, but I just want to make sure we have a name
so we can communicate this.
So what name?
Package was mentioned, not quite correct.
It's related, but the by name construction that Selman has implemented
for Nix packages uses package function, package fun as the notion.
It's correct, but it's a little bit awkward.
Not super thrilled about that one.
Derivation, again, that's not quite what it is.
It's the thing that will produce one.
So because of that, I kind of like the, it's a proto derivation.
It shall become one, but this is technical and people already run away screaming
once we say the word derivation.
This will make people run away screaming even harder.
So that's not quite, that's for academic papers.
Blueprint, kind of nice, it kind of makes sense, right?
But it's pretty sterile, it's very rigid, it's not very fun, we're not very human.
I kind of like recipe, right?
Because it's this notion, you know, it's like you're cooking, it's human,
it's food, everyone likes food, I like food.
You have all sorts of other fun little concepts that could come into play,
like hey, you substitute one kind of flour for another,
or you substitute one ingredient for another,
you could tweak your ingredients when you're cooking a lot,
you know, you ran out of eggs, or I don't know, you don't like that much pepper.
So that kind of gives us the notions that we like,
but again, if someone comes up with a good name, I will go with it.
I'm not strongly attached to anything.
Any name is better than no name.
So, yeah, recipe is kind of a fun one.
It gives us cookbooks, variations, it's human, it's friendly, it's colorful,
blueprints are usually very static and ugly.
And here's some other things we could do.
So once we have this thing as a concept and it has a name,
well, let's say we're using flakes, not everyone is going to,
but if you want to, hey, why don't we just expose these things just directly?
Like, hey, here's the functions that I'm going to be using,
and then from this, you can, from pretty mechanically, pretty generically,
just create your overlays and create your actual hard packages,
actual packages that are for a particular system or for a particular environment.
But let's expose this as a top-level thing.
And this has some benefits.
So, hey, there's no system here.
You didn't have to pick a random system just so that you could expose it
so that other people can override the system you gave.
Like, that whole problem is now gone.
You could do cross-compiling because I could just pass whatever inputs here I want,
and these are self-contained.
We're not taking anything else from scope, hopefully.
There's an obvious translation.
So let's use that translation, implement it,
I'm sure you can come up with many different ways to do so
that have different usability trade-offs.
And let's try to make this like a top-level output.
That's just something that we use.
NICS packages should expose them.
Right now, one of the things people often probably do
is they use the file system layout of NICS packages
as like an unofficial API to grab these things.
Well, now that we're migrating all those, those are all going to break.
Those are all going to like the by-name construction,
do we want to really rely on where it is in the file system?
Like, no, let's just put them and expose them as the functions that they are.
Right? We're all about functional language.
Let's celebrate functions.
The next thing about this is now if you start using this in the Flake ecosystem,
there's no lock files needed.
This doesn't refer to anything else.
There's no system, there's no base NICS packages.
You take NICS packages and add this,
but I can grab recipes without even evaluating really much.
I just have to like parse the thing and grab the function.
I don't have to evaluate all of NICS packages just to end up overriding it
with another NICS package.
Like, that's where you get lots of memory and lock file bloat.
So, I don't even need the lock file.
I don't have to read the lock file.
I don't have to interpret it.
Right? And that prevents now a bunch of lock file bloat
and the consumers of this.
I could consume this thing without ever caring about that.
That's a benefit.
So what do I want?
Hey, all the various frameworks that we have out there,
like, just expose this as a top-level output.
Like, nothing stops.
Technically, like, I do this today, nothing stops us from doing it.
Now, there's no tooling that makes it easy, you know,
NICS Flake Show where other things don't present it
or we can have documentation for this thing
that tells you that this is a standard.
But we can start doing this and there's no, nothing stops us.
And we can start making it nicer and nicer to use over time.
So that's kind of an idea.
Yeah, I'm just kind of, I guess, reiterating that,
hey, these are just pure functions.
And that way, again, there's no lock file issue.
So some additional thoughts.
I guess they got cut off.
So you show this thing to people, show it to a beginner,
you go, okay, here, here's how you define your own thing,
here's what you do, and you start describing it to them.
And you almost inevitably, they think about this top thing
as if it was like an import statement.
And they always ask, what can I put here?
Like, what is this thing?
And you go, well, if you can grab stuff from NICS packages,
okay, cool.
But they also, you can grab additional stuff,
stuff that you just kind of composed in with your overlays.
And so they, I don't know, is Lib in there,
or My Package is in there, or they only, the upstream,
like, what's in there?
And we don't really expose it.
Like, this is the scope that's available to you at the time.
Why don't we actually expose it to people?
I want to be able to search within the context
of what is possible for this,
what's visible to call package, basically.
Let's let the user search it.
Why not?
It's valuable information.
They want to double check that what they just defined got in there.
It's good for exploration.
It's good for analysis.
You can now analyze what's available in your scope,
even, to make decisions.
So, that's another kind of thing,
a little bit beyond the first thought.
So what next?
We don't need technical changes.
Again, you could do this today.
You could do this by using a kind of flake schema.
You could just do it yourself and just kind of rely upon it
or check for it.
But I want to, all the time,
just kind of get this convention started,
and then at some point start to get some feedback.
Like, is this good?
Part of me talking here today is getting an idea from everyone.
Is this on the right track?
Is this helpful?
Do we need an RFC for this?
Then we start adding support into the various frameworks and libraries.
We can start, actually, at some point adding some support for this
or some of the utilities we have to make it easier.
And then, you know, we make that developer experience even better.
Like, how do we move these around?
How do we update them?
How do we borrow them from another cookbook?
How do we inherit cookbooks?
That sort of thing.
How do we mix this or combine this with all the teaching materials that we have?
So there's, you know, it's not super easy,
but we have things to do.
Another notion that I ran across
that actually seemed pretty useful in practice was
if I have defined all my recipes, right,
you can kind of say, hey, well, using some base,
add this stuff to it.
And once you can define this and use this thing,
it kind of looks like the with keyword.
It has a lot of other, like, benefits.
You can start to define how it interprets what you gave it.
So here you can start to kind of say,
oh, I don't have to import the thing.
I can just refer to it by path.
Or other niceties.
But it's another kind of useful abstraction that I'll just say has worked out well
in my opinion.
All right.
I think that's a blank slide.
I don't have time for a demo, but, like, this kind of works.
I use it today.
It's useful.
It's friendly.
It gives us a lot of benefits that I like.
I'd like to expand upon it and make it kind of less of, like,
a unique, strange thing that I do and something more that might be a bit more common.
I've got some references in here.
And I am willing to talk to people about this sort of idea,
this sort of thing.
So come talk to me.
And I guess we're open for thoughts, questions, comments, tomatoes.
Thank you very much, Tom.
Is there any questions?
Thanks for your presentation.
If you want to expose the recipes inside of Flake, how would NixBuild work?
So NixBuild, right, doesn't take a recipe.
NixBuild needs a package.
This is where the distinction matters.
So you would still have to say, hey, my packages are, with this base Nix packages,
bringing all my recipes.
Right?
But that's a simple kind of way to say, it's kind of like the for all systems.
It's kind of a very simple helper that can do this thing for you.
And that could just be kind of injected into the templates.
It could be used and tell people, hey, here's how you do it.
Here's the easy answer unless you need more power to, like, only, you know,
but the easy answer will be, hey, convert all these recipes into packages.
Boom.
Now NixBuild has it available to you.
So you would need a lock file and Nix packages input inside of Flake?
Yes.
So if you want to build a package, yes, obviously then you need a lock file,
other things, but I can still grab those recipes directly without reading your input,
without reading your lock file, because it's independent of them.
But yes, if you want to actually build a concrete thing,
well, you have to have a concrete base.
Any other questions?
Back.
Yeah, this is kind of just a curious one, a technical one.
In your proto derivations example, you have a function which takes nothing.
And it, well, sorry, it takes a set, but the set has nothing in it.
And it would be unexpected argument if you passed it.
Yeah, my data at the bottom, what is the purpose of my data by comparison to everything else?
Why put the set as an argument there?
Well, it just doesn't need arguments.
Great.
In that case, call package goes into no arguments.
Let me call it with the empty function, and the result in this case isn't even actually a package.
It's just some data.
All right, so it's a way of interfacing call package because it expects the function as its signature.
Yeah, you could process these in such a way.
For example, one trick I've been playing with is to process just a simple string as if it was the arguments to run command.
It's kind of nice.
You could just throw a script in there without having to define a lot of things.
This is kind of serves a marker.
Again, you could interpret this set in different ways.
I don't know exactly what the best way is, but you can kind of figure out different types are handled differently.
So you could detect, hey, oh, this thing is a function.
Great.
In that case, defer that evaluation, or it's a piece of data converted to a run command, or it's a path.
It's a path, okay, imported.
My call package does this for you right now.
If you give call package a path, it goes, oh, it's a path.
Let me import it, and I'm expecting inside of that path there to be a function.
Or you can pass it directly a function.
So this is just another little trick.
Again, whether we use it or not, I don't really.
Something fun.
Thanks, Tom.
Let's give it up for Tom again.