Okay, all right, is the mic working?
Yeah.
All right.
Great.
Okay.
All right.
Welcome, everyone.
I'm going to talk about a program that I've started at Red Hat called Kernel Development
Learning Pipeline that I run with a small group of engineers kind of as a side project.
So first, I'll talk about what it is, first of all, why it's a good idea, why we think
it's a good idea and we're doing it, a little bit about the program and some of the growth
that we've had in the program over the past year, two years approximately, and then conclude
with some resources about the program.
So first of all, what is KDLP?
So that stands for Kernel Development Learning Pipeline, credit for that acronym to Julia
Denham, actually, and we are building a comprehensive pipeline for Linux kernel talent and low-level
talent more generally, because as it turns out, there's not a lot of younger people getting
into the kernel.
It's not an extremely popular area of study, it's barely really taught in school.
So right now, the main component is we have a course that we're teaching at UMass Lowell
and also on the Linux Foundation's platform.
We are trying to recruit interns from this course and from this program and through kind
of our network.
We bring people in and then they serve as TAs and help develop the course and improve
the content.
And then ideally, we bring them in and recruit them full-time, that's the goal.
So yeah, why is this a good idea?
So like I was saying, a lot of senior Linux kernel engineers are getting somewhat close
to retirement.
That's kind of, definitely a much higher, I think, average age in kernel and in like
low-level engineering, at least in the US that I've noticed, than in other areas of
software engineering.
And it takes a very long time for people to learn the Linux kernel, especially today,
because a lot of people don't even learn C in school and they're generally separated
from a lot of the low-level computer science concepts.
Things that people 20, 30, 40 years ago, so I'm told, were learned kind of as a standard
thing.
Today, it's sort of a niche topic.
There are people getting into it, but only kind of in-ditch communities at certain schools
if you're exposed to certain people or certain online communities.
So the kernel itself is barely taught.
People take maybe one class in opening systems and a lot of the tools, like some of the more
advanced get usage, for example.
It's difficult to learn that kind of stuff.
How do people learn how to do email patches without trying and failing and just getting
roasted online?
It's kind of difficult.
So for a lot of companies, it's difficult to find talent.
I mean, I know at Red Hat, it can be somewhat difficult to find people who have some Linux
kernel knowledge and are actually interested in doing it.
There's lots of people who are capable of doing it and just maybe don't like it that
much.
So, of course, the way that we're pitching this to Red Hat and the way that it's a good
idea for any company is that we think we can bring more value to the company for a lower
cost.
But for the community, what we are trying to bring is to train the next generation of
developers from all sorts of different backgrounds, as opposed to just people who happen to bump
into somebody at a recruiting event, which is how I got into the kernel.
I just happened to run into somebody at an event that he wasn't even supposed to be
at, in fact.
So a little overview.
So there's kind of three main sections of the program.
So the first is this kernel development course that we've developed that is taught at UMass
Lool.
So we've created this from scratch using, originally based on the Linux device drivers
third edition book, but we've kind of gone further from that.
That's actually one of the latest books on Linux kernel engineering and it's published
in 2005.
But yeah, so we've been working on developing our own curriculum with a number of different
labs and some slide decks and kind of a whole, you know, a set of things that people could
do to learn the kernel.
And then from there, we tried to bring people into internships as we're able to do that
or other kind of more second level or hands on experiences where they can further develop
those skills and then ideally bring them in as full time engineers at Red Hat or I mean
the beauty of open source is if they get recruited somewhere else or if they go work
somewhere else in the open source community, that's still, that's a win for everyone in
the community.
It's a win for all the companies working with the community because everyone working on
the kernel is working on the kernel.
So a little more about the course.
The goal is to introduce students to Linux kernel development.
You know, obviously we can't teach them the entire kernel.
We can barely even introduce them to one subsystem, but the goal is to teach them kind of what
they need to learn to teach themselves and to teach them what they need to know to work
in one of these open source communities, specifically the Linux kernel community, but those skills
I think are pretty applicable to other kind of more niche open source areas that can be
somewhat intimidating for a lot of people to get into.
We mostly just require C language skills.
I mean that's obviously the most important thing when you're working in the kernel.
That's a basic Linux experience, some programming experience in general, but not need a ton
of Linux kernel experience to get started, but really the most critical thing to see.
So we teach kind of an overview of just various kernel features and subsystems, kind of describe
like a map of the kernel, what there is, what, you know, how the different pieces connect
together, how you can interact with it.
We approach through device driver development, that's a good way to get people into, you
know, to work with a lot of the different APIs, because to write a device driver you
need to interact with a lot of different areas and you end up building something that's somewhat
kind of complete piece that works on its own, and then we introduce people to more advanced
usage of Git.
I mean people may be familiar with GitHub, but you know, believe it or not, a lot of
people don't know the difference between Git and GitHub.
So we help out with that and get them started with email patches and rebases and things
like that.
We also talk about BPF Trace and other tracing, which are things that they, I don't think
BPF Trace or BPF in general is taught in any school that I know of, at least not in undergrad
courses, you know, NCSCope and other ways to explore large projects and repositories,
because you know, in a computer science program people generally don't get exposed to working
on like large ongoing projects.
You know, you work on some, you know, one and done thing, and you know, then you send
it in and no one else looks at it and you get your grade and you move on, right?
So we're introducing people to work on larger code bases, which is something that's somewhat
unusual for a university course.
So of course, all these course materials and the assignments completely open source.
We've linked it, it's all on our website, and we're continuously improving it.
You know, the source for the website is also on GitHub, you know, if anyone has any suggestions
or changes, we're always open to them.
So we have undergraduate and graduate level university courses that run at UMass Lowell
that we've created over the past couple of years.
It became kind of a full-fledged course in the fall semester of last year.
So it's relatively new.
And this semester we were also running it as a kind of dual program with the Linux Foundation's
mentorship platform.
So we have people applying on there to, I will also talk a little bit more about that
later.
We run through the course and kind of work with at the same schedule and at the same
pace as the people at UMass Lowell, but they're from around the world and they don't need
to be enrolled as students.
They can come from anywhere, right?
And kind of the bottom line is we try to reimagine how to teach this kind of material from first
principles because it doesn't really make sense to teach it from a typical, like memorize
this and do the exam kind of perspective, right?
The goal is to get people to be able to explain things in their own words and to work on open
source projects and work in open source communities.
So we've replaced exams with presentations where they actually explain the work that
they've done in their own words to their peers.
So from there we recruit people as interns and try to bring them in.
So we find people in the course who are enthusiastic and capable about so by the time we bring
them in to the internship, they also have a lot of prerequisite knowledge that they
would need.
And so they're able to hit the ground running and do a lot more in their three to six month
period or longer than they otherwise would have because actually, so I had a, the manager
who hired me at Red Hat originally as a co-op, he said he didn't like to do summer internships
in the kernel because by the time three months are over, they've just gotten on boarded.
They've just started to understand how to work with a Linux kernel and then it's over
and then they got to go.
So he only really did co-ops.
But the goal with this is if we kind of vet people ahead of time and we give them the
skills that they need and give them kind of, you know, the get skills and the, you know,
they compile the kernel and they have the background, they actually can get some value
from an internship and we're more likely to bring people into internships who actually
want to work in the kernel and then they're more likely to go off and, you know, and work
in the kernel in general or work in open source communities.
So we pair them up with rail engineers and, you know, they work on various initiatives
within Red Hat that are, you know, that are, need people to learn these kind of new and
old combinations of old and new skills.
But the goal is to train these new people to work in those areas.
So we also bring them in to help with our program and we've had people come in, they
take the course and then they come in and they turn around and then they're TAs and
they work on improving the course.
Maybe they do a couple lectures here and there.
They do some grading and they kind of service TAs.
So another thing that we are doing is we have this kind of new kernel devs group within
Red Hat, which I thought I'll just briefly mention.
It's a little bit tangential, but, you know, once people get into Red Hat and when people
are working in the kernel, especially in a remote job, which a lot of these positions
are, people get very separated, they get very siloed, there's not a lot of socialization.
So we run a group to, you know, just bring people together with, for a meeting with very
little agenda.
You know, occasionally we have a presentation here and there, but just to talk about things
and share concepts and tools and questions and, you know, it's a Red Hat specific thing,
so they can ask about specific things related to their job and a place where, you know,
there's, I mean, managers are allowed, but it's not a specific structured meeting for
a business purpose.
And I think people who, you know, you might get that in an office somewhere when you're
just walking around and talking to people, but it's a little bit more difficult to get
when you're in a remote position.
So we also bring interns who we've recruited into that group, you know, and people from
elsewhere within Red Hat who are interested in the kernel and they can ask questions and
find resources and, you know, if they enjoy it, they can switch into the kernel.
Because there are actually a number of people who are interested and they just don't know
where to start.
They're just not a lot of good resources.
So now I'll talk about some of the growth of this program over the last couple of years.
So first of all, we've partnered with Red Hat's main educational initiative, Red Hat Academy,
which we found out about only through doing this program.
But they work with universities mainly on delivering and facilitating kind of systems
management, you know, what do you call it, system administration and cyber reliability
engineering, kind of the more, you know, the standard kind of Red Hat certifications that
they do.
But they don't have much software engineering.
In fact, I don't think they had any software engineering component at all.
So they were happy to work with us and, you know, we're happy to work with them.
And they made us these nice posters, which we have down here if anyone's interested.
Or I guess they're called like leave behind sheets, technically.
We also ran a workshop last year for interns and co-ops in Ireland.
And that was just kind of every other week, kind of a casual thing, you know, over time
we had fewer and fewer people, to be honest.
But the people who stayed around really enjoyed it and I think hopefully some of them will
end up working in the space or, you know, in open source generally, hopefully the feedback
for them was generally fairly positive.
We've also been connecting to various educational programs in different countries.
A lot of this stuff's very preliminary, but we're hoping to kind of package what we've
done in our course in a way that can be replicated at other places and other universities and
other countries to bring people in from different places in the world.
And then we've also partnered with the Linux Foundation, specifically their mentorship
platform, which I will talk about next.
So like I was saying, the Linux Foundation.
So we have just put our course on the Linux Foundation's platform essentially as a mentorship,
but it's the same thing as the course.
We've worked with Shua Khan, she's been very helpful.
She runs this Linux Foundation's mentorship platform and, you know, great person to get
in touch with if you're interested in learning the kernel or have questions about this.
She's also a kernel maintainer if you're not familiar.
So we are running the same course, and this is kind of experimental, we're running the
same schedule with the same assignments with a group who are doing the course for credit
at UMass Lowell, and at the same time, people who are just, they just apply it online, and
they're just doing it.
There are people, you know, just on random continents, some of them are students, some
of them are working, some of them are doing, you know, I have no idea, honestly, some of
them are just probably, you know, having a good time, but they're doing the same assignments
and on the same schedule, and, you know, they submit their assignments to a different mailing
list than the UMass Lowell cohort, but they're in a shared Discord server and they're interacting,
and, you know, we're seeing how that goes, and it's an extremely diverse group of mentees.
We have people from, I think like five different continents, you know, people from various places
in Europe, a couple in America, South America, someone in Mexico, like three or four from
Africa, Nigeria, Kenya.
So a very interesting group, a couple in India as well.
So a couple of statistics on what we've been able to do.
It's a fairly new program, but I am happy to show some results.
We were able to hire two people full-time from internships who originally did the, went
through the class.
This was during a year that was very difficult to hire people.
There was a hiring freeze, so I think two is still, it's a lot more than zero.
Two people who did co-ops or internships with us were recruited to Amazon and Microsoft,
but one of them liked Red Hat so much and liked working with Open Source so much better
than Amazon that he is pretty interested in coming back, leaving Amazon and coming back
to Red Hat, which is pretty cool.
But overall, we've had seven interns and co-ops who have been trained and have gone
through this particular program via this KDLP thing.
And last semester, we had our biggest class, well, until this semester, we had 12 students,
undergraduate, mostly graduate students, graduate and undergraduate courses are the same, which
I found out, same content.
And we have about a dozen, a couple more than a dozen students this semester, and the course
is actively going on right now.
It just started a couple of weeks ago, and I think about a dozen in the LFX mentorship
as well.
And this is by far the most diverse group by gender and location, which is pretty interesting.
So now, just briefly, some program information.
So the team is, well, it's me.
Julie Denham is a program manager.
I think she's in the chat answering questions, if anyone's interested, back in Boston.
Charles Marabelle, he's the content lead for the course.
And Dennis Alexandrov, he is an intern who's been extended and also TA.
He's been working on the course since last summer, originally went through the course
from McGill University.
And yeah, acknowledgments.
I just want to give a shout out to two Red Hatters in particular.
First Heidi Dempsey, the research and innovation director in North America.
She's been a longtime supporter of this program from the beginning, from like, you know, before
it was even an idea, you know, when we were talking about doing something in, like, February
2019.
So her support's been great, and, you know, recommend, you know, doing research and innovation
with her if anyone's, you know, interested in doing that.
And of course, Mike McGrath, the vice president of RELL, who is the executive sponsor for
this program within Red Hat.
Now, a couple resources.
We have a mailing list, which is linked up there, which we pretty much just used for
giving a quarterly update newsletter, which we just started this quarter.
And it's read only, so you're not going to get spammed with a ton of information.
If you subscribe to that mailing list, you get invited to a weekly office hour session.
That happens Tuesday at noon Eastern Standard Time, which just for asking questions about
the program or the kernel in general, or you can just send me or one of us an email.
And then on the right, we have our website.
We have a section of our website that talks about the information, specifically the structure
and content of the course.
We have the page on the UMass Law Catalog that talks about the, you know, that just has
a course description, so in case anyone's interested in checking that out.
And the mentorship page on LFX.
Then down there on the bottom right, we do have a crowdfunding page on LFX.
If you want to support the program development and diverse engineers, it's a very, you know,
we'll be able to bring people on through that platform as TAs, potentially give them, you
know, bring them on and continue them kind of beyond what we're able to do with just
Red Hat.
All right, any questions?
Hi, I think there is an offer and demand problem.
If you consider every Linux developers, it will be like 1,000 JavaScript developers.
And also people at the university, if you give them the choice between JavaScript and
Linux, they will have more opportunities going with the JavaScript.
They cannot do a mistake.
However, if they go with the kernel, like, I don't know, in my country, for example, there's
a lot of, there's a lot less jobs and you have less opportunities.
So what's your opinion on this?
Does people know the choice between like classic software development and kernel software development?
So if I understand that correctly, you're saying there's fewer jobs in the kernel.
And so, you know, what would I say about, just about what, it was like 1,000 to one
or something, you said, well, I mean, that's true, but the competition for those jobs is
much more intense.
It's much kind of broader, you know, skill area.
I mean, it's definitely, you know, I'm not condemning JavaScript developers.
JavaScript's fine.
It's necessary, you know, the web runs on it for all its problems.
But it's also true that, I mean, working in the kernel, you know, the Linux kernel, there
are fewer jobs, but there are a number of very solid jobs, and it doesn't, learning
Linux kernel skills is somewhat transferable to other areas of development.
You don't have to work as a Linux kernel developer.
You don't have to, you know, go to Red Hat and work on RHEL and do backports or whatever.
Right?
You can kind of just do it on your own.
You can do, you know, just some other low-level project and just having knowledge of the kernel
and how it works and the knowledge necessary to work on the kernel, I think, could benefit
people in many other areas and just is an overall boost to people's skills that is somewhat
rare to find these days.
So even for a JavaScript developer, if they know how the kernel works and if they know,
you know, behind the scenes, you know, what's going on at a system level, I think that is,
you know, can't be anything but a good thing.
Thanks for the talk.
I have a question about the audience, so is it mainly targeting students at the college
level and then partnering with university, or do you also consider a conversion program
like existing software developers that want to switch to more low-level kernel development?
Yeah, so the course at UMass Lowell is, you know, a course at UMass Lowell for people
enrolled at UMass Lowell, but the program on LFX, the whole idea of putting it on there
is that we could accept people from anywhere in the world doing anything, students, non-students,
you know, U.S., non-U.S., whatever.
So any background, as long as they have some knowledge of C, good internet access, decent
English and a computer, then we can take them.
The only limitation is our ability to grade the assignments and handle that many students
and like the resources that we have to work on the program, because this is, you know,
for the three engineers and, you know, potentially a couple more people who are interested and,
you know, a couple of interns, right, it's, this is all a side project for us.
Like, I'm mainly a RELL engineer.
That's my main job.
Like, this is just a side project, so we don't have a ton of resources and, yeah, we're basically
limited by how much time and resources we have to run the program.
But we're, yeah, we encourage people to apply from all sorts of different backgrounds.
Okay, thank you.
So back in the day when I used to participate in the development of kernel, it required
us to basically build the entire mainline.
Your slide showed something with Raspberry Pis.
Is it feasible to build the entire kernel using a Raspberry Pi?
So, that originally comes from, we used to do a lot more stuff with Raspberry Pis before
the chip shortage.
It became more difficult to get them and then, you know, I think generally like a lot of
the introductory stuff we were doing didn't require them as much, so we moved somewhat
away from them.
But, yes, we did work with Raspberry Pis and we were compiling the kernel on the Raspberry
Pi.
So I was able to do it on, on the three, it took a lot of weird tricks and it took like
a day, you know, because I think it was like single threaded, and on the four, I could
do it.
It was a little bit faster, but if you didn't have cooling, the system would overheat unless
you used fewer than four cores.
But you can do it, and in fact, you can install Fedora, you can install CentOS, and you can
install Rail.
Oh, yeah.
Yeah, I know that.
Yeah, all right, that was my question, thank you.
Hey, he had said, with the, the alternate, the non-university rate for doing it, you've
got people here working at the same time, possibly, I was just wondering, since it's
also, you've got the same assignment schedule, what is the time commitment like for someone
trying to get into it, but might have full-time work?
What is the time commitment like?
Because it depends how quickly you complete the assignments.
Okay, so it's, well, the, the lectures are the, the sessions that we have, which are,
you know, on, on Google Meet, they're also recorded and we post them there about their
two of them per week, generally, and they're, they're each an hour and 15 minutes, and it
goes for, you know, like a standard 12-week semester.
In terms of the assignments, I mean, if you're, you know, I think at the beginning, setting
up the environment, send email patches, if you haven't done that before, and compiling
the kernel, and, you know, we have one assignment where you write a shell, you know, that can
be, I mean, yeah, it depends how familiar you are with, with some of the concepts.
But, you know, I think of it as, it's just a, like, same commitment as a, you know, maybe
medium to, like, junior to senior level undergraduate computer science course, approximately.
I mean, that's, that's how we're designing it.
Sure.
Hiya, thanks for the talk.
Any of these courses, like, I'm, I'm curious about back, back-porting and forward-porting
stuff.
So, when, when, when you're doing this course work, are you working on a specific version
of the kernel?
Like, yeah, this course is going to be on 6.2, because I find that a lot of problems that
I run into is, okay, this driver was just introduced, let's say, in October, but I need
it on 5.13, so I need to back-port it, and that might be trivial, right, might just be
copying some files, but if the API has changed in some way, it can be really difficult to
figure out how it changed, because the documentation, to my knowledge, is not great there.
So, is that some, something that you actually sort of discuss in these courses?
We haven't done back-porting specifically.
We don't, I don't think we even, we don't have any assignments that's specifically
asked for, for back-ports, because we have to, we design them in a way that, you know,
each student is doing, like, relatively unique work, so it's kind of, you know, and they're
all posting it in a, in a public mailing list, so, you know, we need to figure out how we
can kind of either generate assignments or, or figure out assignments that are basically
by design, difficult to impossible to, to just, to cheat in.
And so, so with back-porting, I mean, we could figure something out.
We haven't really looked into it.
We talk about, you know, get cherry-pick and rebase, and we talk about the, the background
that you would need to do that, and I think we may have done a demonstration of it in
the course.
But that's a good idea.
Do you anticipate that the recent admission of Rust as a programming language, at least
for driver development in the Linux kernel, will cause renewed interest, especially among
younger talent?
And are you already anticipating for that in the program?
Potentially, yeah, we've, we've discussed it, and we've heard some people talking about
it.
We had a few people who were in the course, who were excited about Rust, and Rust in
the kernel.
So we, we have seen some interest in the kernel.
Probably, yeah, it's probably increased interest from young people, I would say.
But personally, I need to, I need to learn Rust.
I don't know Rust.
Do you mind if I do a little bit of, oh, sure, yeah, they're here, but people take us
with them.
where you were asked, you know, to work on vice-drivers and then, you know, non-handling
veterans and so on, actually, when we fly, so they were teaching us since, I think it's
never been more, you know, for the people, maybe like, what are you, and you're a serious
person.
I mean, I'm probably fine.
Sorry, let me get the mic away from you.
Oh, okay.
Yeah.
All right.