Hello. Can you see my slides? Yeah. I have only wide background, so the light at the
top shouldn't be a big issue. And yeah, we're good to go. Okay. Hi, I'm Stefan. I work on
generally open source stuff as a freelancer, and I'm here to present something I've been
working on as a site project in the last few months. This is part of the Game of Trees project,
which I started in November 2017 at an OpenBSD hackathon in Berlin. It's compatible with
Git repositories and the Git network protocol, but apart from that, it's not trying to replicate
Git specifically, but it's just the idea to reuse these formats because they're very widely used.
And they're fairly okay and well designed, so we can just keep using them and not make up our
own. And yeah, because it's written on OpenBSD, it uses a lot of OpenBSD specific APIs. There's
actually a portable version that's maintained by Thomas Adam, who also does the T-Max terminal
multiplexer portable version, and you can install this on various systems. And I think Thomas always
likes to also explore more options for other systems. If you're interested, if yours is not
listed, you can talk to him. And yeah, it's ISC licensed because it aims to be basically as
pleasing to OpenBSD developers as possible. That's the whole idea. Now, what we currently have
is what's working really well is the client side. And this is basically sort of feature complete at
this point. You might want to have some more convenience things, but all the basics are there.
Everything is working. You have several frontends which I'll present in the following slides. You
have a lot of code that's shared by these frontends, which I've labeled library here because it's in
the lib directory of the source tree. One thing that this program does, which is very specific,
is that it will not touch repository data outside of programs that are separate and are called
lib exec helpers. From the programs point of view, if you use the library, you don't see this. You
just like say open a repository and fetch me some objects and so on. But internally, it will
actually start other programs that restricts themselves a lot using pledge and unveil and so
on. And those will actually pass the repository data. This is the current list of commands. And
I'm quite happy with this set actually. I've been working with this set for the last five years or
so. They've slowly been added over time. But I feel very productive with these. And I don't miss
anything. I know that some people would like some additional things. But at this point,
we're mostly like fine tuning. And you can read the manual page on this URL if you like.
You can actually read it from start to finish in order to get a good idea of how the system works
and how it's supposed to be used. There's also a got admin utility which sort of mirrors CVS
admin or SVN admin in the sense that if you're doing something that only requires like specific
things where you do something with a repository specifically, you would use that command. This
isn't complete. There are some things that I would still like to add here, which we'll go into
later. But it's already prepared a lot of code for the server that I'll talk about. Because
for example, dealing with pack files is necessary for the server as well as this tool. We have a
curses command line, a base terminal browser thing. You can read commits with that and look at
diffs and blame files and so on. It's working really well. And most recently, there's a developer
Mark Jamsak who added a lot of convenience to this like vertical scrolling, diff stat display and
all sorts of nice things. It doesn't work quite well on repositories that have a lot of merge
commits. I found that some repositories are hard to browse if they use a lot of merges. But for
simple repositories, it's really good. And if something is missing and you feel like you would
like to use this on a repository with lots of merges, you can please make suggestions as to
what we could improve there. You also have a web front end, which is sort of like CVS web or
VUVC. And it's also using the God code internally to show you files on a web browser and commits
and logs and so on. That's written by Tracy Emery. And most recently, Omar Polo has been doing a
lot of refactoring there and added a templating mechanism, for example, to deal with generating
the HTML, not from printf, but with something more generic. And it's quite nice. It also has
RSS feeds for tags, which is probably rarely outdated, but I think it's kind of nice. You can
be notified of new releases that way. Okay, so about the server. So the goal of one of the major
milestones for any version control system that's ever been developed is that eventually you want
to be self hosting. And so far, we've been using a Gitulite setup for this project. And that's
working well, but I would really like to be able to run this on an OpenBSD server using my own
code. So after putting this off for a long time, because I always thought it would be a lot of
work, I finally ran out of things to do on the client side and said, okay, I'm going to look into
several things now and started talking to people at Hackathons in September and summer last year,
basically, and started working in September. By now, you can install it on OpenBSD current. It's
not yet in the portable version. Thomas and Omar were going to look at that, but it might take
some time still, but eventually it should arrive there. Now, the main use cases I want to support
with this are exactly two. One is, of course, I want to be self hosting for my own open source
projects and maybe also private repositories. And the other is I want to enable what OpenBSD is
using now with CVS, which is anonymous distribution of source code over SSH, where you know that the
server you talk to is genuine and should have the right source code for you, but the client doesn't
need to authenticate. And every time I want to get source code from a platform like GitHub or
GitLab or other forages that exist with God, I have to upload an SSH key because they will not
accept my SSH connection. And because God only uses SSH, it doesn't implement HTTP support. This
is really annoying. And it's not really a technical problem to do this. It's just basically that in
their software, they didn't foresee this use case. But I think it's very nice. And you can actually
go and try this now if you like. This is the code that I'm talking about running on a server and
it's serving God code and God portable. You have the Husky fingerprints, which you can not take a
photo of or whatever. It's also on the website. And yeah, if all of you all at the same time would
now go and trigger this, you'd probably trap my SSH rate limiter, especially if HostM is behind
that, which I hope not. But yeah, be gentle. Maybe if you want to clone from this repo,
pick a slide number in your head from between 10 and 37. And when the slide comes up,
you start your clone, then you'll be fine. So yeah, I'd like to explain a bit what the
Git protocol is doing because without knowing this, you will not understand what a server should be
doing. And it turns out that if you leave out HTTP and all this stuff and just concentrate on the
playing it protocol, it's actually really quite simple. If you don't, if you also ignore some
protocol extensions, which we haven't implemented yet. So this is like really a bare bone clone
that that we will go through. It's not very complicated. The main thing to understand is
that when you're using SSH, the Git client will actually go and run the login shell of the user
and then give that a command to run. And Git basically hardcoded the names of these executables
in its protocol. So you cannot be a Git protocol without calling Git upload pack on the server
when you log in, right? Also there's Git receive pack for the other side when you're when you're
when you're sending something. Anyway, if you run God clone with the dash v flag, you will see a
trace that is very similar to what I'm showing now. It's I've left out a few bits. But initially,
so this is only Git protocol version zero slash one Git protocol version two changed a bit some
things in a good way. But I haven't implemented that. So we're seeing a version one trace.
Initially, the server just sends one message which says I have one of the branches I have has this
comment hash and this name. And oh, I also have some capabilities. You can see in the trace,
these are hidden behind a null byte. Because I suppose very old versions of Git clients didn't
really understand the capabilities yet and the null byte made them not read that part of the
message. So they and also for version two, they did the same thing, hiding a version
announcements behind two null bytes, because then the next kind, you know, this is a bit
hacky, but seems to work. Don't worry about the capability capabilities. It's not important what
they are. What's important to understand also is that each message is wrapped in a packet line,
they call it. And that's simply a length plus data framing format for these messages.
So then the server keeps sending messages for every branch it has. And here's one more,
its main branch happens to be the same as had, because had is a similar to main, but you know,
not important. And the client just keeps storing these. And eventually the server sends a flush
packet, which is just a zero length packet and says, okay, I'm done. And in response to which
the client will tell the server what it wants. So the client sends similar messages also includes
its capabilities in the first message it's sending. And basically says, oh, yeah, I want this
commit and this commit and this commit. And eventually it also sends a flush packet to
terminate that list. Now if we're doing a clone, right? So we have nothing. But if we already
had commits, we could now tell the server what we have by sending half lines, which look just
the same as the want lines with more commit IDs. And the server then builds a second set of commits
in its memory to say like, oh, okay, the client has all of these already, I don't need to send
those and don't need to send any objects that are hanging off these commits. It's basically just an
optimization to keep the pack file small that will be sent next, right? So you're not doing a full
clone every time you do a full clone initially. And then once you have something, you tell the
server what you already have. So you only fetch the new stuff. And yeah, because we're doing a clone,
we're just setting a server we're done. And now the client's protocol is already finished. So this
is basically the last message the client will ever send. And the server sends one more message in
response, which is in this case, a knack, not acknowledged. I don't know why they chose these
words, aka knack. But essentially what these do is for a knack, the server keeps sending knacks
while the clients are sending half lines to say like, I haven't found a common ancestor yet,
please send me more. Because without a common ancestor, the server cannot determine a subset
of the commit graph to use for the pack file. Because if the client sends totally unrelated
commit hashes, the server doesn't know, then the server cannot use this to optimize the pack
file. So it keeps sending knack. And in another case where you would have a common ancestor,
the server would send an act and commit hash. And the client would then stop sending half lines
for this branch. The exact details of this part of the protocol are a bit complicated. And they
kept adding extensions to this behavior. So the actual knack and act processing depends on various
options that you can set in the protocol, which are all documented in the Git docs. But it's not
important for us here now. Basically, the server just tells us, well, I have no common ancestors
because you don't have any commits. That's fine. And then the server starts calculating the set of
objects it wants to put in the pack file. And what Sony has colored, Git calls us something else.
It calls us like counting and enumerate. I don't know which step does what. But what we do is we
have the whole graph and we keep coloring nodes in the graph. It's kind of like mine or theirs or
something like this. And then eventually we have a subsection, which in this case would be all of
it. And of all the commits first, and then you go through these commits and traverse all the trees
and collect all the trees and blobs that you need to include for the client. And then you have a
lot of objects. And you sort them in a certain way. And you go through and check whether you
already have a delta for any of these objects and whether the delta base will also be included in
the packet sending so that you can avoid creating a delta for this object. You just reuse the delta
that you already have somewhere, which is an optimization for performance and very important.
If you don't do that, your server is going to be super slow. And then you deltify some of the
rest of the objects and you're good to go. Now you know what you need to know to start
generating a pack file stream. And you start sending this out to the client. And the client
downloads it. Once it has everything indexes the pack, which is a step where you have the pack file,
which is full of compressed and deltified objects. You don't know what's in it because the server
didn't tell you anything about the objects. You just told the server, send me this. The server
sends you something. Now you don't know what's in there. And to use the pack file, you always need
to have an index for it, which tells you which object ID is at which offset in the pack file.
So you just read the whole thing. And because Git uses intrinsic object identifiers, you can
calculate the IDs yourself based on the contents of the blobs and the trees and the commits and so on.
So you build that up. And then for any of the deltified objects, you also need to make sure
that you can actually combine all the deltas to get the right content. And that's the last step.
That takes quite a while. And then once you're done with that on a big pack anyway,
it takes a long time. And then once you have that, you know, okay, I have this pack.
The commit I want it is in there. All the objects that are hanging off of it are,
you know, by nature of the hashing structure that Git is using are there. So that's fine,
we're going to use this. Then you just create a reference for the Git client to find its initial
commit and you can use the repository. In the push case, it works slightly differently.
You still have this reference list announcement at the beginning. And instead of saying what it
wants, the client proposes reference updates to say, oh, I would like to change the main branch
to point to this commit. And I would like to change or add this tag or something like this.
And then it just sends a pack file. And then the server has to index this and
figure out that everything is fine. And whether it wants to change these references or not.
And give feedback to the client to say, like, yes, okay, I've,
you have changed this branch or you've added this tag and so on. So that's it for the protocol
overview. You can find a lot of documentation in Git source tree about this.
They moved the files recently. So if you have an older
Git source checkout, it might still be in documentation slash technical,
but in the current version, it's in documentation slash Git protocol
dash packed attack system is the main one for this, but there are also others,
similarly named files, which you can also read if you want to know more.
Okay, another thing we need to talk about, because this is important to understand why
we would need to write our own server in the first place, because there are already several
server implementations, right? Why do we want our own? Well, when you write server software,
especially an open BSD, there are a few design patterns that we use that are not
commonly used elsewhere, I would say. I mean, I've never really seen them used widely outside
this project, so it's a bit unique in that way and the way it does things, but these things
are important to us. So for example, you know that SSH recently had a release where they had a
double free and advisory products like yesterday, I think, or two days ago said like, oh, this is
not believed to be exploitable. That is because of this. It's not because SSH code is generally
great or something. It's because of the design patterns. And so we want these design patterns
to be used. And so one of the things you do is that you split your program into several processes
that have different tasks. And for each task, you decide what kind of system calls does this task
need? And how can I make sure that a process that has network access isn't also able to
start new programs or open files and so on. There's unveil which restricts the view of the file
system and allows you to completely hide like your dot SSH directory, for example. And other
things, basically, it says the program, for example, the God client says, I need the repository,
I need the work tree, I need slash temp, that's all I need to see. And I don't need to see anything
else. When you start new programs, you always fork an exec, which means that when you do the
exec, the program will be restarted from scratch and OpenBSD's memory randomization will kick in
and load all the code segments and text segments and stuff in different locations again,
which you do for every request so that when somebody learns information about the outer space
from an info league, they cannot use it on the next request. You have messages over pipes to
communicate between these programs. And of course, you will have to have access to files
and networks somehow, right, especially in isolated contexts. And there what you do is you pass
file descriptors over these pipes so that one process opens resources and the other
less privileged one is using them. So these are the these are the patterns we use.
Okay, and so basically, this is what this is. It's a Git server that runs as this kind of
multi process program. It only supports SSH. Git user account I mapped to regular shell accounts
because I didn't want to reimplement user management. You can have a special purpose
login shell for these users to restrict them, if you want. And access permissions are said
per repository. I don't want to go very complicated and make it like per branch or something. It's
just like, no, if you have access to the repo, you have access, which is good enough, for example,
for OpenBSD's model where you get an account and you can commit anywhere.
And when you configure this thing, this is basically what you need to do. You create your
repositories, make sure they're owned by the right user that you run the demon as. And you have at
least one repository in your configuration file, which has a path, but the repository is and access
permissions for either, in this case, the example would be a group of developers, which you have
in ETC groups and an anonymous user, which we can only read. Now, my initial implementation
of this looked something like this. It was functional and I could write a test suite for
it, which was the main part. This could actually be used to fetch and push changes. But the design
wasn't very good in terms of this multiprocess aspect because the parent started, then it
started a reader process and a writer process and that was it. And then all these processes were
always used for every connection. It did allow us to at least get this up and running, though.
And I don't know, I asked for a bit of review and got shocked responses to say like, no,
you're doing this all wrong. Fork and X needs to be done per request and so on. So yeah, okay.
But at least functionally, it was already quite okay. And the repository code there is
reusing a lot of the code that I already had for like God admin and so on. So I mostly had to
rewrite a lot of code for the parent process from scratch, which was all of this. This is
what it looks now. So the parent basically encompasses or used to encompass all of this
functionality and we'll go through each one by one. So right now, in this current implementation,
you have the parent when it starts up, must start as root in order to be able to do certain things
like open, like start the listener process as root, for example. And it uses pledge as
standard IO proc exact, which means basically standard is you always want that it's like
printf and stuff like this. Then you have proc and exact, which allows you to fork and execute
programs. And you can also send and receive file descriptors. And that's it what it can do.
It also currently does an unveil on itself. So with an X permission, so it can re execute itself
with different option flags to start other versions of itself, basically that we will start later.
I'm not sure if this is really sound because it used to be said that unveil would inherit to child
processes. And I'm not sure what happened to this. Currently, it does not. So it does not inherit,
so I can do this and not lose access to, for example, the slash temp directory in the processes I'm
starting next. But if that ever changes, we would have to adapt this, but it's not a big deal.
You start a listen process, which opens the actual Unix socket that this demon accepts connections on.
So basically, if you're a local user on the system, you can always access it through the socket,
but you would normally run this shell that we have to, which does this for you and speaks the
appropriate protocol. It then drops privileges. And the listen process runs as just standard IO
synaptic Unix. Unix is needed to operate on the Unix socket. It also does an unveil because
the Unix pledge allows you to bind other sockets and bind would create other sockets for you
somewhere. And we wanted to prevent that. So by unveiling everything, basically hiding everything
with unveil, there's no way to create additional Unix sockets for this process. And this process
is also, as an initial kind of dust prevention mechanism, this enforces a connection limit
per UID so that not one user can just connect to the socket and spam it and prevent access for
everyone else. Now, the shell is one of the most sensitive parts because this is where users log
in and you actually confine them to this program. So you want this to be reasonably secure. It starts
out with standard IO, receive FDN Unix to be able to connect to the Unix socket. But once it's
connected, it drops that capability so it can no longer open new ones or do other things related
to that. It only has a file of scripture it can talk on. And that's it. And then it starts
demonstrating these packet lines that we saw to messages that are internal to the program
and go over the pipe to the parent. The parent will then start an authorization process which
only runs once. And what this does is it gives itself access to the password database of the
system using the SCAT-PW syscall and also hides all the file system. And I think this is, this
shows something very nice about Pledge and Unveil when used in combination because I'm actually
reading ETC password and ETC group files, right? But Unveil, as per Unveil, I shouldn't be able to
access those. But because I declared that I want to use the password database, the kernel knows
that this process is okay. It's okay for this process to access those files. So it bypasses
Unveil in that specific case. Which means I don't have to worry about how the security mechanism
is implemented. I don't have to go and say, oh, is my libc when I ask for users going to open this
file? Well, maybe I should add an exception for that. Or is it going to do this and such and such
syscall? I don't have to worry. I just say like, Pledge, I will do that. And Unveil, I will do
that. And they take care of it, which is great for a programmer. It's really nice to program
against this. So what this process then does, of course, is matches the users that are logged in
against the access rules in the config file you saw earlier and reports the result to the parent
and just exits because that's all it needs to do. It's just a one-shot thing. Now, the parent
starts two processes if authorization has succeeded. And the shell is kind of waiting
because it's like, hey, I sent a message, but you haven't responded yet. But yeah, we're busy,
we're setting up. So we start two things right now, a session process and a repository read or
write process. Currently, the naming of these is horribly bad. It just was the best I could come
up with. And it kind of grew organically from the initial setup with those three processes you saw
earlier. But for example, the repository write process is not actually writing to the repository,
which you'll see later. So I'm not very happy about this. And also, the session process is
basically the most powerful component of the system right now. It's the only one that can
actually read, write, the repository and create files in there. It can also do the same as slash
temp. And for that, it needs all these pledges with like read path, write path, create path.
And it also needs file attributes and file locking because when it changes references for
clients, it needs to make sure that they get locked so that you don't have file system races
where two clients commit at the same time and then you end up with a reference that's been
overwritten. It also creates temporary files, which the repository process needs and gives it
the file descriptors. It handles installing of the pack files and so on. And it has the
git protocol state machine in it. So that's a bit, I would like to continue work there to split
this up more, but because I had to have a functional implementation and I had to, like,
I wanted to have something functional to clone from, which is there now, which is on the internet.
That's fine. But going forward, this needs to be revisited for sure.
The repository read and write process is apart from the name for repo write. I'm okay with
how that's worked out. Both of them can only read from the repository. And what the reader does is
it is responsible for creating a pack file and streaming the result to the God shell over
a pipe that is created by the session process and handed to both the shell and the reader.
And the writer is responsible for receiving a pack file and indexing it. So the indexing
is almost done. So the indexing is done there. Okay. I have one minute left, one minute and
a half. I quickly go through some implementation improvements. It's still like to do. So we should
verify what the client has uploaded. Currently we trust it, what to do. The config file is
parsed every time a process starts, which isn't ideal, which works, but it's bad if you're changing
the file while the process is running. Yeah, session I already mentioned. And the state machines
have some funny bugs. So these really need to be rewritten. They're basically like switch statements
and if and so on. And I'd like to properly separate that out with tables and state transition
functions and so on. But it was just a quick way of getting things working. But we already saw
like thousands of flash packets flying through this process because an end of file on a socket
triggered a flash packet and that was kind of stupid. This has been fixed, but there will still
be other bugs like that. We should have some built-in checks so that commits can be verified
according to project policies and things like denying merge commits if you don't want them
or binary files and so on, preventing a forced push. I'd like to have commit notifications
where you, for example, send an email or you can send an arbitrary HTTP request so that
if you really want to have a post-commit hook script, you run it somewhere else and we'll
give you information and trigger it. Yeah, also it should really keep track of what this space
it has when it accepts pack files and not fill the disk and fail. We should be able to remove
redundant pack files that have accumulated over time. I'd like to add SHA2 support
and enable it by default once that works so that we use the SHA2 because we have zero
production deployments right now and unlike it, so we can just use the new format they've already
defined. Service at rebasing is another thing. I'm out of time, so I'm not going to go into that,
but I think this is it. Sorry for the quick part of you. Thank you very much. I
encourage you to ask a question about the hallway. Okay, good.