Next up, we have two speakers for the prize of one.
They are going to talk about everything to do with an open, open, more open source version
of Talescale.
So let's give an applause to Christopher H. Gouan.
Hello.
Okay.
This is cool.
Hello.
My name is Christopher, and I'm going to, together with H. Gouan there, talk a bit about
how we use integration testing to kind of reimplement the control panel or the control
server of Talescale.
So first a little bit about us.
Juan Fontalonso is the creator of Talescale.
He works for the European Space Agency on cloud and DevOps and infrastructure.
He claims to have been my first manager, but I think that's incorrect.
And he has the attention span of a goldfish.
Which makes the whole collaboration very fun, and I'm here with Christopher.
He's a top contributor of Talescale and one of the other maintainers alongside me.
He's part of the technical staff at Talescale, and part of his time at Talescale is to work
improving Talescale.
I was his manager, at least from a hierarchical point of view.
And one of the challenges we have is that he always finds these kind of super niche
languages, like OCaml or things like that, where he tries to reimplement headscaling.
But first of all, how many people here know Talescale and headscale?
Good.
That's pretty good.
So for the people who don't know, we'll do like a quick tweak what is Talescale.
So Talescale tries to solve this problem, where you basically sit and you want to connect
your organization or home or something like this, and you have an old school or legacy
VPN concentrator, where you connect into your kind of perimeter, you have access to absolutely
everything, there's a single point of failure and a massive bottleneck.
And it tries to do this by creating like a mesh VPN that uses direct connections wire
guard and kind of facilitates this for you using techniques like natural and has a very,
very powerful client that will make sure that you always reach what you're trying to get
to, and it offers a lot of different kind of granular access, and you get a lot more
power compared to your old school bottleneck, single point of failure VPN.
And in Talescale, the clients are open source, at least for the open platforms, and what
they have is a closed SAS.
But still, they are quite open when it comes to explaining how the whole thing works.
And in March 2020, they publish a blog post basically explaining how the whole thing worked,
how they use these natural techniques so you don't have to open the ports in your router.
And there was a phrase in this blog post that gathered my attention for a little bit, and
was basically talking about a coordination server, that the clients talk to a coordination
server, the core of this services service offering, which is essentially a shared drop
box for this wire guard public keys.
So I was pass up by that, and basically I took that open source clients and started reverse
engineering, basically a lot of print apps to see what kind of payload were they sending,
what kind of endpoints or protocol they were doing.
And yeah, this was around April 2020, in June, I had a lot of free time at that time, and
in June I did the initial release.
I talked to my friend Christopher about tail scale, and he was very happy distributing
wire guard keys with Thansible, which, yeah, so I kept doing my own thing for a while.
Head scale gained a little bit of traction, and around mid 2021, he joined because he
was quite curious about the whole thing.
But he was afraid about breaking stuff, and that's why kind of we are here, although
he was not afraid of making a logo, but I think it's super nice.
So what I've learned doing this reverse engineering exercise is that the tail scale clients talk
to what is basically a web service.
This web service receives metadata from the clients, like the endpoints or the wire guard
public keys that they use, and assigns them IP addresses, like you would having a classic
traditional VPN service.
As everybody knows about everything, you can establish this mesh network across the clients
without interference because the data doesn't go through the web service.
So we arrived to the initial stage of head scale, the illusion that everything works
and kind of worked until it stopped doing.
So we had this web service, we implemented the web service, the series of endpoints
that we found in the reverse engineering exercise, and we were assigning IP address
to the when a node arrives, and what happens when a second node arrives?
Hey, we want to tell that I'm here and I want to find my friends and I want to communicate
with them.
So in order to handle that, and to handle the metadata that you need to establish the
connections, we developed a little bit of a state machine that will handle, and you know
has arrived, there's been a change in the map of the network, and we need to distribute
the updated metadata that we have.
However, at that time, I was kind of learning go, and we follow a little bit of a weird
approach when handling concurrency, which was basically adding more Mutex every time
we needed it.
And this is a problem, because at the end we ended up with a great Mutex for this state
machine, and this is a very big problem because the Python track is tomorrow, so the grid
logs are over there.
So what ended up happening inside the state machine, or what didn't end up happening,
was that basically some of the failure modes we saw was that a new node trying to register,
and then we burned a couple of CPU cycles trying to calculate some stuff, and then we
did nothing.
So no updates were sent out or anything.
Sometimes we would have a new node joining, and we would compute everything, send some
network traffic, we just omitted the new information, that was kind of crucial for everyone to know,
so it ended up not working.
And sometimes a new node joined, nothing really happened, but then eventually something
happens and it sent out an update to everyone, and that was, you know, useful.
And sometimes on the individual update channels for each node, some of these aforementioned
Mutexes kind of deadlocked up the whole thread or the GoRoutine, and then we just never sent
updates to particular nodes, and sometimes we just deadlocked the whole server and you
kind of had to kick it to make it come back to life.
But still there was kind of this notion that it did work pretty well eventually most of
the time, and it gave this illusion of working, because what you often saw was that you had
like three nodes, and only two of them actually talked together, and as long as those had
received the updates they needed, you know, the user was happy and you're just like, ah,
it works, so I'm going to press the star sign on GitHub and share it with my friends.
So, but we figured that eventually this would like caught up with us and we're trying to
get to this stage where we, you know, it works most of the time, so what we did have was a
fair amount of unit tests, but the problem with unit tests is that we're trying to reverse
engineering something, that we're also learning how it works, and what we spent a lot of time
on was misunderstanding how it was supposed to work, writing wrong, well, writing unit
tests that would pass, but they were wrong, so you kind of have like a, a passing test
and it's an entirely wrong implementation, and 90% of what we were actually trying to
do was integrate with a third party software, and this is where we get to actual integration
tests, so what I started doing was I found this Docker test framework, which basically
allows you like programmatically create Docker containers, so we started making tests that
spun up a head scale container, it created a bunch of tail scale instances also running
in Docker, and associated them with a couple of users and tried to like emulate the entire
environment so you can test everyone to everyone.
We had them join the head scale server, and since it takes a little bit of time for everyone
to catch up with each other and, you know, send the updates and stuff, so we put a sleep
of two minutes in front of the test, which is a terrible idea, but, you know, you learn,
and then after that sleep runs out, presumably everyone is now up to date and can talk to
each other, so we had a test, the most basic test is, is my network working, can everyone
ping everyone?
So we tried to do that, and of course that didn't work because of all of the errors we
actually had in the code, and I ran some initial like, tried to make some statistics on my
laptop and out of like 100 test runs we had 70 failures, that's pretty bad, but at this
point we're starting to approach like, we have an actual goal that we can measure so
we can improve on this, and quite rapidly we figured out that these two big blocks of
problems that we have is associated with two things, so one of them is the being able to
reliably send updates to all of our clients, which is the kind of deadlock problem that
the update channels were just locking up and didn't really work, so we made a massive,
massive rewrite PR that re-did the whole logic and made sure that we always were able
to send an update to the client as long as it was connected, and then the other problem
was the state machine that was very broken, and then we kind of figured out that we can
make a global state, and we tried to simplify it initially and optimize later, so basically
a global state, how can we determine if everyone is up to date, and make sure that we know
when you last received the successful update, and if not we have to re-issue one to make
sure that you know everything.
However, changing the Rambo culture takes a little bit of time.
We kept merging staff without proper integration testing, but as Christopher said, we didn't
have the incentive, we didn't have the pressure because the thing really worked.
It's not the same when you are in your home lab and you join a node than when you are
joining 100 nodes within one second, so if you are slowly joining machines to your tail
net, things were working.
However, the project was gaining popularity, and we were increasing more and more in contributions
in external PRs, and this was around August 2021, or September, something like that.
So it was great, we were getting to a point where we could improve headscales with confidence.
We had a point of view, given that the project started as a reverse engineering effort, we
had a lot of staff that was not that great, we could improve or maintain the compatibility
with these third-party external clients that we are using, and we could improve from a
community point of view, I'm going to talk a little bit about this now.
For starters, we could improve from a technical point of view, we could do massive refactoring
within the project, or implementation of the second version of the headscale protocol
without breaking the existing users, the only thing that breaks is probably the mental health
of the reviewer that has to deal with 3,000 lines of code.
But that's a different thing.
Then as I said, we have this minor small detail that we completely depend on a third-party
client, because we are using exactly the same official clients as a stale scale, however,
we have a very good working relationship with them, and every time that they change something,
we get a heads up.
However, we keep within our integration tests quite a bit of commitment for support in this
client.
So we target the head of the repository, we target the unstable releases, and we target
nine minor releases of the client to make sure that nothing breaks from their side or from
ours, because I mean, it can happen.
And then I think integration testing also helps the community, because we as maintainers
can trust in a better way those random PRs from random unknown people that appear in
GitHub, which is something that is not given.
And in theory, or that's what one would think, is that by having integration tests, contributors,
those external people that we don't know, should also feel more confident when submitting
a PR.
But that's a theory.
So it does still come with some challenges.
So one of the things that we see occasionally is that a PR comes in and it doesn't have
a test, and then we ask nicely if they can add a test, and then the contributor disappears.
So some of the times we're trying to improve on this thing and kind of like always get
them in.
So what we try to do is, if they truly disappear, we try to pick it up if it's a feature that
we really want and we are bound with to do so.
Once we try to reach out and kind of sit and help them write a test and kind of onboard
them in this kind of things, one of the tests actually for our, there is an SSH feature.
And the test for that, I knew the developer and he was also in Norway, so once I was
dropping by Oslo, we sat down for an afternoon and we worked on them together and paired
on them.
That's not available for everyone, sadly.
But you know, we always try to kind of like get this test message out there in a way.
But there is a couple of other challenges as well, and that is that adding the test
raises some sort of learning curve.
So you know, you need to know go test, you need to understand our test framework, you
need to have Docker and all of this kind of thing, whereas it's not writing tests that
are a lot less code.
And it's hard to convince people how awesome tests actually really are, that they're not
really a chore and that you really, really thank yourself later for doing them.
So some of the things we're trying to do to even make this barrier lower, since we're
so heavily dependent on this for compatibility and everything, is that we're making like
our own test framework, V2, because we depended on a lot of repeated and copied code and there
was a really high bar for adding new tests and it was really hard to update and change
and it did depend on time.sleep, which was, yeah, haunted me so many times and it couldn't
really be ran in parallel for many of the previous reasons and the documentation wasn't
really good, like I knew how to use it, one knew how to use it and then that was about
it.
So a couple of other people figured it out.
So what we're trying to do is we're abstracting things a bit away, so we have this concept
called control server, which is what essentially head scale is and the tail scale product,
the software as a service and it's implemented as like head scale in container and it exposes
convenient functions that now have Godox support and all of these things to make it easier
for developers to actually use it and then we have the tail scale client, which is implemented
at tail scale in container and it has the same type of convenience functions and what
this allows us to do is previously the two files on the right here, sorry, on the left,
is two different versions of the setup code for the tests because when you needed something
that was slightly special, you had to copy the whole thing and then make a new file to
be able to write a test case like you see on the other side here, but now after abstracting
that away, making it a lot more configurable, we allow people to write more or less regular
test cases, but you just set up what we call a scenario, which is a head scale with a given
amount of tail scale nodes and then you let them ping each other or something like this.
So what do we test right now?
We tried to, we kept all of the original tests, so basically we make all nodes, join the network
and we make them ping each other to verify that we have a fully functioning network both
by IP and magic DNS, magic DNS is tail scales DNS system.
We test tail drop, which is a file sharing features, a bit like Apple's airdrop and we
send the files from every node to every node to make sure that they work.
We test all our registration flows because we broken them a couple of times, so it was
better to do it that way, which is pre-authored keys and web plus a command line flow and
even open ID we currently have tests for.
We try to isolate all of our network from the internet and test with our own embedded
relay server because tail scale depends on some relay servers that we also embed in
our binary and we have preliminary tests for the SSH features that we support, which is
like authenticated by head scale so you can SSH into your machine and we test SSH all
to all and we try to do negative tests.
And also we test our CLI because if you may change something, you don't want to sit and
type in every single command in a structured way manually because that's just painful.
So in the future, we want to also kind of improve this granular access control that
tail scale offer.
Currently this is a very good example of where we have added a lot of unit tests and they
all pass, but they're all wrong, so well, they're mostly wrong, so we have to kind of
redo most of this into integration test first and then kind of backfill the unit test once
we know how the implementation is actually supposed to work.
And one of the things we've been dabbling with, especially for this ACL feature, is
to use that control server abstraction we have before and use the tail scale product
to test our tests because if they pass on the public server, we know they're correct
and then we can use them to verify our thing.
And then maybe run tail scale in the VM instead of Docker to test it properly, but that's
more of a benefit for tail scale than it is for us.
So if you're just here waiting for the next talk, a little bit of a TLDR is that, I mean,
we cannot understate how important having this integration testing when we depend on
an external party has been for the development of health scale.
I reckon also the head, like the name, is also excellent, ponytail scale would have
been worse.
We have, I mean, with integration testing, we are able to maintain this compatibility
with the client and we are able to take contributions from third party developers, otherwise it's
a little bit more difficult to develop this trust across the internet, right?
And even though the tests are not perfect and we still have to migrate unit tests towards
integration tests, I think this is one of the keys for the success of the project.
So some extra things, tail scale is hosting a happy hour at a brewdog by the station.
This QR code takes you to a sign up form, I'll quickly switch back to this slide at
the end, but I have like a question slide as well, so, you know, we go through this.
Basically this is how to reach us, Github, we have a Discord community, and we're very
happy to talk to anyone who wants to talk to us here at Fostem, so please feel free
to reach out and I'll leave it at this one if anyone has any questions.
We have some minutes, I think.
Thank you.
While I have your attention, we have a go for that lost there wallet, look to the left,
look to the right, front and back, if you see a wallet that is not yours, please come
right to the front, it will help this person a lot.
Thank you.
After you look for the wallet and you have a question, raise your hand and I'll try
to come with this microphone.
How come the tail scale guys are not mad at you, and not only are not mad at you, but
they hurt you afterwards.
I mean, part of it, is it working?
Yeah.
No?
Okay.
I think part of it is that they are quite chill, I mean, they could have, they are quite
chill, they could have taken this way worse than they have, and I don't think we are competition.
We are focused on self-hostors, on home labs, perhaps a little bit of a small company.
And what usually happens is that people that use headscales at home, then they go to their
companies and they talk about tail scale, and when you're in a company, you actually
prefer to pay for the service.
So it's like a way of...
It's like a way of selling headscales, sorry, headscales also.
Okay, thank you very much.
Last round of applause.
If you have any questions, you can card them in the hallway track.