So, my name is Robin Geuze. I used to work for TransIP at first and Team Blue after they
merged with a bunch of other companies for about a decade until a month ago. During that
time period we transitioned from running our own closed source DNS server software to running
open source DNS server software and just like the talk we just had, that happens to be power
DNS. So, I'll take you through the issues we had going from closed source to open source,
which roughly took the entire time I was there, about nine years. So, yeah, let's start. So,
how it started for me. TransDNS, which they called the home root DNS software, was written originally
in about 2003, 2004 and it had the DNS support added in 2012. When I started working at TransIP in
2013 as a PHP coder, I was asked to help them debug a crasher in the TransDNS code. It basically
came down to a buffer overflow because somebody had, one of our customers had managed to put more
than 16 kilobytes of text record data on one single label. The really quickly quick fix was to
increase the buffer to 32 kilobytes. And one small disclaimer, I was involved in almost all the work
that I mentioned here, but there are some things that I didn't do myself or just consulted on,
stuff like that. I'll try to make a distinction about it, but I might miss some stuff. Yeah,
so back then it was a really basic setup. We basically had three servers. They were all running
TransDNS. There was no load balancing. The signing stack was built using DNS stack tools for those
few people who still know what it is. And there was a lot of automation on top of DNS stack tools
in PHP to make all of that work and ultimately upload stuff to the registry because we were one
of the, we were a registrar, so a lot of the stuff was automated. All of this DNS propagation was
done to cron jobs, which means it was very slow. It took roughly five minutes to propagate a DNS
change, which back then wasn't really a big problem. But as we went on, it became more and more an
issue, especially when we got let's encrypts and you needed to quickly update your DNS to get your
certificate signed. We had at the high, I think we still have roughly one million zones in the
setup, most of which so about 80, 90% are DNS signed. There were very few people back then that
actually knew stuff about it and dared to work on it. I think maybe three or four people, one of
which was I. It had very bad RSC compatibility, which I will get into a little bit later. Adding
new record types, which Kevin mentioned like SSHFP was a lot of work because there was a
interpreter in TransDNS itself, which had to be written in C and writing interpreters in C for
stack strings is not fun. And well, I fixed that initial buffer overflow block, but the main
problem was there just not a lot of bound checking in the code. So yeah, there were a lot of hidden
bugs that probably should be fixed as well. So we took a few initial steps because initially,
because we had the three servers, there was no loan financing, we meant that if we restarted
TransDNS, one of the servers would stop responding until the restart was done. And the restart took
roughly 15 minutes because every single record would get loaded into memory. And since we had a
million zones, I think it was like 25 million records or something back then, it just took a
lot of time and might have used the quick DNS zone parser stuff. So the first thing we did was
implement load balancing. This was before DNS. This was a thing. So what we tried initially was
relay day, which some of the BSD folks might know. It did work, but we had a lot of weird issues.
It was really hard to debug. And so eventually we switched to using HAProxy for TCP, which works,
nothing more to say about it. And I wrote something rather quickly in C roughly based on the
TransDNS code to forward the UDP stuff. That worked quite well and actually enabled us to
actually iterate on the TransDNS code because we could do save restarts without having to worry
about queries being dropped. And that allowed me to fix the glaring issues like there not being
any bounce checking in the code. So we had less risk of buffer overflows. And I fixed a lot of
the EDS issues that were becoming a problem at that point. Eventually when DNS was a little bit
more mature, we switched to that because otherwise I had to maintain another piece of software and
I really didn't feel like that. In the meantime, it did improve the TCP stack a lot in TransDNS
because we noticed that especially SIDN, the.nlRegistrar registry, did a lot of TCP queries
and the original implementation was basically just spawn a new thread for every TCP connection,
but once you get to about a thousand threads, that's not a great solution. So I changed to a
polling-based model, worked great, got pretty high performance, and we never had a problem with it
after that. The only thing I changed later is when we moved to Linux, I changed to ePool.
Yeah, so SIDN had validation monitoring and we kept getting reminded about the fact that we were
doing a lot of stuff wrong. So yeah, we actually had one specific case that basically covered most
of the, I think it was about 80% of those errors, and that's, it's 62 issues, but they have the
same cause. So the first issue was the incorrect handling of wildcards. So if you have a wildcard
that, for example, star.nl, then you have a record c.nl, and then you try to resolve a.c.nl,
it should not hit the wildcard, because c.nl exists, which means you should return a no-data,
or an extra main in this case, but transdns didn't really care, so it would just return the data
from the wildcard. Very useful, makes it a lot easier to configure DNS, but it causes some issues,
especially with DNS validation. The second issue was basically the same only in the
empty non-terminals. If a.b.c exists, and you try to resolve b.c, even though there's nothing
specific on b.c, you should say there's no data, rather than it's a non-existent domain,
also causes the DNS validation errors. Same basic cause. The solution was to switch from,
in transdns, to switch from an ordered map that used the type and the domain name as the key,
to a map that only used the domain name as the key, and have an array in there with the type,
which could also be empty, so we would immediately notice if there was a label in our way. That
worked well. I actually did it this next slide, so the only problem is we couldn't just deploy
that, because we might break stuff for our customers, and customers get a little bit difficult
if you break stuff for them. So what I decided was, okay, for the NSIC it's broken anyways,
because the NSIC enables for the resolvers would just return errors when you have one of these
labels. So what I did is fixing the two steps. I initially enabled it only for the NSIC queries,
so the correct behavior, and kept the wrong behavior for non-DNSIC queries, and in between we
just covered a large amount of queries. I think I did two days of DCP dumping, and milling it down
to the actual unique queries, and compared what our name servers would respond for DNSIC versus
non-DNSIC. For everything that had a difference, we contacted the customers, and told them, hey,
you need to fix this. I think it was only about 20 to 30 customers. It was actually not that many,
so that made it a lot easier. And then we just, at some point I decided I'll flip the switch.
There were a few customers that didn't respond, but at some point you just have to decide to.
Don't give a fuck. One other small issue we have with RFC implementation was the NSIC
implementation, because almost all of our zones use NSIC tree. The NSIC implementation was not
as well tested as the NSIC tree implementation, so it was wrong, like really wrong. I just rewrote
it from scratch, and then it worked, but yeah. So we started to think about moving to PowerDNS,
and the main reason we did was because SIDN announced that we would no longer get a DNSIC
incentive for domains using the NSIC algorithm 7. So that's the RSA plus NSIC tree algorithm.
That would cost us a bunch of money, and that's a very good way to stimulate people to do stuff.
So at this point we decided to buy the bullet and just start over from scratch,
and build a really new, more modern setup. We picked PowerDNS, basically,
partially because we already had some experience with it, and we didn't really want to deal with
zone files, because we had a million zones, and putting them all on a file system makes things
annoying. So PowerDNS was the only one where we thought, oh, this allows us to do changes via
the API. We don't need to worry about having separate zone files for every single zone.
So we needed to pick a PowerDNS backend to use, because PowerDNS is one thing,
but you still need something to put stuff in. And there we sort of had to hit a problem,
because PowerDNS is really fast, because it's literally just a hash map in memory,
so it can basically do instant answers. And while the PowerDNS, as you go back in,
is very nice and flexible, but it's not really fast, especially because we had a lot of zones
that would not get very frequent queries. So they'd have a lot of non-active data,
which means the query cache wouldn't really help a lot, which means that we would have
a lot of SQL queries continuously, because they would get queries sometimes. It's not a lot.
The bind backend had the same problem as all the other name servers. We didn't have API support,
and it would mean we needed to use a lot of zone files, which we didn't want to.
So introducing the LMDB backend. This already exists at the point that we started looking at it,
because Hubert had written it. It's very fast, and it has support for the API, which is really nice.
It only had one major issue. Because of the way Hubert had implemented it,
it didn't really allow records bigger than 512 bytes. We have quite a lot of zones.
So I decided to fix that in the end. I wrote a pull request for the Power Genius team,
and I think that was pretty quickly accepted into there. It also included some migration code,
so the older the LMDB database would automatically be migrated to the new LMDB database format.
It also improved performance in some corner cases, but that was not really the goal of this patch.
So then we started moving over. We built a setup. It was really cool. There's a lot of automation
around it. It does actually do all the zone transfers via XFR, even though Kevin just said
it's a bad idea if you have a lot of zones. But in practice, it works quite well, except for one
issue. Every first day, our updates would take ages to go through. Basically, we traced it down to an
enormous bump in the XFR queues. We would literally have 400,000 XFR queued up. So that was a bit
of a problem. So the reason this happens is because Power Genius renews its signatures every
first day of the week. Very nice. We don't have to think about it. Problem is, if you have a million
zones, that takes quite a while, especially because we were running our hidden primary on a VM,
so it was also not that quick to answer queries. So we could have just shown more hardware in it,
but we decided to look a little bit more at a more sustainable solution, because, well,
if it works with one million zones on the Phosomers scene, it will still work if you have 10
million zones. So I discussed it with the Power Genius guys, and I came up with a solution which
is XFR priority levels. So rather than treating all XFRs that need to be done at the same level,
we gave more priority to things that are user-initiated. So if you initiate an XFR
via Power Genius control, it will be first in the queue. Whatever else is in the queue,
that one was treated first. After that, there's the API, notifies, solar refresh, and signature
refresh is the lowest priority. That meant that, yes, we would still have a quite a large queue,
but we could still process our updates very quickly. That was included into Power Genius,
right with us. Well, in 4.5, it was included. We still saw the use queues, but those own
updates would pretty quickly propagate. So that, for us, was fine. We never had a problem with it
after that. Yeah, and then we had some other issues, most of which were in a minor solved in
the low-banus layer or just fixed in Power Genius updates. The TCP performance is still
something I want to look at in Power Genius just for fun, as a open-source developer. It's on my
list of things I want to improve. We had some various smaller bugs in the NMDB backend because
it was quite new. We were not the first one that ran it at really large scale, but we were one of
the first ones, and we did see some problems that nobody else had had yet. One CVE we discovered
literally within the day of rolling out a new version, so that was very fun for Peter because
he got to roll out a new lose a day after he released the previous one. We had an issue that
there were certain query patterns that we would get that were specifically designed to target
a weakness in Power Genius. That was a transient as we didn't care about them, but Power Genius did
get affected. We eventually resolved this by adding some detection at the low-banus layer that would
just block queries for those affected domains. It would mean that that customer's domain would
have limited functionality, but at least it would still work, and all the other customers would not
be affected, which was for us the most important thing. Yeah, so some closing thoughts. Yeah,
migrating a home root setup is really not for the faint of heart. However, running one is also
not for the faint of heart. Yeah, it is worth it. It just gives you a lot more flexibility because
now adding new record types is just a question of adding them in our front end and making them
work. Whereas before, we had to add the new record type at every single step in the stack,
and it just really took a lot of time. We can even, in theory, add different brands of
secondaries. Currently, there's a few issues that were prevented, but it's relatively easy to solve,
so we could just run not as a secondary or NSD or even bind if he would want to do that for
some reason. What I did really, really notice is don't try to do this in one go, because it's a lot
of work, and you'll make mistakes. If you do it in smaller steps, the mistakes will be smaller,
easier to fix, and it also just feels a lot better if you can accomplish some things in
between rather than trying to do it all at once. One thing I wanted to ask, DNSSEC incentives,
they work both when trying to get people to use DNSSEC, but they also work to improve the
quality of the DNSSEC, because we've seen, especially in the.nl zone, because I've also
been involved in that work a little bit, some very bad implementations that got fixed when the
rules were made stricter, including ours initially, but were even the worst ones. Yeah, that is it.
For the people that would like to see, I've open sourced DNS before I left the company,
so I can see it myself as well, so that's fun. It's on GitHub. I've also put the URLs for
the two major pull requests I made. There's a bunch of other ones, but I haven't put all of them
in there, and that's about it. So, questions?
So, that makes it a bit of a more concern in your case, and I used to be a customer. On the upside,
most of these methods probably weren't noticed by the majority, but I think you should take it
more seriously if you were a company that actually makes money out of posting here.
So, the comment is that we both made mistakes. It was a bit related to the talk Kevin did,
so a lot of the things that we said are related, and the comment is Kevin was only doing it for
himself, and we were doing it for paying customers. Yeah, I agree. When I started, there were a lot
of issues, and I've tried three years to attempt to fix them as much as I could. To be clear,
I wasn't hired to maintain trans-DNS. That just happened to be something that got shoved
into my lab because I knew some C and C++. I became pretty passionate about it.
I came pretty quickly rolled into the PowerDNS community. I also added a lot of contributions
to DNS when that was getting started up. I agree with the initial statement. I've tried to fix it
as much as I could. Sometimes, you set out with certain criteria. You build something that can meet that criteria, and it scales to a certain point.
Eventually, you get to a million customers, a lot of customers. The company would start off with a million customers, and maybe at the time that this was a good system, things would fly to me. But as the business grew and things grow,
you have to do exactly what he did. You evaluate it and say, you know, it's time for something different. He identified that and made the changes accordingly.
A brief resume, he said that sometimes due to scaling, you run into issues that you hadn't foreseen when you were in it, and he set something up. Just taking a step to resolve them in the end can be a good thing.
There was a question there. The question is how did they get them to agree with over sourcing it. At the point that I open sourced it, I was sort of CTO slash head of R&D of the Dutch part of the organization.
Also, I only open sourced it after we totally took it out of production, so it's mainly a historic interest thing.
Did you ever consider open sourcing trans-DNS before switching?
So the question is, did we consider open sourcing before switching? No. And I'll tell you, we weren't very proud of, at least I wasn't proud of the source quality.
I didn't write it myself, all of it. I only contributed to it later. I tried to improve it as much as I could, but it's still not...
It's very focused on just doing one thing, and it's very good at that, but it's not very applicable to use by others.
So I think it's interesting now to see some of the tricks to make things really fast that you can see in the code.
But beyond that, I would never use it in a production environment other than the one it was in, because that one was built specifically to run around that code.
What's actually the motivation for implementing the DNS hosting and trans-DNS? I think even around the time when you started as a company, there was not a software available that could have been used.
So the question is, what was the motivation to implement their own DNS software? So to be clear, trans-DNS was implemented in 2003, so this was roughly when power-DNS started to grow, but the problem was that there were already quite a lot of zones in there,
and it just got a little bit cumbersome using bind, because that was the primary name-server software you'd use back then, and yeah, that was the main motivation.
Bind was getting annoying because you had to have a lot of zone files, and everything was running on 3BSD using UFS, so there was a 32,000 files per directory limit at that point, which also didn't help.
I mean, there's ways to solve that, that's not that complicated, but that was the main motivation as well, as I think there were some performance issues in bind back then that were relatively easily resolved.
The other alternatives would have been GGB DNS, but that had its own things, like the guy that wrote it, not saying you should use it.
Anything else?