I'm Wookie, and I have come to talk to you about the year 2038 problem.
I would quite like to get feedback and not talk all the time, which means I have slightly
too much information to give you in time available.
I will try and get this about right.
So just to make sure people understand what the problem is, time t on Unix is a 32-bit
signed int, which started in 1970 and counts in seconds, so it rolls over in January 2038,
at which point bad things will probably happen.
That is now less than 15 years away, which isn't very long when you consider the sorts
of systems that will still be running 32-bit code in that time.
There are a lot of other things which will also go wrong at various dates, but I'm not
talking about those today.
Now, I don't claim to be a particular expert in this area, but I have been taking a look
at it for the last few months, because Debian needs to do something, and we should work
out what it is.
So I just kind of try and relatively quickly cover the problem and the issues as far as
we know them.
I was hoping to have done a few more tests before I got here, so I could be a little bit
more explicit about things that definitely will or will not break, but we are where we
are and see what people think and also discover from you stuff that you know will break, because
I'd like to make a list.
So obviously, most people already use 64-bit computers, which have 64-bit time to, and
this problem doesn't arise for, I don't know, many centuries, but there's still areas where
cheap really matters, and those people still use 32-bit, so cars, TVs, kind of embedded
controllers for buildings and heat pumps and plant and all that stuff, and also cheap
phones.
Talking to people in the business, 32-bit Android is still a thing and will remain a
thing for quite a long time, so, and the problem is that the stuff that people are still doing
that with tends to get installed and left for a long time, so 15 years already isn't
very long, we're a bit late.
Quite a lot of the stuff that will break is already installed, you know, well, too late,
but there will probably be more, so we should probably fix it.
Most of that stuff will be kind of embedded distros like open embedded and Android, you
know, built for the thing, so the set of stuff using kind of Debian-style binary distros
that this applies to is fairly small, but we should probably still fix it.
Some 32-bit architectures don't care because they already started with 64-bit time to sensibly.
Big distros, I don't think care about 32-bit anymore, if anyone wants to contradict me
on that, that will be useful.
I'm pretty sure RedHut's given up already, I don't know about Fedora and how long they
tend to...
Okay, excellent.
Obviously, most of the 32-bit architectures are varying degrees of obsolete, you know,
but sort of still in use now, but probably definitely not by another 15 years time, or
at least stuff that will still be installed, but I think the one thing that does still
exist and is still being used fairly heavily is on V7 and Debian intends to carry on maintaining
that for as long as people are using it, so we kind of have to fix that, and this problem
is harder if you're a binary distro than if you're a source, rebuild everything every
time distro because how does the transition work?
So what has been done so far, Aunt, who unfortunately isn't here, he did quite a lot of work on
this starting in 2017 with Deeper fixing a lot of kernel stuff at the bottom of the
stack.
Turns out the Pearl people fixed this 12 years ago, although in a slightly odd way, of course,
it's Pearl.
Muzzle was fixed a couple of years ago, G-Libsus fixed last year, year before, and I think
quite a lot of stuff, and because some of those embedded people especially have been
building, you know, with new Muzzle, which forces 64-bit time to, quite a lot of stuff
has been fixed, but I do not have a good handle on how much stuff still is not fixed and will
just break if you build it with it, you know, it won't even build, never mind work, or stuff
which will build, but it will just explode your file formats or eat its old data or whatever.
So again, I would like feedback on that.
So quite a lot of people have done quite a lot of work, Aunt tried rebuilding Debian
in 2020, but everything was too new and too much broke and it didn't get very far.
So we've just done it again a couple of times last year, and, you know, just rebuilding
the base part, that's what I've done, and that seems to work, and doing an ABI analysis,
Steve Langer said he did a load of work in Ubuntu on how many libraries actually change,
which we'll get to in a moment.
So how does this work?
So G-Libsus 3.34, the way they've done it is to support the old 32-bit ABI and the
new 64-bit ABI, and each package, you know, you set a magic variable, are we doing 64-bit
or not?
So it's a per-build thing, and G-Libsus doesn't just change.
So normally an ABI transition, you know, you get the new library, it's different, you
build against it, you get the new thing.
So you don't have to do anything.
But with G-Libsus we were given the choice, it's kind of unhelpful really.
So something somewhere has to set the magic variable, otherwise you'll get the old thing.
They have some file format issues as well, the utump and co files, and apparently that's
fixed.
QEMU allegedly is still broken, but somebody has a plan.
So just to make this a bit more complicated, G-Libsus sets file offset bits to 64 if you
set time to 64, so you have to have large file systems if you're having 64-bit time.
So that's two different ABI changes, and that's been around a long time and people have been
fixing it for ages.
Again, I don't know how much software doesn't work if you turn large file systems on, doesn't
build, gets it wrong.
My impression is that most stuff has been fixed and is actually using that already.
But again, I'm not sure of the numbers.
GNU-Lib, if you use GNU-Lib in your build system, it will just turn on 64-bit support
if G-Libsus provides it.
So if you build with a modern G-Libsus, you'll get 64-bit time to, unless you specifically
set this magic variable you've never heard of to stop it doing that.
So some people are going to have been already bumped into 64-bit time when they weren't
necessarily expecting it.
And the last autocomp release was they decided that if you turned large file system on, you
should have 64-bit time to, either the opposite way around to G-Libsus, but effectively that
meant anybody who had ever put in large file system support any time in the last 15 years
suddenly got 64-bit time to if they re-auto-confed, which some of us thought was a bit radical,
so we told them not to do that.
But the thinking there was, it's time.
We should change.
Please will people get a move on?
So, you know, it wasn't completely crazy, but I think that was, that would surprise
too many people, I think, but we do need to work out how we're going to do this without
surprise.
Yeah, so people transition, but like when they're expected to, not because something
they hadn't noticed changed.
So the point here is if you use time t in any struct, especially in a public ABI for your
library particularly, or any file, then if the size changes, the ABI changes and your
file format changes, and that is the transition we have to deal with.
Now, you know, new ABIs is not uncommon.
It happens all the time.
Libraries change, add new stuff, adding new stuff, that's okay, but change stuff.
But this is a big one because it affects a lot of packages, but not as big as some we've
done in the past.
File formats I think is harder to quantify and work out quite how big a problem that
is or still is, and it's kind of up to each, it's mostly an application level thing, you
know, if your app stores files and it needs to be able to read its old files and its new
files without exploding.
Disk formats is a bit trickier, but again, mostly that's been dealt with in the kernel,
I believe.
So the fundamental question, especially for Debian, but kind of more generally ready for
everybody is, are we just going to update the existing architecture with this new feature
which changes the ABI, which is what everybody's done so far.
You just rebuild it with the new 64-bit time and stuff changes, but we still call it ARM
Linux, going to ABI HF, or whatever, if in I3-8.6 world equivalent.
That's not too bad if you rebuild from scratch every time.
It's quite a lot harder in the binary distro world where we expect to upgrade things as
we go along.
And it does change the ABI.
So in a sense, this is kind of wrong because that triplet defines an ABI.
But on the other hand, we do this all the time, LFS changes the ABI, and nobody said
we need a new triplet for that.
But from Debian's point of view, arguably a lot of people have said, well, this is too
big and scary and everything might break, so let's not do that.
Let's just start a new architecture with a new triplet, and then we'll know it's all
done, everything will correspond, and we're much less likely to get random breakage from
a bit of half and half time to 32-bit and 64-bit time.
But on the other hand, other stuff will break just because you've changed the name and the
triplet and stuff.
There's always some breakage in there.
The problem is that if we decide we need a new architecture and everybody else has done
it with the old triplet name, now the old triplet name means two different things.
It means the new ABI or the old ABI for in Debian ecosystem, and that doesn't seem like
a good place to be either.
So I don't know.
If we could all agree what we're doing amongst distro people who care, that will be really
helpful.
So as I said, glibc doesn't enforce 64-bit time t, so if just building against a new
glibc doesn't put you in 64-bit world, something somewhere has to set the magic variable, and
the question is, does d-package do that, or glibc do that, or gcc do that?
I mean, it seems to me glibc should probably do it, but at the moment that's not a thing.
We can work that out.
As I said, the new triplet is kind of easier.
We just start again.
It will give us the opportunity to call it ARM32 for the rest of the time, which would
be a much better name, but a very small bonus in comparison to how much work is involved.
So in a way, this transition is just like any other large transition, and we've done
it before.
Libc5 to Libc6 was massive, and that was like everything had to transition in one go.
Everything was smaller back then, which made it a bit easier, and it affected everybody,
so everybody had an incentive.
The problem here is that there's only really is an RMHF problem, but if we have a big transition
with lots of libraries, everybody has to wait whilst that piles up and then goes through,
and all the other architectures, proper architectures, will complain and go, will you people get
a move on?
You've bunged up the whole world.
We have done this before, back in 2007, apparently long doubles changed from 64 bits to 128 bits
on a whole load of relatively minor architectures nobody cares about.
Not now.
I mean, they probably cared a bit more, 13 years ago, and the world didn't explode.
I didn't even notice, so it kind of been that bad.
So I had a look at some numbers.
How big is this problem?
Is it completely intractable or not?
I think that's quite an important question.
So six and a half thousand packages out of our 35,000 have TimeT in them at all.
So that's how big it could possibly be, but an awful lot of that is not in public APIs
or file formats.
Again, I have really no handle on how big the file format part this is, but the API part,
we're getting a reasonable idea.
So in the bottom 150 packages that you make to just bootstrap a system, 85 of those are
libraries and seven of those actually change with TimeT, so that doesn't sound too bad.
Five language sets done a very useful bit of work on Ubuntu, up to all 767 library packages.
209 didn't analyze for tedious reasons, we can go about that, but of the 558 that did,
17% of them, i.e. 82, did change the API.
So that doesn't sound too crazy.
If we assume the same sort of fraction in the bit that's not yet analyzed, there's maybe
115 libraries or something like that, which would need to transition together.
So that's a lot, but it's probably doable.
I've just done some fairly random experiments having built both standard RMHF, RMHF with
large file systems and RMHF with TimeT and large file systems, and just tried putting
binaries in between them, and nothing obviously blew up, it didn't just immediately fail,
so it's not that bad, but I didn't have a good set of things you should actually check.
What are the tests I should run having installed some of this on the old system to see whether
it broke?
So I think we need a list like that to kind of get a handle on how bad that part of the
problem is.
So here's a few things that have been mentioned as potential problems, NFS version 3, apparently
the time there sometimes is signed or sometimes isn't depending on the client implementation,
so some of them won't break for another 150 years, some will, that will annoy some people.
I don't know how many NFS people haven't moved to V4 yet, probably some.
Apparently X3 is a problem, I don't know anything about that.
XFS was a problem, but I believe that's fixed in kernel 5.10 or something.
CPO was alleged to be a problem, but then somebody else told me that it has 11 octal
digits and therefore 33 bits, so we've got another 150 years before that breaks.
Does anybody know for a fact?
There's this whole room that should know stuff like this.
Only INN will break, but that's a relatively, that's a thing, it could be fixed.
32-bit wine on 32-bit systems with 64-bit time to allegedly my break, but then that's
only on I3H6, I don't think we care about that, so I'm probably going to ignore that.
So yeah, what else?
Again, we shall come to some questions in a moment.
I'm sure there's quite a lot of other things, but I don't know what they are.
So yeah, it's question time, I have done that reasonably quickly, excellent.
I would like feedback from people.
So this point about the, are we doing a new architecture or not?
Thinking so far in Debian has been kind of, this feels very big and intractable and we're
not sure how bad it is, so it's a lot safer to do a new thing.
But having done some research, the research I've seen suggests that maybe it's not so
enormous we can't do this, so I think an in-place transition probably is doable.
There will probably be some breakage, but alone you'll be in RHF, so it's like we could
just annoy those people for a while.
It might eat your files a lot, I think that's the thing that is really going to piss people
off.
If you install a new thing and half way through, it just corrupts files, they could be important.
That's not good.
So that's the driver for going, let's just do a new triplet.
But I don't think anybody else wants to do a new triplet because most people have sort
of fixed this with a standard transition within the system and a rebuild.
So I'm not going to say what I think just yet, but I have been developing an opinion
having done this research and I would love to get a list of things like, is Debus going
to break?
Does it have any times in it that's, it's like half your things are using a new time
and the other half are using old time and Debus has, or things like that, you know,
IPC mechanisms as opposed to ABIs.
I think the ABI problem we understand, it's the rest of it I'm a bit worried about.
Yeah.
I'm not quite sure how to, does anybody want to say anything?
Somebody wants to say something?
Have we got a roaming mic?
Yes.
Excellent.
Steve, yeah, see.
Hi, Wookie.
Hi.
So, of course, having discussed about this a bit in the past, this is most complicated
on RHF, as you said, it's probably not possible to necessarily change the ABI of I386, because
there are a whole load of old binaries that people will not be able to run anymore.
And the only reason you'd ever care about running I386 is for shitty old binaries.
It's less of an issue there for RHF, so that might be feasible.
Yeah, so we need good test suites for all of these things and for all, for good integration
tests, as you say, for either end of IPC and that kind of thing.
I have no idea on those.
I wish I did.
Okay.
I have the same concern about 32-bit wine, which is, people care about that for running
shitty old Windows binaries.
I mean, I care about that.
I have some shitty old Windows binaries, and that's what it's for.
I mean, somehow, Win32 has become this lingua franca of, I need an old program to run on
any machine, and now, like, if we look at what Valve is doing with Proton, they've just
decided, fine, Win32 is the standard for it.
So I realize there's a difference between Win32 and the architecture that it runs on,
but Wine32 specifically, as I understand it, 64-bit wine doesn't run 32-bit Windows
binaries.
Correct.
So the thing is, wine already does, ABI translation, that's what it does, right?
So it seems to me that if the underlying system has 64-bit time calls, I'm not sure it necessarily
can't work with the old binaries.
Oh, that's something I haven't thought about.
I may misunderstand exactly how this works.
So I don't know whether, in fact, that is fixable, or, as I say, whether anybody cares
enough about i386 World, just leave it alone.
I hadn't thought about the fact that that relies on the Windows time format, which isn't
faced with this problem.
Yeah.
So yeah, I'm not sure whether it's actually a big deal, just to me, someone would have
to do some work in some old code that everyone's left alone for a very long time, and nobody
can remember how it works.
I also wonder about RMV5, there's an awful lot of Raspberry Pi ones floating around the
world right now.
Yes.
It's a good point, actually, are there any Pi people in the room?
What have the Pi people thought about this problem?
I haven't talked to the Pi people yet, and he's quite right that it's a significant constituency.
I'm not from Vine, but I remember something about them improving their abstraction so
that you can run 32-bit wine applications on a 64-bit wine.
So to move the 64-bit, 32-bit translation into wine so that the native libraries are
all 64-bit.
I'm not sure if that would actually solve the year 2038 problem.
Fair enough.
So we're not sure.
Totally good.
So does anybody have things which they know will break, which have not been mentioned?
Yeah, because, I mean, you've shown us this magic autocon variable that switches the transition
to 64-bit time to off, but you cannot be sure that somebody used, well, maybe an unreleased
part of Cnulip into his autoconf code or whatever, and there are lots of exotic build systems
out there where somebody may just say we are using 64-bit time, and then you have some
random piece of software that is compiled in one way, and some other random piece of
software that is compiled the other way.
So it's kind of already, we are already in the mess, and we just need to find the best
way to get out of it.
And the other thing that I wanted to say is that, I mean, I'm personally, I'm kind of
a bit of a fan of the Cnulip.
Who want to keep some ancient software running with binary compatibility.
So yeah, I think, as I said, the big boys just abandoned 32-bit world, so they decided
it's not their problem.
Yeah, I'm also not from wine, but for sure, regarding what I said before, yeah, 1.8 is
basically going to start to support running on 64-bit, so the 32 binary, so that would
be a problem.
Another thing you mentioned, like, stuff like D-Bus as well won't be a problem because,
I mean, D-Bus is not tied for time, it's basically passing numbers, you define the kind of...
It's not passing time-structs around to people.
Yeah, I mean, it's my passing time, but it's not the time type, so basically you define
the kind of...
I don't know how it works, but yeah, any sort of protocol like that which was being given
structs by one application and taking them out by another, if they have a different understanding
of how big the time variable is, it won't work.
Yeah, you know, that's exactly true, but I mean, in that case, it shouldn't be a problem.
Why it might be a problem for stuff like using sockets or FDs, that's indeed an issue, and
that's...
Right, I mean, just to respond to it, Andreas, the, what's he going to say?
No, my brain's failed.
No, that's right.
The fact that people, as you say, already there's a certain amount of randomness in
what's built out there.
If we're not noticing a huge number of problems from it, then maybe it's not too bad, right?
It could be viewed as a good thing, as a good sign, I don't know.
I mean, in any case, when you said that you have to do a large-scale rebuild, and that
source distros have it easy, that's not entirely true, because we will also have to schedule
for our users a large-scale rebuild.
True.
Easier, I think.
Yeah.
There was someone up there.
Yeah, there's at least an NTP which has some time of time t and build a system.
Yeah, that's 20 days six, that runs out.
So actually, that's first, and maybe an even bigger disaster.
No, I further understand the other time, t2.
I need to find that link for you.
Okay, yes, please.
If anybody wants to send me examples of things that should be tested, that will be brilliant.
I think it's probably the way to do it.
So the other thing that I've failed to get onto is there's a list where we intend to
discuss this and produce a plan.
Quite soon, I think we should do something like this year, whatever the hell it is.
And we need a reason, some representatives from each, it doesn't need to be very many
people, but the fairly small number of people who actually care about this problem.
Which you're particularly worried about, yeah, I guess that's our time anyway.
So thank you very much.
Thank you.