Oh, last talk of this session is about the RUM archive, which is a data set of anonymized
real user monitoring measurements.
Now I know what some of you are thinking, Robin, if it's a data set, why does it have
a palm tree in the logo?
That doesn't make any sense.
But think about it for a second.
What happens if you go to a palm tree and you shake it?
Something interesting might fall out, like a coconut.
And the same thing happens with the RUM archive.
If you shake it a little bit, something interesting might fall out.
But for both, you need to be a little bit careful.
Because if you're not, the coconut might fall straight on your face, leaving you scarred
for life.
So we need to be a little bit cautious in how we query the RUM archive, and we'll get
to that later.
The first thing I want to explain is what is actually in there.
How do we get the coconuts in there?
So currently, all the data is from the Akamai Ampulse products, which basically means we
have a lot of Akamai customers that have Ampulse, and they let us put a piece of JavaScript
on each of their pages.
So every time a page is loaded, we send what is called a beacon, which contains all the
performance measurements and a lot of other metadata for later analysis.
Now, usually, our customers only see their own data, obviously.
And here, we want to make this more publicly available.
So we have to do a couple of things first.
First of all, we filter the data.
We only do the top 100 customers in terms of traffic.
We anonymize the data.
This includes stripping all of the URLs.
So you won't know which measurement belongs to which site, which is a sad but necessary
operation that we have to do.
And then we further aggregate the data so that many similar measurements are actually
combined into a single histogram for later analysis.
This gives us two data sets.
One, of course, for the page loads, and then one for third-party resources.
These will be things like Google Analytics that are loaded from external URLs by many
different customers.
And so we can also offer some insights on that.
We have most of the performance metrics you would expect, including some others, like
RageClicks.
This is if people got very frustrated.
They start clicking the same area of the screen trying to make it work.
For the third-party resources, we can also show if they were loaded from the cache or
not.
Very interesting.
But crucially, one of the things we try to make the difference in is that we collect
data from all the different browsers on all the different platforms.
And you can, of course, also query on those from the data set as well.
Now, you might be thinking, Robin, sounds fine, but don't we already have this from
other public data sets?
And partially, yes, this is true.
We are blessed with very good web and web performance data sets, but we still feel that there are
some gaps in there, gaps that we hope that the RUM archive might help fill, especially
when it comes to things like cross-browser and real-user monitoring data.
So let's say you're interested now and you say, how do I actually get access to this
data?
The main way is through Google BigQuery, where most of the data is stored.
BigQuery is a very powerful, very flexible platform.
It's sadly not the cheapest.
It does cost you a bit of money.
And even if you're willing to pay, it can take a while until you get useful data out
of this, which is something a colleague of the Mozillaeans here today noticed a while
ago.
The reasoning was sound.
They were trying to look for user agent Firefox on device mobile, expecting to get Firefox
mobile data, obviously.
It doesn't actually work, because in the RUM archive, Firefox is really just Firefox desktop.
If you want mobile, you need Firefox mobile for Android and Firefox iOS for iOS.
This is because we at the RUM archive put stock in consistency above all things.
Now, especially for newer users, going to BigQuery directly is sometimes a bit of a
big hurdle.
So we also have a cheaper way, which we call the RUM insights.
This is basically the team saying, OK, this is what we think most people will want to
know about this data.
We do the queries, and then we have some ready-made visualizations on the website for those as
well.
They also do the access.
Sadly, though, even the RUM insights don't really help much for the Firefox mobile use
case.
As you can see, Firefox in our data set is definitely present on the desktop side.
On the mobile side, none of the variants actually hit the 1% cutoff that we put for
generating these diagrams.
This is one of the many insights we can get from this data set, of course.
Because having a nice coconut is of course all nice and dandy.
You can't really do much with that, right?
What you really want is you want to get to the juicy inside of the coconut, in this case
the coconut milk.
Now, I can hear some of you thinking, you think Robin, there is no such thing as coconut
milk.
Okay?
Coconuts cannot be milked.
They do not have nibbles.
And you would be correct for the latter part, of course.
But there are still ways to get milk out of this.
You know, you could hit it with a machete.
Or if you're a bit more sophisticated, you could hammer a screwdriver into these black
spots there.
You could still get something out of there.
The point is, there are many different ways of getting the milky insights out of the data
nut.
But they don't all give you the same results.
And a good example of this, I found when I first started querying the room archive, I
just wanted to know, you know, roughly mobile versus desktop.
What are we dealing with here?
And when I plotted that out, I actually saw this weird periodic pattern.
You have these bumps and valleys in there, which seem to suggest that people switch
the type of device they use every three months, which of course makes no sense.
Okay?
And anyone who's ever done this kind of analysis already knows what this is.
You know, this is of course just a bit of temporal interference.
Because what I did not want to do was have a separate data point for each and every day
that would be way too expensive in Big Merry, right?
So what I want to do is just have one day per month.
And naive as I was, I chose the first day of every month.
Now this is not always the same day of the week, of course.
This can very easily be a Saturday or a Sunday or a holiday, where you would expect more
people to use mobiles than desktops, of course.
The solution is of course also very simple.
So the first day, we just use the first Tuesday of the month.
Not the Monday, because that's often also still a holiday or a vacation day.
But Tuesday should give us more consistent results.
It's not fully foolproof though, as I found out.
The first of July last year was a Saturday.
So the first Tuesday of July was the fourth of July, the big US holiday.
And that definitely does show up in these metrics.
But this is not just something specific to the RUM archive, and
every temporal data set has this.
But I think it bears repeating because people keep making the same mistakes
there, including me.
Now diving a little bit deeper, looking at the different OSes that we see.
On the desktop side, it's probably somewhat as you might expect.
But on the mobile side, we have a very outsized representation of iOS devices.
At nearly 63%.
And I say outsized, because if you look at the actual sales numbers,
globally, iOS fluctuates between about 15 and 20%.
Even if you look at some of the richer countries, like let's say Australia,
you expect a more 50-50 split.
There are several reasons why iOS is overrepresented in the RUM archive.
One of the main ones is that Akamai, as a company,
is mostly present in the richer Western countries.
Right?
And our customers are mostly from industries like e-commerce, luxury goods,
travel, that also address more richer end users as well,
that are more likely to be on, say, iOS devices.
So there is definitely an ingrained bias in the current RUM archive data set
that you need to be aware of.
But that doesn't mean the data isn't useful, in my opinion.
We can still do much interesting stuff with it.
For example, I think this serves nicely to highlight one of the big problems
I feel we have in web performance right now,
is our maybe somewhat overreliance on the Core Web Vitals and the Google Crux data set.
You might not notice, but on iOS,
you actually have no browser that can give you Core Web Vitals metrics,
not even Chrome.
This is because on iOS, every browser is actually Safari in disguise.
Apple forces you to use the underlying WebKit engine,
which does not support the Core Web Vitals.
And so the more iOS traffic you have,
the bigger your blind spot for those users is going to be
if you only use the Core Web Vitals and the Crux data set.
And I might say, Robin, that's only a problem for the customers represented in your RUM archive.
And I would argue that the RUM archive currently does not maybe represent the global web,
but I do think it's somewhat representative for, you know,
for example, the e-commerce industry,
which is definitely one that we consistently target when we talk about web performance.
So I do think this can lead to interesting insights on that part.
There is a silver lining to all of this.
As you probably know, the EU is trying to force Apple to allow other browsers properly on iOS.
Apple is dealing with this in one of the most disgusting ways ever.
In my opinion.
So I'm not quite sure how much this is actually going to change in practice,
but still it is a step in the right direction.
Okay.
And even if this doesn't happen,
we can still do some cross-browser comparisons
by looking at other metrics that are readily available on all browsers.
And we actually started doing this in the RUM archive,
because we have those metrics, of course.
I had hoped to present them to you today.
But we want to be sure that we are 100% correct in our interpretation of this
before we release any type of summary on that.
So not yet, but soon we are working on this.
I don't want to leave you hanging for today, though.
I do still want to give you something to take home.
And this is because there is a shining ray of light in the darkness.
Because a couple of months ago, Firefox actually announced
that they will now start implementing largest contentful paint.
First Corel Vital available in non-chromium browsers.
And this actually went live in stable Firefox about two weeks ago.
And we already have some of the data in the RUM archive, which I looked at.
And if you compare this, you will see that Firefox is actually faster than Chrome,
sometimes a little bit, and later percentiles significantly faster than Chrome,
for LCP.
Now, what I think this means is that Firefox has won the browser speed wars.
We should all immediately switch to Firefox and dump Chrome.
No, it's much too early for that.
We don't know if this actually means that Firefox is faster,
or if they just use a slightly different algorithm, or they identify different elements,
or it's just a different type of site that Firefox users visit.
We don't know, right?
So don't read too much into these results.
I just wanted to have something to start the discussion,
to start getting people and ties to actually look into what the core reasons for these results are.
But so, useful things for the future.
We're talking about Corel Vital, so you might ask Robin, what about INP coming up?
INP is actually already well supported in the Ampulse products.
So you can see here, this is an INP screenshot from the previous speaker's website, Tim's ScaleMates.
You can see Tim has a ton of work to do.
He claims he has the fastest website in the world, but we can all see the proof that it is not true.
Shame on you, Tim, shame on you.
So INP is in Ampulse, it's just not piped through the Remarkive yet.
We expect this to happen in the coming months, and so we can also start analyzing data for that.
So up until now, I've mostly been talking about the milk in the coconut, right?
But we all know there is something else in the coconut as well.
The flesh, the meat of the coconut.
We rarely eat this directly.
We usually process it into other foods, such as, for example, these delicious coconut cookies.
These are actually kind of a Flemish specialty, I think.
We call these Rotskis.
I think there are amazing, amazing cookies.
Now one thing you might see is that there are several individual cookies in this box, right?
But they all look kind of the same.
They're all quite similar.
And sadly, that is also something that we see for the third-party resources that we have in the...
He's not human.
Kind of human.
So third-party resources that we have in the Remarkive.
Because if you start looking into this, a lot of them are from Google, as you might think.
Most of them are ads, or tracking, or analytics, right?
Most of these are things that the typical end user probably would like not to see loaded on the pages they visit.
So it's a little bit ironic that we have to go all the way down to number 98
to find the first sign of something that is created to try and mediate some of this,
which is the very first cookie consent manager, the GDPR backwash, let's say.
I say try to deal with all of this.
I'm a bit skeptical that it actually works.
But I mean, the fact that you have 100 entries before the first cookie consent manager,
I think is a nice one-slide summary of some of the things that are wrong with the web today.
Now, this was a bit of a downer, so I also wanted to end on a better note.
So I went through the whole list, and almost near at the end at number 498,
I find something that we all were hoping to see, which was, of course, the jQuery mouse wheel plugin.
With 13,000 downloads every single day, half of that is from Tim's site, as we just heard.
And then the other one's there.
So jQuery is still going strong.
Let's hear it for jQuery.
As I said before, we also have some other stats on these third-party resources.
For example, how often they're loaded from cache.
And at the median, this is actually quite low.
It's only about 2%.
I definitely think that the browser cache partitioning plays into this.
It gets better to higher percentiles.
But so most of these third parties are not actually loaded from cache.
This might not be a huge problem, though, because most of them are also quite small.
Most of these are tracking pixels that are just a few hundred bytes in size,
though there are definitely outliers.
So one of the bigger ones that I found was a Google Ads JavaScript
that was 131 kilobytes compressed.
That's massive.
And that was loaded over 260,000 times in a single day.
So a very big impact just on that one external resource.
Now we had a lot of different resources, a lot of different cookies.
Another thing that we have a lot of is browser versions.
Because browsers a few years ago, they started updating themselves fairly regularly.
So for example, Chrome releases a new version almost every month.
And the question there was, how long does it take for most users to switch to the new
version?
That's actually quite good.
Because here is that within two weeks, within the month, over 75% of Chrome users are on
the latest version.
And most of the remaining ones are on the previous version.
There is a very short long tail of versions present in the dataset.
This is very similar for Firefox, which also very aggressively updates.
But here we do see one interesting data point, which is the blue one here, which starts in
August.
And even in December still had about 13% usage.
And it turns out this is something they call the extended support release, which I think
is a long-term support version there.
Probably mostly used by companies, I would imagine.
So you do have a bit of a longer tail there.
But other than that, Firefox is also a very cutting edge, I would say.
This is of course contrasted with Safari.
It's not an entirely fair comparison, because with Safari we don't have the minor version
numbers like with the others.
But here we still see the global trends, right?
The latest version of Safari is 17.
Even after two months, didn't even reach 50% of the Apple product population.
And if you look even version 15, which was released over a year ago, is still at about
14% of all page lists.
So clearly in Safari you do have a lot of older versions, up to a year and even older
present in your dataset.
You can't really rely on newer features being readily available there.
A very fun one there was Facebook.
They have a ton of versions, often multiple per week.
And their clients apparently also update to the new versions very, very quickly.
Meaning that I often only had one data point per version, which messed with my graphing
library.
It tries to draw a line, finds only one point, and then decides to just draw nothing at all.
Now interestingly, this is exactly what would happen if you would leave me alone with these
cookies.
You would know that there was supposed to be something there, but there's no physical
evidence of it whatsoever left.
So a couple of other things.
So this is again from Tim's website.
So Tim has his own very extensive ROM setup, as you by now all know.
But even for people with their own ROM, I think it's useful to have the ROM archive next to
that so you can compare both of these.
For example, this is for the navigation types dimension.
The biggest part is normal navigations.
You click a link, you go to the sites.
You can also have back forward navigations.
People press the back button, which should be much faster because it should still be
somewhere in the browser loaded.
And they have things like reload.
So people actually hard reloading the page.
Now for the back forward navigations, you want to see as much of that as possible.
Because people, that is the faster that you can get.
You can see here that Tim has clearly very well optimized for this use case because he
has a lot more people doing back forward navigations than the averages that we see in the ROM archive.
So good work, Tim.
The same goes for reloads.
Reloads you actually want as few of those as possible.
Because when people reload, it usually means something has been going terribly wrong and
they're doing the have you tried to turn it off and on again, Mattage, to try and fix
it.
So Tim is only at about 1% there.
Well that is much lower than what you see in the aggregated data there as well.
So it can be useful even if you have ROM to compare to see where might we improve or where
are we actually doing better than others.
Or let's say you want to move into a new region or a new country that you don't have ROM
for yet.
You can try and get some ideas about what the situation is there before you actually do.
And so to thank Tim for everything that he does for the web performance community, I
actually brought him a little gift.
It's a palm tree scale model, Tim.
I don't really know what to do with these.
Maybe you can have one of your tanks drive them over or something.
I don't know.
But so thank you, Tim.
Thank you for that.
Another thing I really wanted to look at was single page apps.
I have to admit something.
I am still on Twitter.
I still call it Twitter as well.
And if you're on Twitter, sometimes it seems like everything is react.
Nothing else exists on the web anymore.
All of it is react.
All of it is single page apps, which I really hope is not the case.
But when I looked at this, I was somewhat surprised because more than 40% of all page
loads in the ROM archive are actually single page apps, which is much more than I would
have thought.
Now, for web performance people, this is actually good news.
This means we have a lot of job security down the line.
So that's good.
It's a little bit weird.
And another very interesting point here is the difference between hard and soft.
So hard means you load the initial load of the single page app, the spinner that we saw
before.
That's basically the hard.
The argument being you download more, it takes a longer time to load the very first
time, but after that everything is much faster.
That's usually the selling point for an SPA.
But if you would take this at face value, you would say for every hard load, there was
only one soft load after that, where you would expect a lot more soft loads and hard loads.
And if that is actually the case, then that whole argument for SPAs doesn't actually hold
through at all.
Now, that's not what I'm saying.
We need more research.
I need to look deeper into the data.
There could be other explanations for this.
But interesting to think about.
I definitely did not expect these results.
And I would love to compare these with other datasets there as well.
I'm running out of time.
I had a little bit about HTTP3 there as well, including some things where I got very angry.
But let's skip that because I really want to get to this final page.
Because we all know that coconuts are amazing.
They are exceptionally delicious.
You can make a lot of different products for them.
But you can make them even better if you combine the coconuts with something else.
For example, delicious Belgian chocolate.
I think you can get into a very much a 1 plus 1 equals 3 situation with this.
In case you haven't tried this Belgian coconut chocolate, it is to die for.
Definitely try it out.
What I'm trying to say is that currently the RUM archive only has Akamai impulse data.
We are very much open to other RUM vendors or even large sites with a big RUM presence
contributing data to the dataset as well to hopefully help us remove some of the biases
that we've seen to get a better picture of the actual global web in the RUM archive
there as well.
Some of you might think, sounds interesting Robin, but this is going to be a lot of work.
Isn't it?
No, no, it's actually super easy, barely an inconvenience.
Because if you look at the SQL query that we use to put impulse into the RUM archive,
that is only 1.6K lines of SQL.
Only 1.6K lines.
Very simple, two hour stops and the data is in.
Well I guess the message is clear.
The RUM archive is now open for business.
What I talked about today is really just the highlights, the top what we can do.
We are literally just started to shift the coconut milk there.
So if you want to help out with that, please come.
If you have any questions, if you want us to run some queries for you, if you want
to help with the analysis, please let us know.
If by now you are just really, really hungry, I would say please come and try out some of
the excellent chocolates and cookies, because there's no way I'm taking them home with me
today.
Okay, so please, thank you.