So, hello everyone, and thank you very much for resisting this late in the evening, and
we have our last talk, and we have Emily Haro.
Welcome to the localization system and tool chain management, and he's going to have
a talk on the road to int-message format.
Oh, the int-message format, I should have checked it.
It's okay.
Hi.
So, the last talk by Mathias, if you were here, was a lot about where we are now, what
we can provide now already in Pontoon and otherwise, how localization is now happening
in Mozilla.
What I'm going to be talking to you about is what's kind of coming up, what are some
of the next things in localization that we're working on and that we think are really quite
important.
And so, yeah, I'm on the same team with Mathias and my staff, the software engineer, but I've
been doing this sort of stuff kind of for fun, for ages, it feels like now, so it turns
out that when you get really into localization in JavaScript in particular, there aren't
too many other people who are that into it and then somehow you might end up hired by
Mozilla to do the things you were doing for fun, for pay.
So that's kind of nice.
Hint, hint, you know, it's a good company.
In addition to working just on code at Mozilla, I spend a lot of time in a bunch of different
standards bodies working on the standards for localization in particular.
And some of the work I'm presenting here is really the work that's going elsewhere
than just at Mozilla because we want to have, we fundamentally want to make the world a
better place, the internet a better place for everyone, not just Firefox users, but
you know, everyone's internet isn't better if they use Firefox, but you know, you know,
you're here so you might have heard this one before, but yeah.
On localization, this is again covering a bit of what Matiasz was saying that quite
often localization is one of those aspects of how do you really build an application
or a site or anything that comes up way too late.
You end up making some choices early on and then you end up needing to live with those
choices later and they might not be the best stuff, best ones.
And the need for localization comes after you've made the choices or you discover that
hey, this thing, oh good grief, we need to support Arabic now, that'll be interesting.
And a lot of, the sort of scope of localization is interesting because there isn't necessarily
one right answer, so of course we're working on a new right answer and you know, there's
an XKCD comic on that.
I don't have it on these slides, don't worry, but you know the one I'm talking about.
So things could definitely be better.
So we're trying to make some of this improvements happen.
It should be easier to localize content and there should be a common way of doing this
so that the experience and use, the benefits that you get from using software and libraries
in one place can map to elsewhere.
Right now, there's a lot of differences in how localization ends up depending on the
formats you use and the tool chains you use and all of this and that is not optimal.
And fundamentally, a lot of actually when you start getting deep into it UI and UX design
ends up being limited to some extent by the fact that most of localization work is working
around strings rather than the complex structures like HTML allows us to represent and other
aspects that make life more complicated.
So we want to improve all of that.
So let's start with this.
This is nominally something simple.
Hopefully most of you can read HTML to figure out that here we have this small little span
that says that Brussels is the capital of Belgium, I've lived here, I know it's more
complicated than that, let's just go on.
And Brussels here happens to be a link.
So how do we make this localizable?
How do we, no, how do we actually localize this in a way that works really in the end
for everyone?
And one way that we're trying to sort of build towards is something a little bit like this
that you could add an identifier to the element there where you say that this is the Brussels
message that we're really dealing with and include in the HTML something like what we
have for CSS now where you say that here's this resource that's attached, here's a link
to a resource that's necessary for figuring out what's really the content of this page.
And then separately you have a message here in Finnish because you know I can and I could
not pick between French and Flemish and because it gets complicated.
I've lived here, I know, Brussels and Belgium.
And here the format that we're using, I'm going to get to that later, but there's a
couple of interesting things here in particular that the fact that we're marking up the Brussels
text there as the contents of the text of a link so that we'll be able to map that to
the link, the AHRF that we have in the source document there in English.
And because it's, you know, of course a little bit more complicated than this, it happens
to be a link to Wikipedia.
So in this particular case, but not usually at all, we could allow the translator to say
that hang on, this link in Finnish should really go to the Finnish Wikipedia page on
Brussels rather than the English one.
And this is like, I can present to you, you can see the screen, you can kind of get what
you're looking at here, but honestly getting this to a state where you can get a translator
who's not a developer to see this and understand what they're supposed to do and not screw
it up and provide useful things, useful content in all the languages, well, the languages
that this translator is working on, it gets kind of hard.
So we're trying to, you know, make that a thing.
And the rest of this presentation is really going to answer these three questions that
I kind of would have hoped some of you would be asking, but you're not.
There are really questions in my head I wish you would be asking, you might have as well.
But these are the questions of the theoretical guy in my head might be asking, what's the
format of this thing that we just saw and is this really going to work like everywhere
and how's this going to make my life better now or do I need to start using this whole
new thing and that's going to be a pain.
So I don't want to do that.
To tackle the first one, the answer ultimately to all of that is to standardize everything.
And the first thing we're going to talk about standardizing is the message there itself.
And one particular thing that some of you might have noticed is that it had curly braces
around the text there, around the Brussels on Helsinki, sorry, Brussels on Belgian Pääkkarpunki.
Sorry, what's that?
And this is because it turns out that when you're building a message formatting language
like this, oh good grief all the corner cases.
Oh good grief is it like hard, like proper hard because you're trying to write a formatting
language that developers understand and then get the developers to write content in that
language that translators understand without needing to have the developers necessarily
understand how translators think.
So you need to find an intermediate language for the communication to happen that explicitly
limits and forces the communication to work in a way that works.
And this is one of the reasons why some parts of this work have been in the active standards
body for like three years so far.
Yeah, one reason for those curly braces there is that quite often messages get complicated
because you need to vary different parts of them depending on different variables.
In English for instance it matters is it a he or a she or a day who might have you know
done the action here of sent an invite to a party.
So we need to have a language message format too which I'm presenting to you here to enable
this sort of a communication.
And of course it gets more complicated than this because you can have stuff like here
we have a need to include something more in the message of the relative time like say
three days ago that's included here.
So the language needs to allow for internal variables for this message to be definable
in a way that translators can kind of see what's going on and hopefully not touch it
too much because hopefully they don't need to do that but still be able to do so if they
really really need to.
So this is about the space of what's possible in most current no in some of the current
message formatting languages.
At least project fluent which we maintain and work with and maybe one or two others.
But when it gets really more complicated than that this is this gets on the edges of not
really even supported anywhere.
When you have here what we have are multiple different variables being defined and then
the matching on which of these messages really the message we're building it depends on how
many people as well as the gender of the host.
So this isn't even a full listing of the whole set of possible when cases that could
be selected here but this is all possible.
Quite often happens when you really want to formulate UX experience that is approaching
natural language and this is again referring to what I mentioned earlier a lot of this
stuff just isn't is the choices that people are making now regarding message formatting
how do they formulate it are driven by the limitations of the technologies that we have
available for us.
So UX itself is being driven in certain directions because message formatting is hard and you
don't end up really having messages like this in your UI if you care about localization
because whoever is filtering your messages before they go to the translators the localizers
is going to tell you yeah no you can't do that they're not going to ever be able to
work with it.
So please fix and then you end up even maybe building the UI differently in order to accommodate
these needs.
With message format too which this is I kind of hope we can get beyond that have the possibility
and the options of having even richer content in everything that we're working with.
But the second question there was about is this really going to work everywhere and yes
and we're doing that by trying to make much of the work happen at the lowest possible
appropriate level for the work so a lot of this is happening in the Unicode consortium
and then we've got work going on in TC39 for JavaScript.
It's being added to the ICU libraries provided by Unicode as well and eventually we're hoping
to get probably in what we do discussions ongoing about the structure of the HTML stuff
that I was showing you earlier because that doesn't exist either yet.
And one particular part of this I'm my background is as a JavaScript developer is that this
is the first time we're really adding something to the JavaScript language itself at the level
of like JSON.parse where you have this string representation of a thing that's not JavaScript
and you get an object or a thing out of it.
I think that's really cool but we're still working on it.
And the part here that makes this extra interesting is that we're not just talking about a new
syntax but effectively through the work we've been doing it's looking an awful lot like
everything in every single message formatting language that currently exists and is in use
somewhere that is, you know, that we can know about that is not like closed and proprietary
is supported in the data model that we end up with for message format too.
So for example to answer the earlier talks questions about how do you get support for
something like Fluent into software like translate toolkit, one quite probable answer for the
general case of this is that what you'll be able to do is take messages that you have
in dot properties files, Fluent, GetText, ExLiv, pretty much anything and parse that
into defined data model structure for message format too, then be able to work with that
using tools, runtime, whatever and possibly from there get it out in a different format
altogether that's then supported by other tooling.
So it's a lot of this work is trying to figure out that hang on, messages aren't really
all that complicated as data structures in the end or we can at least express the level
of the complexity, so we should enable.
Hello again.
So yeah, think I was about done with this slide and going on.
One key part here is that all of this is already real.
So what I showed you in HTML is not exactly what we use internally at Mozilla, but it's
effectively the same as how Firefox is now already translated.
We have by now literal years of experience of working with tooling like this and seeing
how it empowers UI UX development of a relatively complicated piece of software like Firefox
to improve itself and to enable easier and better communication between developers and
translators and so we're bringing a lot of that knowledge and experience into what we're
doing in the Unicode Consultium when designing Message Format 2 which is yes, taking inspiration
but also learnings from Fluent and many other systems that make it honestly a better than
Fluent currently is for instance which is why we're now pitching that as the really
cool sexy thing even though I mean if you're interested it is the currently coolest thing
around that's real, this is still in progress, so you know you could be interested in that.
As I mentioned the syntax itself for messages is getting defined under the Unicode Common
Language Data Repository Technical Committee, it gets complicated in these things and there's
an implementation available in ICU72 for Java and the JavaScript proposals, there's two
of them at stage one currently for this are progressing in TC39 which is the body that
defines JavaScript effectively and there's a polyfill package for JavaScript if you
want to start playing around with what Message Format 2 looks like and how you can work with
it but yeah, all of this is of course completely public, all of the repositories, all of the
work standards are being developed completely in the open and I mean honestly localization
is one of those weird places where we don't need to filter anyone on credentials for like
anything because in terms of who wants to actually participate in the standards actions
and standards work it's enough that you show up and you show some level of interest and
we'll let you in in all the like inside clubs and because there aren't any, it's a community
where really you can, if you're interested you should not be afraid of someone saying
no you don't belong here because you do, we need always more people participating.
Yeah, there's links to me as well and also this talk is available at the URL there at
the bottom, it's also attached to the talk on Pentebarf, but yeah, that was me.
Are there any questions?
The question is what really makes message format to better than fluent and one particular
example is when you get to complicated stuff like this, is having the effectively enforcing
the data structure that you end up getting from this to be one that contains full messages
that you end up representing to translators.
Other than this it gets into really nitty gritty details, the other big benefit of message
format to over fluent is that message format to is becoming a unicode standard rather than
effectively a project built entirely from within Mozilla.
So the question here is about seeing the sort of typing that you see, the colon number
and the colon relative time and actually the colon gender is the same sort of thing here.
What are those and are these custom or centrally defined?
And the answer is kind of yes and no and it's complicated because what you're looking at
here are effectively functions that act a little bit like types but they're not exactly
like types.
They're declaring for example that the count that we're getting, let's handle it as a number
but also let's in the value of it that we end up assigning to count other use an offset
of one.
So it's an operation happening on the input argument count and on the third line in the
match for the host's gender we could imagine host being some complicated object that's
defining a whole person and we're picking the gender information from that more complex
thing.
But yes in many cases they work kind of like types.
Influent, these are the capital number, capital date time and capital platform functions that
can be used in this sort of way as well.
Just be loud.
I'll repeat your question.
I feel like people are going to start saying like, okay, we can put links in now and then
it sort of escalates to what actually in this locale this whole structure of the page doesn't
make any sense anymore so we have to kind of like switch things around a lot based on
locale and I'm wondering if you have any thoughts on like where what's out of scope here perhaps
and if there are any other tools in development that you know of that kind of like are exposed
to not just message forwarding on small sentences or whatever but also restructuring pages based
on different locale.
So if I've understood the question is what happens when you come from a when you have
a complicated thing like a whole page that you're translating and in comparing the source
locale and the target locale, the target locale ends up having very different structure that
might you know be go much deeper I suppose than just the simple link that I'm showing
in this example of how does this really work.
The answer is it's complicated and it depends on your use case.
This work in particular is trying to build tools that could enable that sort of representation
within message format 2 so you could end up somewhere really complicated but you probably
don't want to.
You're probably in that sort of a situation needing to build more tools that are more
specific to the use case that you have.
When you have when you need to reformat a whole page in order to do work with a specific
locale it's there is no universal answer to this.
This is the closest thing but I don't know where it's really going to go.
We have a question in the live stream.
Translators often are not programmers they already struggle when translating strings
with HTML tags and other technical terms.
The message format curly braces syntax might be difficult to understand and error prone.
So here we're talking about something let's take this example of if you put this in front
of a translator yeah you don't.
This is not really what we want to do.
What we want to do is create a format that enables a like HTML a representation of something
like a message in a way that is relatively readable but is not necessarily easy to edit
and modify for someone who doesn't exactly know what they're dealing with.
A little bit like what happens if you take JSON and put it into a Word document and
then you start editing it and then you have to figure out that oh there's a curly quote
somewhere that ended up screaming.
This sort of thing can happen entirely well when you end up dealing with complicated messages
like this.
So the answer here is that you end up using tooling that gets this to not be presented
as one thing to a translator but three yeah in this case three or more different messages
where you end up asking a translator wants to translate name invited you to her party
on relative date and then a second to ask them to translate name invited you to his
party on relative date and in Finnish allow a translator because Finnish doesn't he and
she translated the same word.
So in Finnish the equivalent of this message would end up being effectively just the third
case without the whole matching because the structure of the language works differently.
So you do end up when working with messages of this level of complexity effectively needed
to rely on tooling but the wonderful thing about message format two is that we can transform
this representation of this message into any other representation of this message that's
hopefully going to work with whatever tooling is then available for the actual translation
work to happen in.
So XLIF two for instance or other targets that are commonly supported by software used
for translation or some really simple representation that can be mapped then back to this but still
allows a translator to just see a simpler thing at once rather than a really complicated
thing.
Think there's more questions but are we out of time?
Two minutes.
Guy in front, yellow.
My question is I can guess that you are targeting the new message format as a successor of all
previous attempts at message format.
It is relatively easy to make sure that everything representable in previous message format is
representable in the new one.
How are you solving the problem that you are really encompassing all the different languages
in the world?
Because like all the examples we saw were in English perhaps some of the others might
be like French or another in the European language.
The case here is just for female or male languages with much more complicated noun systems.
In some languages you might be writing a single message in several writing systems.
How do you make sure that the new message format encompasses all these different strange
cases for localization?
If I understand the question right you are asking how do you make sure that this isn't
really what seems to work for English in a couple of languages around English but hopefully
all the languages?
Or a sizeable number of languages.
The short answer here is that with Fluent we are already doing exactly this using representation
of messages that is very close to this.
For instance, at Mozilla from this experience we can say that the simpler than this structure
that we have for Fluent ends up working in all of the languages that we need to deal
with through Fluent.
We need to deal with Fluent which is about 100 for Firefox, 200 overall for all of the
different projects that we are currently translating.
Separately from this the work being done for message format too is by no means done really
from an English language point of view.
With the main contributors currently working on the specification my background is Finnish,
there's a Polish guy, then there's a Romanian, then there's a Sri Lankan and there's a couple
of others who are on the periphery of this who are from a much wider variety of backgrounds
than this.
So, we are bringing in ensuring that these sorts of considerations are actively being
remembered to be taken care of.
To some extent we are relying on the expertise that we have, to some extent we are relying
on the experience we have with working with similar formats than what we are presenting
here.
But also we are trying to build a core specification for message formatting that is sufficiently
small but modular and powerful to then enable the support later on that is required by human
languages.
We are trying to limit to just being able to support human languages but it might go
a little bit beyond that too.
I think we are out of time, I am very happy to have people come and ask me questions after.
Thank you.