Okay, please welcome Dorota, who will talk to us about Wayland's
input method. Thank you.
Yeah, this is really embarrassing. You have already seen my mistake when I was trying to set it up. Well, yeah, I have another confession to make. Yes, it is completely my fault that Wayland input method is broken.
But I hope that after I explain it, you will judge me fairly. And me is Dorota. So, if you recognize this device that I try to use, it's Deliberant 5. And this is the reason why I'm even involved in this.
And a couple of years ago, when the project was starting, I got hired to work on the input method to make sure that people can actually enter text on the phone.
And there was a phone that you can't really text people with. It's not quite great of a phone. And we have already a solution. As you can see, this phone does not have a keyboard.
So, this issue is not a new issue. So, people have already thought about it. The input method is a way to make sure that text gets entered on the computer without dependency on a keyboard. So, you can see some examples here.
In non-Latin scripts, they use a keyboard but not in the direct way. You press the buttons, but something else comes out. Text recognition, like handwriting recognition, doesn't need a keyboard at all.
And finally, this is what is useful for my case, making sure that you can type on a phone, can use an on-screen keyboard. But how do you plug it into the actual computer? You can use Wayland for it.
Wayland is a graphical system, not quite. Wayland is actually a set of protocols. A set of protocols for connecting the user to applications. Applications, and user is represented by input and output devices, because that's really how the user looks like from the computer's perspective.
And considering that I wanted an input method, it kind of fits to replace some of the keyboard's typical competences. So, maybe that's the way to do it. And it seems that people already agreed with me on that.
There were existing protocols in Wayland for inputting text. And you might think that this is where I messed up. I tried to implement them wrong. In reality, it's much worse than that. Those protocols were not adequate. They were basically prototypes.
So, I took it to myself to make new versions of them. And this is where I really messed up. I hope you forgive me.
Let everyone else judge me after you see the presentation. It gets worse.
Yeah, SqeakBor is an implementation that I used for it. I made some mistakes. One of the mistakes was sticking to keyboard roots. So, there are some operations that you want to do as a user that you expect from your mobile phone to be able to do that are not inputting text, but you live in kind of the same area.
Like going to the next text field or submitting a form. You have this big enter button on your keyboard, right? Or like navigating text, moving the cursor left and right. This is not inputting text. This is something else. What is it?
So, I thought, yeah, I have to do it. And you can see this is a screenshot from keyboard. And the key is actually there. It does something because I decided, well, we already have a keyboard emulation thing for other reasons.
So, maybe we can just reuse it and emulate the keyboard when it's not about text. And that was at that end. But why? When you are using an input method, you are concerned with text. You want to submit text to other applications, so you submit text directly.
But when it comes to keyboards, you don't design a keyboard protocol about text. You design a keyboard protocol around key events. Press key, release key. This is what the application gets. It's not so straightforward, but actually it's even worse.
Instead of submitting buttons, you submit numbers. There is no preset relationship between numbers and letters. So, you have to decode them. And because you have many different keyboard kinds or layouts, see, this is a very tiny keyboard. It doesn't look anything like an overall one.
This keyboard can input two different kinds of characters depending on which layout it is. This gets complicated because those layouts called key maps, they can be different. And despite that, in Wayland, even if you have no matter how many keyboards you have connected to your actual computer, virtual keyboards, like emulated keyboards or not, they all get presented as a single one to the application, which leads to problems.
Because how is the application going to determine which keyboard you actually used and what character did you actually want when it only sees the number? Okay, you can send the key map, the table of relationships at every key press or whenever the user switches.
If you press the smashed both keyboards at the same time, it's, oh, God, that's getting complicated. How do you disentangle all those modifiers, all the key presses? Basically, the Wayland protocols system is meant to care about corner cases.
It's meant to work for everyone and not just for my cases on the phone. So even if you might never type from two keyboards on the phone, someone else might be interested in doing that. So basically, Wayland maintainers told me that this is not going to work. We are never going to accept it in this way.
So yeah, what do I do? I want my software to gain wide adoption, but a protocol which doesn't work for intended use case without the supporting protocol, the supporting protocol being the keyboard emulation, without keyboard emulation, my input method was kind of useless for typing on the phone.
So, well, I ran out into it without knowing what I was getting into. And this is my mistake. I hope you will forgive me. And this is not the only mistake I made. Oh, wait, maybe you can forgive me knowing that I actually know the solution.
This is one of those cases where I know the solution. I think that what will work is a new actions protocol which combines all those things that are not relevant to the text input but are still useful. So maybe we can do that.
I actually started some work on it. I'll talk later about it, but this is not the only mistake I made. I made so many mistakes.
So, digitalization is a difficult problem. So, I have a question to you. If an application that I want to type into, if it has a small lag and I keep typing, what should happen? If I type two separate words, what should appear in the text input afterwards after the application comes out from the lag?
Come on, come on. Yeah, you should get two separate words. In my protocols, you cannot have it this way. This is a bad thing because when the application starts lagging, you drop events and it can lead to stupid results.
But the reason synchronization is this way is I wanted to make sure that the way we even have synchronization, the input method must be aware of what is inside the text field for the trivial reason of having completion suggestions.
And I designed the protocol in such a way that every time the input method sends an event asking the application to put something in, it also sends the identifier of the state that it thinks the application is in.
So, it always says, please do this considering that you are in that state. And when it sends two events, one after another, it only knows about the first state. So, by the time the second event is sent, the state has been changed by the first event.
So, the second event is going to get ignored because I wrote this in the protocol. Please ignore this event. Yeah, so this is awkward. But I did it for a reason.
So, the reason is kind of complicated and many people would say it's theoretical. Why do you bother? Maybe just accept it because this is what's going to happen.
But I thought that maybe there are some reasons to let the application change its state from another source. Like someone who doesn't like me would say, oh yeah, you're going to type on your on-screen keyboard and on the real keyboard at the same time and modify the same text.
Well, that's stupid. But I was thinking actually before this presentation, I have really weak examples here. This is my example that maybe the application can do auto-completion in a smart way, in a better way that the input method would not be able to.
So, the input method should not get in the way if it conflicts. But there is a better way. If there are multiple people editing the same text document, I'm sure you have seen a system state that, then the text field is going to change a state independently of the input method.
So, there is a need to track the state accurately and maybe drop events. I don't know. That's the problem. It's actually quite hard and I have no solution to that.
So, yeah, maybe knowing that it's a difficult problem will help you forgive me for that mistake.
Right. But none of this is going to get fixed unless someone picks it up. If you can see, my last comment was four years ago and the last time it was actually accepted was two years ago.
That means that after I was moved from this project to another, the development will slow down a lot. So, there is a need for someone to pick it up.
And if you want to pick it up, you are free to contact me and we can chat about those problems and maybe some other problems because I'm sure those are not the only bugs that I left there.
So, you can catch me on my blog and on MasterDone as well as Matrix. And if you don't mind mistakes once in a while, I am also looking for work on free software. Thank you very much.
Thank you.
Thank you. Do we have questions? Yeah.
Hi. Has there been any interest by any companies or industry about financing this kind of work? And have you heard from Qt in any way because they drive their own protocols?
Right. I know they are forced by some of their customers to a little bit look into upstream input method protocols. But I think they just looked a little bit and then decided, nah, not interested. We have our own stuff.
Oh, I didn't know about that. We need to talk more about it. But I have applied to... I'm a little bit angry about the situation. Understandable. Yeah, we have worked with Roman on this problem before.
I have talked to Gnom and I have talked to NLNet about this a little bit. So...
You talked also to NLNet.
Yeah. I still haven't gotten proper answers, but we'll see where it leads in the next months.
Hi, Lurota.
Hello.
So I just wanted to comment quickly that it's not only your fault. I am one of several Wayland developers in this room who conspired to attend, not only because we like you and thought the talk was interesting, but because we also felt blame for what happened and saw the title of your talk and thought we ought to be there to accept some of the blame.
So you did good. You did great.
Thank you. Someone already forgave me.
More questions?
Maybe you can talk with Alfa as well. Maybe I can do some connections.
Yeah.
Any more questions?
Hello. So input method is clearly difficult, right? There are so many different layers and there are so many different considerations that need to go into that before you can create a sort of standard protocol that everybody can kind of agree on, especially on some mobile devices where we might not all agree on or completely understand everything that goes into that.
Having seen that firsthand, how do you think, what's the correct approach for not just input method, but any other, because Wayland is trying to do a lot of different protocols like this, how should we try to approach developing these sorts of protocols from scratch?
Because it's a big task. There's lots of different users, right? Like some people try committees and we have problems with committees sometimes. Some people try to just do it on their own and then that can cause problems.
So the reason that this succeeded even in a limited way was that I had a company which was really invested in a product and they wanted to roll it out.
So basically you need some person who cares and that was me for some time.
And the other way I found like experimenting with Wayland protocols is kind of difficult. There is WR routes but I found that even though I attempt to do my best and there are viewers who review my contributions or also try to do their best, there are still bugs months and months later.
So I would think that maybe we need to think about the easiest way to prototype protocols. I don't know. That's on one side and yeah.
There was for the specific case of Wayland input method protocol, there is also competition that does not go through Wayland. So there's iBus for example and I think KDE or GNOME are also kind of investing in iBus a little bit instead of this.
I can't blame them. Maybe it's easier to skip Wayland for some protocols. Of course you lose the security guarantees that Wayland provides but sometimes for prototyping at least it might be a good option but is it really useful if your target is Wayland?
I'm not sure.
Sorry.
Given that a Surface Flinger and Android's input system are part of AOSP, have you looked at that to see how much that differs or does that model kind of, is that too different from what Wayland does to be inspired by it or to see what approach they take?
Yeah.
Similar to QTs in built IAM keyboard that talks directly to the application and can be bundled into it.
Actually I don't know much about Android and Surface Flinger but I think one of the different mistakes that they made was that I did not look at other systems enough so maybe they already have solutions for some of that.
Thanks a lot. Final round of applause for Dora Tante.
Thank you.