Okay.
Welcome everybody at high noon at Sunday at Fosdem in Jean-Saint.
We'll hear now David Walter.
He will tell us about an on-cloud implementation for schools in Bavaria.
And scaling up for 5 million users.
A warm welcome to David.
Thank you David.
Hi everyone.
Today I will talk about how we did on-cloud implementation
to scale up to 5 million users or will scale up to 5 million users.
And I will talk a little bit about our school project.
So what is our target?
We want to scale up to 5 million users and we want to be better
than every other hyperscaler we know so far.
But before we dive in I will give a quick introduction about
who we are talking about.
My name is David Walter.
I'm project lead of the Bavarian School Crowd project.
And I'm also responsible for the experience,
the customer experience in on-cloud and for the security.
I'm using on-cloud as an open source user administrator since 2014.
So I'm quite familiar with the project.
And yeah.
On-cloud itself launched in 2010 and the on-cloud infinite scale implementation
we are talking today about had general availability in 2022.
Right now we are already hosting more than 2.5 thousand tenants
and we have over 1 million downloads of infinite scale.
And it becomes more and more a backbone for service provider.
As it is already during the press, we got acquired quite recently
by Kiteworks, our new mother firm.
Kiteworks is a security first company and it empowers on-cloud very much.
It helps us to drive security, to drive privacy and to drive compliance
in the same way as they do because Kiteworks is a security first company
which emphasize privacy compliance a lot.
And it also helps us, not only in the project but as a company itself
to provide 24.7 operations and 24.7 support.
There is much more to say but please read that slide afterwards
because I think we are kind of short in time so I uploaded the slides to FOSSTEM
and please read the slide if you are interested.
More in those details.
So just to give a rough idea because I don't know how much you know
about the Federalist State System in Germany.
I want to give you a brief overview.
We are talking about the Federalist State Poverty which has 2.6k schools
compared to Germany with 32 thousand schools and European Union 200 thousand schools.
Because of course we would like to have some kind of prototype
in what we are doing here in Bavaria and to achieve higher goals as you can imagine.
That's why I put here some numbers to give you a rough idea about what we are talking about.
We are talking about 1.6 million pupils.
We are talking about 116 thousand teachers.
Roughly 2 million parents in an overall Federalist State of 13 million inhabitants
which leads us basically to an infinite scale instance of 4.7 million named users.
Why I am focusing on the named users we will come later when we talk about load and performance testing.
So what were our goals?
We have some on a really really high level we have some major topics like data sovereignty,
a most secure system, a most secure fight against shell solution.
We have digital accessibility as of course one of the main targets.
But firstly we need to compete with the hyperscalers
because this project will be always compared to Google, to Microsoft Teams or whatever.
So we have to focus as much as we can on availability, on scale, on reliability as much as we can.
So on a more functional way we talk about maximum integration,
the integration of a messenger and of different other solutions that we will come later on.
We need to scale out without any limits.
We have seen it in the ICDays where we had a partial lockdown in Bavaria recently
that we need to scale out really fast and without any limits.
We have to target updates without zero down times.
And the most challenging when it comes to the project, we had the first 50 schools on board it
after 32 weeks of project duration which was really really challenging.
So let's get on to the base.
What is underneath the project?
When we talk about the infrastructure which is provided by Plus Server,
we are talking about four data centers in Germany.
We are talking about two S3 locations.
We are talking about four metadata locations, 240 Kubernetes clusters,
104 petabytes of S3 storage and up to 60,000 virtual CPUs and 120,000 gigabytes of RAM,
which are quite big numbers.
The project itself or the Kubernetes stack itself is based on the server in CloudStack and Gardner.
That's the logos you see.
The applications the user sees are, as I mentioned already,
of course on Cloud Infinite Scale as its core and the only Office document server
for writing, Excel sheets, presentations and PDF viewer and annotation.
But why did the customer choose, or why did they choose on Cloud Infinite Scale?
Just as a very very rough overview.
The first thing which was very important that we can liberate the user data.
The users have an interface to download their data from cloud to cloud,
from OneDrive, Google Drive, on Cloud Classic, NextCloud,
whatever the schools have already in place or the teachers mostly using their private lives.
We want to liberate the user data.
That's why we have an interface which allows the user immediately to move the data.
The second thing is the spaces.
The spaces is a collaborative workspace which doesn't have an owner.
It has only a manager and it avoids data silos because, as you can imagine, teachers are collaborative people.
So they have those data silos or they have those data silos and when the teacher left or called in sick or whatever,
the material were stuck in his storage so he needed to be sent out.
So we came up with spaces with fixed problem.
Quarter management, as you can imagine, is quite important as well because there is literally a maximum of available storage.
When we are talking about 4.7 million users, there is.
There is just this multiplicator.
The last thing is, it is a technology where I pointed out only one specific thing which makes infinite scales so special.
It got rid of the split brain we had before with OnCloud 10.
Before with OnCloud 10, we always had the database which keeps the metadata and we had the storage which keeps the data.
When one part of it breaks or whatever, you have the problem that you lose more or less all your data.
It is really hard to restore those data depending on the metadata.
I don't know whether you are familiar with what metadata in this perspective means.
It means, for instance, with whom did you share the data?
When you miss this information, because for instance, the database collapsed, you have a real problem.
We don't split this information and we store the metadata right with the data.
The iPad app, just try it out yourself. It is just too good.
I said already we had load and performance testing a lot.
As you can imagine, schools start at 8 in the morning and every school starts at 8 in the morning.
We know we have a huge load peak at 8 in the morning.
We needed to make sure that this really is the software that is capable of dealing with it.
What did we do? We designed two different test scenarios, one was test to fail and one was test to pass.
We know precisely how far can we go and what is our sweet spot to go through.
The target is that 95% of each full request, which means I do an action and as a user,
I feel the action is completed after two seconds.
That is what we are aiming with 95% of the request.
We did it. We did it actually.
The test to pass, the really test was 0.5 seconds of a full painted picture.
Simultaneously, we did X test so we know that the user experience stays the same.
I am running out of time so I am just going.
What I want to emphasize is because we are in FOSSTEM, that onCloud Infinite Scale is not the only product which is in that platform used.
OnCloud Infinite Scale only offers, there are different other softwares in between.
There is a key clock as IDP, there is open LDAP as IDM, there is CLEMAV for the antivirus scan, there is a patch a ticker for the full text search and the OCR.
There is a post fix for sending out the emails.
When it comes to monitoring, also this is Grafana Open Observability Stacks.
This Grafana, that is Prometheus, that is Mimia, that is Tempher and that is Loki.
The operations which is done by our own staff is also mostly driven, more or less fully driven by open source solutions.
We have Ansible with the AVX interface.
We have the OCI by Sonataip, we have the Nginx with the web application firewall, etc.
We have a helm for the package management.
That is quite a big zoo, I would say, when it comes to the software we do use.
We are not alone on the planet with our project, so we need to integrate in the universe which already existed.
The first thing we need to do is to integrate their key clock, their SSO key clock with our key clock.
Why don't we use immediately the same one? Because we need to host our clients, we have to host some other integrations, etc.
The key clock installation went bigger and bigger and bigger, so it was an early decision to say we will host our own key clock and this will be federated with the customers SSO.
Then there is the Metrix messenger by Stewie, which is integrated in a way that you can use the messenger and pick files from onCloud Infinite Scale and send it there or...
...
I hope I understood your question correctly. You are asking about the autoscaling features we are using?
Yes, because the user's pattern is relatively predictable, so everybody connects a day, you autoscale and downscale when people don't connect.
Basically, this is the Kubernetes horizontal pod autoscaler, which is playing a very important role in it.
There are other elements like resolver porting, etc.
We are using all this information stored in the Helm charts.
Of course, we don't scale to zero because we want to stay HA, high-waivable, at 24.7.
That's why we don't scale it to zero.
Of course, we do the scaling.
To give you an impression, we had one IC day last week or the week before.
Actually, it was three days.
Because of homeschooling at that day, we had a load jump of 60%.
It went smoothly without a second of outage.
Thank you for that question.
I was wondering, can you tell us anything about what sort of user feedback you've received today from teachers, students, parents?
Yes. Thank you for the question. You are asking about the user's feedback.
In the first period of time, we had 50 schools which were monitored very closely with LIME surveys.
They were giving us immediate response to the overall feeling.
Then we had some, I don't know how to call it, black box testing where our product management watched actually the users how they do some tasks.
We improved our user interface and user experience according to it.
Over the time, we have every four or six weeks, I'm not sure, check-ins with some certain key user groups which collect their impressions from the schools and report to us.
Does it answer your question?
Okay, perfect.
You showed us you had 32 weeks between design phase and the first onboarding.
Yes.
Did you do a comparison between your vision and the commercial solutions that were available at the moment?
Yes, but this had been mostly done during the tender phase, as you can imagine.
We were checking on features, of course.
We were checking on as files available on how they do stuff.
At some certain points, we also did our own ways because we do have some insights provided by the customers, like for instance, school start at 8.
I mean, it's common knowledge, but anyway, so we had some insights we could prevent to our platform, provide to our platform.
Right now, we are still learning and getting better every day, right?
Because even though I'm mentioning 5 million users, that's an adoption rate of 100%.
Right now, we are at 950,000 users, but after 10 months after the initial start, the first school,
after the first school, right now, we are 10 months behind.
Okay, thank you.
Any further questions?
Hi.
Hi, David.
So since you store all the metadata with your files, I guess then on S3 or object storage,
is there any sort of database for any sort of metadata, or is that all like user data is stored on Keynote, SSO only,
you have metadata of files, which is together with the files on the object storage?
Yes, thanks.
The only database, or more or less only database we are using is Postgres SQL for Keynote.
The metadata of the files are stored on RWX storage within the Kubernetes world, let's say,
and we have our database to keep the metadata, which is the huge difference between onCloud 10 and onCloud Infinite Scale,
because we needed to get rid of this split brain.
It always is a hassle if you have to connect to the database and to the storage at the same time,
and you cannot synchronize those processes 100%, and that's just impossible, right?
So with this approach, we can make sure that this isn't a hassle anymore.
Does it answer your question?
Yes, just one thing.
But you still store metadata and literally file blobs on different places,
so in theory there's still the possibility of some sort of desync between the metadata volumes and the storage?
In theory maybe, but I would say that's a very scientific question.
I haven't seen and I couldn't imagine a vector in practice. Let's put it this way.
Thank you.
You're welcome.
Hello. Thank you. I'm here.
Great work and congratulations.
Thank you for your presentation.
The question is this, is it possible to try to make all this available for little communities in a little laboratory,
or are there components that works only just on the power of the cloud?
Thank you for asking this question.
This is one of the most important things for us at OnCloud Infinite Scale,
because we come from the community, we had developed this in close collaboration with the Zern,
so this is also something we want to give you back.
There is, when you go to, for instance, to get up to OnCloud.dev or to the documentation site,
there's also available OnCloud Infinite Scale as a single binary,
and you can run it in Docker, in Docker compose, or just as a single binary on your computer.
Please feel free to download.
Thanks for that question.
Hello.
Hi.
I would like to ask, like, if there are any top level administrative tools for managing such a big cluster of Kubernetes nodes.
Sorry, I didn't get a word.
So I would like to ask, if there is a top level administrative tool,
apart from like Grafana, etc., that you used to manage such a big cluster to deploy deployments, etc.
Because it's such a big cluster of nodes.
Yeah.
Well, there is the Gardner provided by Plus Server,
which gives us the opportunity to manage the clusters in a graphical way,
and it also has some CLI, etc.
But our philosophy at that point is more put the genius into the code, into the DevOps code.
So when you're asking for something like a Sousa Ranger or something, we don't have that, to be honest.
Yeah.
Okay, thank you.
I will have to interrupt you because time is over for that slot.
First, I want to thank you with a little swing here.
So that's nice.