Thank you.
So hello everyone.
My name is Hugo and here with my colleague Richard and Abbey from the 15 which will be present afterwards.
And basically we thought that it would be a nice contribution basically to foster and to actually present a little bit what we do basically as a.
In the storage topic we are the experts on the storage. There are other colleagues downstairs in the booth so feel free to pass by and get some stickers basically and have some discussions.
So yeah the core of this talk is to present a little bit what non-theft storage technologies we use on disk, on tape and cloud storage.
And to explain a little bit what we do with them basically.
How many of you know CERN?
CERN basically.
Okay good. This is an easy start.
So yeah CERN is the biggest laboratory basically for particle physics in the world.
It's basically based in the frontier between France and Switzerland.
And our goal basically is to really understand how basically the universe was constructed.
And we have a lot of intelligent people, a lot of physicists that trying to basically understand the data that we produce in this accelerator.
CERN is also the birthplace of the web as you know in 1899.
CERN 10 Berners-Lee basically created the worldwide web there.
And he gave it to the world basically.
And actually this changed a little bit how basically in society we communicate and we do things.
And we actually bring in this legacy. We really contribute a lot to open source.
We try to do basically as open source as much as possible.
Where we do not find something that we can find it open source.
We try to build it ourselves and give it back to the community.
And here you will find today a little bit of examples of these technologies that we use.
But before we jump there.
CERN is now a multi-b because of its large Hadron collider.
So this is a tunnel of 27 kilometers of circumference.
It's around 100 meters below on the ground.
We have a lot of superconductive magnets.
You can see basically in this picture there.
And the idea is basically we have different particles, atomic particles that basically we try to collision amongst each other.
And we have high resolution cameras.
It's what we call the detectors.
So here is just one example of one collision.
So these big cameras have particles basically coming from opposite directions.
And we try to basically align them.
So we basically create a collision almost at the speed of light.
We generate a lot of these pictures per second.
Just to give you a rough number is around one petabyte a second.
The throughput that is generated out of the detector, which we cannot handle.
It's really impossible.
So what happens is that we have four of these big cameras around basically the tunnel.
And we have filters basically because not all the collisions are interested for physicists.
Some of the collision they are already known.
So we try to really know, detect the exotic cases that we would like to basically invest.
Traditionally we have what we call basically the processing farm.
And it has been always done with FPGAs and hardware.
But now we are moving a little bit.
Different experiments they use now, for example, GPUs, hyperalyses, basically processes in software to do this filtering.
And we arrive roughly around one terabyte a second.
This is the data that we have to handle to process and to distribute.
So how we distribute this data?
CERN as you can see here in the center is what we call the TR0, like an onion.
We have TR0, TR1 and TR2.
So CERN has its own data center.
It's all the data generated and it really brings to this inner layer of the data center,
which is stored on high, say, throughput, this buffer, and then copied to tape.
And my colleague Richard will explain to you more details what these systems are about.
After the data is stored at CERN, we know that we have a copy of the data we distributed.
And this is what we call the worldwide LSE computing grid.
So this is an international collaboration of many people, many different encounters, many different institutions around the world.
And the idea is to distribute this data across all these 160 data centers in the world.
So people, basically physicists, for example, are in Oxford.
They can analyze the data sets they are interested in.
And at the same time, in the case of a big catastrophe, basically,
because CERN is built with public money from these institutions that believe in science,
the idea is that we can also try to reconstruct, basically, the original data with the copies that we have around.
Today, we start around one activate in disk and one activate in tape systems,
and we'll dig a little bit in the technical details later.
With a lot of things regarding computing at CERN, today we are only focusing on the storage and open source for science.
But if you go to home.CERN, this is the domain that you can go and find more information,
or just pass by the booth and we will tell you a little bit more about other dimensions, basically, of what we are trying to do.
So this talk today is going to focus on three and four of these systems.
So the CERN box, EOS, and tape part, me and Richard, we are going to cover it.
My colleague, Abby, will talk a little bit about what is the theft infrastructure at CERN.
So this is a high-level view of how it looks like.
We have the tape system, we have EOS, which is our software defined storage disk system with commodity hard disk,
because our goal is to really try to use the cheapest hardware that we can find to provide, basically, reliability for the data.
So there is no goal-plated hard disk that we use.
We really try to apply a cost factor to really be the cheapest.
And then we have CERN box, which is a cloud storage platform that sits on top of EOS and theft,
and this allows people, basically, to access the data in a draw-box-like fashion.
We use Anklot, the storage solution for that, because not all the people at CERN are geeks,
and they are basically SSH-ing into a computing cluster and going through Fuse and CD and getting data out,
but they just use computers, they want to get some data, do some basically computations in Mella, for example,
and then they run it into the computing farm.
So it's very important to bring the data that is generated to the end-user devices.
This is a little bit more, let's say, in-detail picture of what we do.
So on the left part, as I mentioned, we have EOS that provides a Fuse later,
and then we have CERN box that sits on top, and this provides access to basically, you know, 100 EOS devices,
also to different computing clusters that are running basically just Linux, Boxes, Almanine right now.
And we also support through Samba protocol on top of Fuse for Windows users,
because we have also a big user Windows population at CERN,
so it's important that we allow these people to also access the data,
and we use the open source and implementation for that.
JupyterLab, we have a system called SWAN, which is a branded JupyterLab environment,
which also sits on top of our storage systems, because it's a very convenient way for people
to just basically run some Python notebooks, for example, to run some interactive computations,
and then from this platform, you can basically scale out the job to big computing farms
using a GT-conder system, for example, for those of you who know the system.
So this is a little bit the big picture of the things, and now I pass the ball to Richard,
who will dig a little bit more in the different areas for the different systems that we use.
There we are. Lovely. Thank you, Hugo.
All right, so by now you understand that at CERN, we have a great challenge in dealing with the data coming in.
We have a lot of data coming in at once.
We store a lot of data over time, because most of the physics data is, in fact, kept in perpetuity,
and we also cannot lose data, because this would upset the physicists,
and of course, lost data equals wasted time at the LHC.
So for this, to solve these kinds of problems, we have developed EOS as our disk storage system of choice.
What is EOS? It is a storage platform, open source.
It is made such that we can store data in an economically viable way with the public funding we receive.
At the scales we operate at.
It is elastic, adaptable, and scalable, so that we can even grow on it as we go along,
and as we increase even in the future the LHC's capacities.
EOS is also very flexible in how it works.
So for one, we have from the start considered the use case of having not just hundreds,
but even thousands of parallel users active at once.
There are physicists on site accessing it, sometimes with multiple clients.
There are physicists abroad, as mentioned, and there are high performance batch computing workloads running this kind of stuff.
So this is from the grounds up in there.
We also support multiple protocols and a variety of authentication methods.
This way we not only can use EOS in the varied scientific system that we have running at CERN,
but also at the other institutions working and collaborating with us.
What does using EOS look like to the user?
Well, in the end you get a file system.
You can interact with EOS through the command line, through the shell.
So you get an EOS shell, which you can drop into using the EOS command.
You can also just run EOS commands directly, so you get your standard commands for interacting with file systems,
LS, MKDR and such.
And then there's also a script mode for more efficient workloads you want to run.
You also get your POSIX-like file interface, so you can mount EOS in a number of ways to your system
and just interact with it directly, as you would with your local file system as well.
For Linux this happens through the EOS XT executable, so you get a few smarts with that.
Or alternatively you use the SSHFS tool to get it running locally.
This is also an option for Windows and for macOS, and for Windows there's also support for SAMBA.
You can also thirdly interact with EOS remotely through remote protocols,
HTTP for instance, and then there's also the root protocol.
I won't go into the details in the interest of time, but this is just a protocol tied to the root framework that is used at CERN,
not to be confused with the root user, completely separate, but just know that it exists and that it's particular to the high energy physics community.
Just a little bit of EOS project history.
It has been developed at CERN since just about 2010.
The code base I think is 99% C++, so we really try to squeeze everything we can out of the system.
Each major EOS release gets its own special name after a gemstone, so the most up-to-date one right now is diopside version 5.
So if you actually go into our repositories and check them out, then you can look for these and you know what they mean.
This one is the important slide.
So if you remember only one slide from the EOS part, it's this one.
Here we explain what the architecture looks like, roughly speaking.
So EOS has two major components, a highly available low-latency namespace and then the disk storage instances themselves.
So the namespace is what we call these MGMs, the metadata servers.
They are storing key value pairs in memory using something called QuarkXDB, which is based on RoxxDB.
And the idea here is that we keep them highly available in part by running multiple instances of these QuarkXDB instances in RAF mode.
So for one real full-sized EOS instance on the MGM, you will have three QuarkXDB instances running.
One will be the master.
The two others will ensure that there's a quorum.
So for each operation you do on EOS, there should be at least two of these agreeing that this was actually performed.
And then if something falls over, hopefully the others can take over.
That's the idea.
The other component is the FST.
These are just storage servers with really as many disks as you can fit into them.
I think a rule of thumb is that you can get one petabyte out of one FST.
So essentially these are the workhorses.
They store data.
They transfer data to other sites.
So for the FST is connected to the MGM, your operation usually when you do something, it goes first to the MGM and then the data transfer actually happens to the FST itself.
The replication of the data is handled by EOS.
So EOS ensures that you get the correct number of copies of your data, that it is spread across independent disks and independent instances as desired, so you don't lose anything.
There's also the option to use erasure coding.
Numbers.
What does EOS at certain that look like in numbers?
Well, it's really almost a petabyte, almost, sorry, an exabyte worth of data stored on these EOS instances.
We store roughly speaking 8 billion files in total spread across these 1,300 storage nodes.
We have running, which in turn are running about 60k disks.
We expect this amount of data to grow almost exponentially even in the coming years.
The funding will probably not.
So we really try to make this as efficient as possible and as performant as possible to get really a minus worth.
Here is a more view of what the actual workflow and data access looks like at CERN.
So during LHC operations, you will have a number of data streams coming in all at once as fast as possible, storing the largest amount of data possible.
This is then after the filtering that is mentioned, but even then you get streams totaling up to 150-200 gigabytes per second just coming in to these various EOS instances.
They are by the way split up, so the way we scale them is horizontally using multiple instances.
You will have for instance one EOS instance tied to one larger experiment and then you will have some which are shared among the medium and smaller sized ones.
While this is going on, while the data taking is going on, you will also have data going elsewhere.
You will have about 40-50 gigabytes per second going to tape, which I will get back to in a moment.
So this is for permanent archival for long term storage and also for getting some extra capacity of the, like, just for extra storage.
And then at the same time the data sharing happening to this WLCG, the worldwide collaboration.
On the right hand side you see then the batch workloads.
So these are what physicists queue on the EOS system using HDConder for instance, which is a high throughput computing batch management software.
This is where the actual, the physics data is actually analyzed and we gain insights about whether or not it confirms or weakens theories.
If you're interested now in deploying EOS, good news is you have options.
You don't need to start with the petabyte.
You can, you know, you can scale it and you can move around this, this, yeah, this gradient of how you want to do it exactly.
On the right hand side you have the more mature production grade experience where, you know, the client interacts with your EOS instance through some form of load balancer.
It points it at the correct, the lead MGM, which is running these core DBA instances and the namespace as mentioned.
These in turn connect to the FSTs for the storage.
On the left hand side you have more of a development environment or a test environment.
It's very convenient for that.
So you can deploy it for instance in your Kubernetes cluster or containerized virtual machine, this kind of thing.
You can, you don't need to use a hard, you know, just disk system underneath.
You can also use shared file systems such as Ceph underneath.
And then, yes, in the middle you have the other options such as a hybrid system.
So if you just happen to have a spare Ceph cluster lying around, you can tie it into EOS as well.
You can find more details about that in our various documentation links.
There's also an extended edition of this talk available online.
But before you check that out, let's speak about tape.
So before everyone knew about CERN, now I want to do another one.
Raise your hand if you've used magnetic tape storage.
No.
That's incredible.
Fantastic.
I'll do the intro anyways to magnetic tape storage.
What is it?
Effectively, magnetic tape storage is you store data on magnetizable media,
but instead of doing it on spinning disks like in your HDDs, you do it on this flexible tape.
The tape is coiled up in this cartridge, which is just a plastic shell around mostly with little extra bits and pieces.
And, yeah, to get actually any data in and out of it, you have to put it into this tape drive.
So the cartridge is very simple.
The tape drive by itself doesn't speak to many things except to a tape server.
So this happens over SCSI for us, so we have other options also available.
The end effect is that, well, you get, how do you say this?
No, actually, for just simple setups, you can use just, you know, a tape drive and some cartridges on your desk,
but we ran out of desks a long time ago for that.
So we put these into what are known as tape libraries.
These are, they come in many shapes, but ours are shaped sort of like storage container-like
with the side portions being lined with these slots for the cartridges and for the drives.
And in the middle, you have robots picking up the cartridge physically
and putting it into its corresponding drive when needed.
What does this result in? It results in that you have very different sort of access patterns for magnetic tape versus disk.
Tapes are most happy when you read and write in, you know, one sequence from the start to the end, one smooth motion, ideally.
If you read in another way, you will have to, you know, stop, rewind perhaps, and this will slow things down.
So whereas, disks are good at random access, tape excels at linear.
In the linear case, tape can be good and even beat disk maybe on the writing part.
When it's, the conditions are not right, it can be slower.
So the way you manage your tape has to respect that sort of thing.
Tape is generally used for long-term storage.
Lots of people use it for archival purposes, though we do that as well, and we also use it actively as active storage.
Here for reference is just what a tape library in our case looks like.
Ours is all nice and dressed up.
So you have this long container-sized thing.
Now the explanation of tape is perhaps a bit negative.
Like why would you use tape if it's not necessarily as fast as a disk or if it's this custom format?
Well, at the end of the day, magnetic tape is quite cheap per terabyte as soon as you get over the cost of, you know,
building up your tape library and filling it with contents.
So to get the most out of our money, we do use this for storing lots of data.
Magnetic tape also has low emissions actually at the end of the day because a cartridge, which is on storage somewhere, does not consume power.
Only when it is actually read or written to will it consume power.
So it's a nice little environmental boom there.
And it's also good for cybersecurity.
A cartridge lying in storage is safe.
It can't be overwritten.
It can't be encrypted by ransomware.
So, and there's an actual, there's an actual time cost to moving it to a drive to be written to.
You also have usually way fewer drives than tapes.
So there's a bottleneck for any sort of attacker wanting to do something nasty.
Finally, tapes are long-lasting.
So as mentioned, we keep data in perpetuity effectively.
And so the long shelf life of tape when compared to disk is a positive.
Finally, what is CTA?
CTA stands for the CERN Tape Archive.
It refers both to the physical installation as well as the open source software we produce to run it.
CTA is open source and it isn't designed to be used in conjunction with disk storage.
So the predecessor to CTA was a hybrid system which did both things.
It became fantastically complex.
And so for the second edition, we decided to not do that.
So we leave the disk stuff to the disk system, in this case EOS.
Though you can also, thanks to the efforts from the community, use it with other disk storage systems such as Dcash.
And then CTA concerns itself only with the tape side of things.
So we only store the metadata associated with tape and these things.
So what does CTA do?
It keeps track of the files you have on tape.
It keeps track of where, on which tapes they are.
And it does dequeuing.
So at CERN, at any given time, there will be lots of physicists wanting the data.
There will be contention for the resources in the system.
So CTA handles dequeuing system.
What makes CTA special in contrast to other tape systems?
It is very archive throughput oriented.
So to accommodate this LHC data producing workflow.
So really the focus for CTA is getting data as fast as possible from disk onto tape.
So we give that sort of an advantage in contrast to retrieve operations.
We use it actively.
So not just as an archive where we ideally never retrieve the data because it's just a backup.
We really do get the data back from tape on the regular for the analysis of the data.
And yes, it has a grown user community.
So the CTA software exists.
So their SEO is not maybe not quite as good.
We only need to beat the Chicago Transit Authority and the Cherenkov Telescope Array.
And then we'll be there.
Again numbers.
We have about 750 petabytes of data on tape at CERN.
50 of those are backup and miscellaneous IT data.
The rest is physics.
These are spread across 60,000 roughly speaking tape cartridges.
They are accessed by the 180 to 200 tape drives with the corresponding servers.
These are spread across five libraries on site.
At CTA, at CERN CTA, we use special EOS SSD instances as disk buffer.
So when the data comes in, we really want to have it go as fast as possible.
And we want the data to spend as little time as possible in there.
So we really want the disks to be quick to move the data quickly to tape
and also the other way around.
This is the important part for the CTA, for the CTA slides.
This is what the architecture looks like.
So on the left hand side, you have your data coming in,
usually from the experiment's big EOS instances,
from where it is transferred to our little disk buffer instance.
This, for the user, happens basically by just copying the files to a special location
called something like archive, and then the system takes care of the rest.
The disk buffer through one of the special EOS components
connects to the CTA frontend, which is the management instance,
and queues the archival and retrieve requests.
The frontend connects to the CTA catalog and the object store,
the catalog here being our database, effectively.
We use Oracle, sorry, but also Postgres is supported.
So in there, we keep configuration for the system and the various metadata.
And then in the object store, we keep our queuing information.
This is CEP.
So there, once something is queued, it goes in there.
The tape servers also connect to these two components.
There we have what is known as CTA tape D, the tape demon running.
And the tape servers will check the catalog for their configuration.
They will then check the object store to see if there are any tasks matching their configuration to do,
and then they will execute them.
So either retrieve data or store data through their corresponding drives,
which are in the library.
How do you interact with CTA as a user?
Well, as a user, if you're an administrator, rather,
you will use the CTA admin command line tool.
This comes with the installation.
So there you can do all your low-level operations.
As well, we now try to make a push to publish our higher-level operator tools.
So these are the things you use to manage your tape lifecycle,
the general monitoring and automation of the system.
So these you can then use as well.
And they basically use the CTA admin command in JSON mode under the hood.
Users just interact with CTA through EOS.
So for them, it's just a special place on the disk system
where it takes a bit longer to get your files back.
If you are curious about using CTA, we have a dev setup that we run, virtualized.
You don't need a full tape library.
You can just use the dev setup.
This runs thanks to another lovely open-source project called MHVTL.
This is Mark Harvey's virtual tape library.
It does require a special kernel module to run,
but it's worth it in contrast to having the physical hardware we're needing that.
So through that, you can deploy test instances or very small-scale instances,
just for maybe doing development stuff on either a virtual machine or a Kubernetes cluster.
The instructions for doing that can be found on our documentation.
And with that, I hand back to Hugo to speak about CERNBOX.
Okay, so we are moving from tape now to cloud storage.
Okay, so what is CERNBOX actually?
So CERNBOX is the cloud storage platform basically that we use at CERN
to expose the data stored on EOS and Thf file systems up to the users in a convenient way.
It's actually a global platform.
The background that you see here on my slides actually are basically the locations of everyone basically working at CERN.
CERN is a very distributed place, so you have people contributing from every corner of the globe, almost every corner.
There is no people in the north pole.
And the goal basically was in 2014, so this is the 10th anniversary of this project.
It's also the 70th anniversary of CERN,
and the goal basically was at that time to actually, we saw that many people were using commercial providers
and actually we wanted to have the possibility to control the data where the user were putting it
and actually have our own jurisdiction on it and actually be sovereign about the data.
We generate this data. We should be able to control it.
And this is how everything started.
And we use basically Ancloud, as you probably know.
This is an open source company in Germany.
And we have been collaborating on using their solutions since 2013.
So in a nutshell, today we have around 37,000 users using the system.
On a multi-basis, we have around 10,000 to 12,000 users.
We store around more than 3 billion files actually.
This number has to be updated.
And this system alone for user data, it contains around 20 petabytes of data.
CERN provides four main things for users.
The first is synchronization and sharing.
It's the typical draw box use case.
I just throw some data synchronizing with my devices, share it with people, with public links,
internally in the organization.
And actually it brings a use case, which is to give people access to the data offline.
If I have the data synchronized, I just close my network connection, I go in the plane,
I can work on my data.
When I arrive to a place where I have internet back, I basically, I can just get it in automatically
without having to manually upload the data.
Another thing that we bring is actually web applications.
So, Office kind of collaboration suite.
We use only Office, Collaborator, Microsoft.
So, we have a plethora of different applications.
And also, scientific applications.
So, some of the scientific applications for physicists, they're very complex to install on the laptop.
So, we basically give them a web environment, usually through JupyterLab kind of platforms.
And this facilitates basically the daily job of people,
especially students that they're at the university, they want to do something
and they don't want to spend time setting up complicated C++ tool changes
to basically just build the software they have to use.
Another aspect is that this service provides access to the underlying systems
through an online file system.
On Linux, we use the Fuse Layer library and Windows through Samba.
And then we integrated with the specific physics protocols and software
and computing farms.
So, basically from CERNBOX, imagine the typical use case of a physicist.
I have a little data set.
I play with it in my laptop.
Basically, from my laptop, I do some analysis.
This analysis, it's basically living in the same storage where the computing farm is.
So, with one command, I can say, OK, now scale out this job.
This job will run the computing farm for hours, days, weeks.
When the job is done, the results will automatically synchronize to my device.
So, I can actually write a nice paper.
So, this is the whole workflow that we're trying to optimize with this platform
because people before were having to learn different tools,
having to pull the data when they were available, etc.
So, we really simplify the use cases for the scientist.
Now, what makes this platform so special across these last 10 years
are basically three things.
I want to explain to you here, and maybe if I have some time, I will do a live demo.
I'll do it right.
First thing is that the data is not owned by a system user.
You know, you use a platform like Ancload or NestCloud.
All the data is owned by the system users they are running with, usually Apache or Nginx.
And we didn't want this because we have a problem.
This means that all the data is compromised.
So, all the user data basically is owned by the user.
So, we have a huge basically held up directory.
Every user has a dedicated UID and group ID.
And when we start this data, we really start it as the user ID.
What this gives to the users is the possibility that they can access the data from a web interface,
but also through the FuseLayer.
It's the same UID that is being used from web and the file system.
And this kind of magic is actually a kind of challenge for institutions right now
because people are using web users.
You have nice OAuth, basically UPNs, UIDs.
But these don't reflect basically the unique side.
These are difficult to match them basically.
So, this is one of the nice things about the system.
The other one is that traditionally, and this is something that Ancload has actually moved away from
after many years.
They use a new model basically that will explain their new product.
But since years, you are using basically this platform.
It was written in PHP and all the metadata was stored in a big SQL database.
And the problem was that if someone was accessing the file system on the back of this platform,
people using the web part will not see the data.
They will have to run some synchronization jobs that will take hours
to basically spawn the new data in the web interface
because its identity was a synchronization between the SQL database and the storage.
And you have two brains that think several different things, so you have conflicts as well.
So, what we did is remove the database completely
and all the operations were joined directly to the file system.
So, for example, in Sanbos, we don't store anything on the database.
Everything is stored on the storage system.
We use standard attributes as much as we can to basically facilitate the operations.
This gives us atomicity as well when we have to move files left and right,
create versions, etc.
The standard attributes that live with the data file.
And this facilitates a lot basically operations.
And now, Unclo with the new product, which is Unclo's infinite scale.
They are actually also doing this.
So, this is a way to step forward for scalability.
And the third feature of this system is actually that from the web interface,
you just set an ACL.
Like, I want to give access to my colleague, Elvin, you know,
read, write access to the folder called red.
And what happens in the back is that also on the storage file system,
there is an ACL set.
In EOS, we use basically EOS ACLs.
On Thephaphase, we use basically the Unix ACLs basically to give access
based on UID and GroupID.
And what this gives is that people from the web, they can access the data in Dropbox.
And people from the, like, you know, Gix, usually Linux computing clusters,
they can just basically CD into the places that they have access.
And these ACLs are respected.
And this basically combines the web world and the storage system world
in a way that people, independent, whatever they are, which device they use,
they always have access to the same data.
And this is a little bit how it looks like the infrastructure behind.
On the left side, we use the Unclo's infinite scale web clients.
This is a single page application running on top of NG Nexa.
So this is the web UI.
Nice people just go there and use it.
We also use the Unclo application for desktop and mobile people that want to
access synchronized data from their laptops or from mobile devices like iOS and Android.
They just use these applications.
And then we run what we call the Riva server.
So the story behind this project is that in 2017, until 2017,
we were running PHP Unclo.
And at that point, it didn't scale out for our needs.
We were storing a lot of files.
And PHP, I'm sorry, there were a lot of PHP fanatics,
but it was not scaling as we wanted.
So I re-broad the server, basically, of Unclo in Go.
And this language was really focused on system performance and concurrency.
And this is what we needed at that time.
And then three years later, in 2020, Unclo's use actually took this component
and actually is part of their new product.
So we are pretty proud that they did this,
because now everyone can profit basically from this integration.
And how this server integrates with EOS and CFFS.
For EOS, we use GRPC.
I'll show you now, GRPC from Google, the open source project.
Basically, client-server communication to EOS.
EOS supports an GRPC server that we can communicate with protocol buffers.
And also with the Xroot protocol, which is a protocol really for wide area networks,
which for high latency.
So it's a protocol that is optimized for that.
And for CFFS, there is a nice binding for Go.
So we use the live CFFS library basically from the server.
We create virtual mounts inside the process, and then we access the information
on behalf of the user that comes to talk to basically to the server.
And yeah, here you have some documentation.
Basically, we store everything on GitHub, open source.
There are also some publications around this software,
so you can really look like, basically, dig, dive into it more.
You can really refer to that.
And yeah, we are downstairs in the booth basically.
So feel free to talk to us, because some of the systems are actually better to show you live,
with common line, how it looks like, the rather gentle slides.
And we are organizing a specific tech-week storage at CERN.
It's basically available for everyone.
So as you are passing by Geneva, or you are interested into these topics,
please come. It's free.
And yeah, it's a place basically we would discuss with people having different challenges
about the storage.
Yeah, and that's pretty much it.
CERN basically is open all the year round, so feel free to come and visit us.
If you are a student, basically, still studying some master's degrees.
There are nice opportunities to come to CERN.
It's how I came actually doing some internship.
I liked what I did.
I stayed there for a couple of years, then I left, and I came back.
So it's a nice opportunity.
It's public money founded by many countries basically that really believe in science.
So feel free to profit from that, and that's it.
You have time for questions?
Yeah.
You pick.
Do you actually have external EOS installations?
So the question is, do we have external EOS installations?
Yes, we have plenty of them.
I don't know all the places where they are,
but I can tell you, for example, in Asia there are some institutions.
Actually, yes, maybe we have a map.
Yeah.
Some of them.
So these are the ones that have EOS and CTA, the tape archive,
but there are other places that have only EOS deployed.
And if you want more information, just pass by the booth later.
I can give you more details.
How much redundancy is in the CTA?
It depends on how much you want to configure.
The default is for the major experiments, you get one copy on tape,
and then usually through the WSG there will be a collaborating institution
that mirrors the data locally there on site for them.
For small or medium sized experiments who don't have that kind of setup,
we do do do will copy.
So then you will get your files and at least we will guarantee you that you will get
two copies and you could in theory set it to something else as well,
but usually two is fine.
We try to make sure and you can also configure this that the files
actually end up in separate tape libraries.
So that way you know if for some reason you know your building burns down
or just that particular library gets damaged,
hopefully you'll have it somewhere else and be safe there.
Thanks.
And on the topic of disk sizes, what do you use and how do you feel about
the upcoming 20, 30, 40 terabyte individual disks that are you going to use?
That is, do you want to take this one?
Yeah, I can take it.
So the question is basically what do we think about the new basically high density disks
that are coming to the market and how we plan to use them.
So depending on the use case for the physics use case, we really don't care
because we have so many data that we have to ingest that even if they are high density
it really doesn't make a huge impact.
For the sandbox part of the project where we only have 20 petabytes
with the new high density disks, this means that we could actually have just one rack
with all the data.
And this is not ideal for redundancy because then basically we are,
you know, our single point of failure is the rack.
And what we have in mind basically is to try to basically use the same data
that we have in the past.
And basically is to try to basically erase encoding basically the files.
A US is a system that you can use replica based model or erasure encoding.
And to find basically a good trade off, there is really not like a perfect solution
but the idea is to find the correct erasure encoding across different racks
and maybe even shared the storage server with other projects or other use cases
so we can benefit basically to really fill the disks.
And another use case that we have currently that we are investigating is the single-magnetic disk also
what basically will have the impact to use them
because it looks like the industry moving to the direction for single-magnetic recording.
And yeah, it poses a challenge because this disk usually happens only
and we have basically use cases where you have to handle random writes and random reads around
and this can basically pose some challenges.
You were saying that you have you're storing data indefinitely
and the tapes have a duration life of 30 years.
How soon is older than 30 years?
So are you having this operation of having to copy all tapes
as is how big is that challenge for you?
Yes, so the question is if the lifespan of a magnetic tape is 30 years about
and soon is older than that, how do you keep this data around?
And for that we have continuous workflows in place.
So this is part of these operator tools that I mentioned.
There's one called REPAC.
So effectively we have automation in place for periodically.
Once we get to a new generation of tape media to take the old ones
and then rewrite that data onto new media.
So that way we always sort of stay on the wave of new technology
or at least you know trading slightly behind it.
We can't of course upgrade everything all at once
because that's just way too much.
There's an actual amount of time spent in reading from the end of one tape to the other.
But yes, we continuously upgrade our media generation.
So I think the oldest right now is LTO7 on site
and we're moving slowly now to, for new data we're using LTO9, the newest enterprise.
And so yes, continuously we will rewrite data onto new media.
This also happens if media gets damaged for some reason.
You guys also use decentralized cloud solutions.
Is that considered a valid option in your stack for archival
or is that like a proof of concept?
For archival specifically.
So the question was do we use decentralized cloud storage?
Was that correct?
File point.
Come again?
File point.
Oh sorry, file point.
Ah sorry.
I don't have the numbers for any cost analysis on that
but I don't think so.
It's going to be, I don't know how you can guarantee with,
I'm ignorant about fine-cones perspective
so I don't know if you can guarantee a specific timeframe
for the retrieval of the data or for the access.
There's also, at the end of the day,
so once it is retrieved onto disk,
we really, well, we need to put it on disk for working with it
and the latency needs to be low.
Like at some point we had another data center
called Wigner off-site, far away.
And just even for that the latency for the physics workflow
was way too long.
Like it's just, it wasn't workable.
We have received many complaints so yes,
it is nicer to have it on-site.
Yeah, though come stop by and speak about file coin.
Backplace has a lot of disk statistics that they publish every year.
So you have it as well?
Yeah, I can answer which one specifically for reliability or...
What disks?
They have half a kilo of it.
Yeah, so the question is basically that back-place probably
is basically some information,
insights about the reliability of disk.
Yes, we do.
There are some papers around.
I cannot refer now to my head,
but we can pass by, we can find them.
These are failing every day at CERN.
Every single day we have disks that are broken.
We have so many that is natural.
And what we have built is basically just to make sure that
when this fails that basically the software can take care
to make sure that there is another replica valid on top.
But I cannot give you the exact numbers,
but we can figure it out.
Short question, 50 seconds.
Okay.
Maybe I'll do that.
Test the guys at the booth.
You can ask as many questions as you want, I'm sure.
We are all the weekend around so feel free.
Yeah, so let's give them a round of applause.
Thank you.