So, as a start, oh yes, I put who am I, so I'm Rodrigo Solio, so I'm FreeBSD power
committer since 2014, what else, so yes, what speak it provides, I don't know if how
many of you knows package provides, a few ones, okay, so sorry, oh yes, so this part
of, so as you know, we have package, PKG, which is a FreeBSD command used to
manage packages, and the package was designed to be extended with plugins, so
you can route your own plugins to run special commands, handle hooks, replace
existing commands, or access the package database just for statistics or have
numbers or whatever, so package provides basically is a plugin you can use with
packages, we extend package to do a special thing is matching a found the
packages for a specific file, basically you have you are looking for which
package I need to install to have this file in my computer, and usually is you
you start looking on the internet, you Google a little bit to know exactly what
the package you need, and because sometimes the packages doesn't provide the
name of the files we have inside the command or so on, so basically it's a
way to found, is a reverse way to find which package correspond to a specific
file, it could be a library, an executable, or just a header like hfile, so this is an
example, so you just start package provides and the libmpg123.iso and
give you the list of packages who provide this, sorry for the contrast, it's not
very good, so you have a you have a for this lib, you have a two
far two packages, we have the mpg123 which is probably the main project who
provides the far, but you see that zoom video conferencing also provide this
library in another version probably for your own and basically it's a Linux
compact library, so as you can guess package provide is a client, so you have
in your computer but it's also a database who contains the full list of files you
can install if you install all the packages existing in FreeBSD and this
is a lot for just one release and one architectures we have one nine million
files installable, so this is a huge database we have to build, so just I
put here this is the numbers for the the latest version so the 13 and if you
look at the different versions of FreeBSD and different architectures you see
how many how many files are and basically this is based on the packages
you can build on FreeBSD for this architecture and version, so you you
can probably see which are the the good architectures, the architectures who
works and you have a lot of applications and those architectures you who are a
little bit not really used or we are missing a lot of tools who
doesn't build for those architectures, you have also the new ones like the
ARMv7 who appears suddenly and it has a big big number and you have the
ARMv6 who is declining a little bit, things still works but as the
numbers shows it's not that popular than ARMv7, so this is
basically how everything works, I have here because I host the database server
myself, so I have the the package provide server here, you have the clients who
connects to upload the database and I use the package FreeBSD servers to sync
and have the list of files because this is the the best and most efficient way
to have the full list of files in a minimum of time because of course you
you can guess it's it's not always possible to build all the packages just
to have the list, it's kind of nightmare, so basically yes I put some
sometimes about these three so I start it on 2017, I have multiple motivations, the
first one was to stop doing grep on ports when I'm looking for a file, it's
nothing more terrible than you try to compile a new a new project and say
oh I'm missing the something H file and you start draping around and it takes
hours just to have the full list of dependencies, so this way you can
found it easily and have the exact number and also you can have a full
list of the packages providing the same file so you can choose the
right one who match for you, usually it's not the it's not the first one
because some packages bring a lot of dependencies so you can choose
the right one and my second motivation was to write a package plugin, at this
time package just was a new thing, we have this plugin feature and as much
as you know I'm probably the first plugin available for PkG and probably
one of the who works, who is still working, I have real users but yes so
yes PkG provide was introduced at the FOSDEM, the one year late one year
after in 2018 and this is where the story starts, this is the daily connections
of people who access the server to upload the database, looks not that much
but I consider package provide like a consider package provide a developer
tour so I don't expect regular people use it that much because it doesn't too
much sense and also we have only people access only when they perform a
package update, I don't upload the database every time so when you are doing
an update, you have updates at this time the plugin looks if we have a
new database so it downloads it, it's right to be oh so oh yes yes this is
what happens when you host your own server from time to time you have a
breakages and in this case was some kind of certificate issues, some
numbers, the question is from where people come from and as you can see it
matches a lot the free BSD developers base so you have people mostly USA,
Canada, some people in South America, mostly in Europe, we have some people
Asia, Australia and unfortunately we now have too much people born Africa who
don't use people not like BSD there so if you take the numbers yes we have the
big chunk of users are in the United States and you have Germany and
France we are quite the same numbers and yes and after that you have a lot of
countries around the world sometimes yes you have what's funny is we have a
Russian Federation and Ukraine and both have the same number I have the same
number of people coming from both countries so strange and from time to
time you have some special guests so I for a couple of months I have people
from St. Kitties and Nevis and if you don't know St. Kitties is a small island
here just close to Cuba and it's basically just a place to put
money and register your companies when you have things to hide so okay we have
developers there just yes a quick look about the database as I say the database
so we have at this time we have 58,000 of ports if you consider we have a
three major releases and say architectures that brings you a lot of
ports so as I say it's something you can build by your own because I don't have
an inner space or CPU also to obtain the list of files it's not possible to
trust PQGP list because those files are not totally complete sometimes you have
replacements who don't during the build and I can just download as is the
packages because it needs time and space to download the full list of packages
and also have impact on mirror performances so I found a simple way
is I try to abuse the package file format as you know packages are just
archived who contains two times the manifest of the package and the
second manifest which is a full manifest has the list of files provided for
the package then comes all the files and the thing is you can just use the head
HTTP request to only download the first part of the package and let's say we took
we can extract the manifest because it works and if the JSON file is valid you
are good if you if you have an error you just increase a little bit of size
just remember it for the next time so this way you can just download the
only the manifest part of the files and the good thing is the HTTP head is a
totally standard method so we are not trying to abuse the HTTP server and we
are not producing or breakages in connections we are saving people band
with because we are we are not we are not using the the package to mirror just
for us and we save also everybody's time yes so talking about the database itself
at the beginning I I designed it as a just a four text file just because it's
easy to read and search but it I have I have this problem is was too large is
was 36 megabytes even compress okay so at the in 2018 at the USB beacon I talked
with talking with mark SP about the the thing he suggests me I can use the
locate file format locate is a UNIX standard tool who store all the is try
to sort all the files you have in your computer so it's easy to found the place
and use a Nagarin called Bgram to store the the data and this this format is
pre-efficient because if we once once are generated and compress it I reduce by
two to not do the size of the file so it's it's it's pretty convenient because
instead of downloading for a couple of minutes the database we can have a just
in one one minute and something it's pretty similar with the package itself
so in time so the package provide us ages as I saw yes I show you you can just
perform reverse searches you're looking for a file give you the name of the
package but you can also use it to identify unexpected the files in the
packages like core dumps things happen sometimes people commit and then the
commit comes with binary files or dumps because of it happens and also you can
use it to search for duplicate files when you have two packages installed the
same far and conflicting for a long time I report this this information to to the
developers and I I try to track exactly how much conflict we have and someone
decide or Stefan Stefan is there decides to start looking at it and use the
database the database I provide to to search for file conflicts and try to
fix it to to to to avoid conflicts during a package install because package the
tool can handle conflicts but the handle the conflicts the worst way it say okay
you have a conflict so or you can't install the new package or you have to
remove the old one so there's no way to and sometimes you you have and that
there's a lot of funny things because we have binaries who have to say exactly
the same name doing completely different things sometimes the game and the
network manager and wow so yes so yeah and I still having six minutes so if you
have questions please yeah the options no I am what I take is the real files
available in in package servers so is the real thing you have if you install a
package it's not something I'm sorry the question was if I scan for options and
the answer is no I just try to track what's on the package servers and the
real files install if you install it's not it's not use the custom if you have
a custom package server it doesn't bring you those are the files you are
installing yes it's just for standard insta-free bsd packages the way we build
it yeah oh the first okay how long is to how long is take to scan the packages
and build the index hello so the first the first con it can take about I think
is I saw I didn't do for a long time but I think it's like half an hour 30 minutes
something like that and once once I build it I can I can just update in a
couple of seconds from one minute so it's that the the good thing is if I
consider because I we have a file a manifesto with all the packages on the
version so I can see which package change so I because I just look at the
package who change a version and based on that and just recovering the the
manifesto it takes yes it takes a cup I think of the maximum is like a five
minutes for for for an architecture and the version yes
the database I use the locate so it's the yes is the locate format so because
because it's really is really convenient for this kind of usage yes yes I
I store it that and yeah yes I am I include that because of the the the way
big Ram works it takes just a line and just a line is it is a path so it's
considered path and there's no extra data around so you can you can use so I
just include the the the package name inside inside the path and use a
special character yes also separator I in this case I use the the star character
because I don't expect people to put a put a star in the package name yes
absolutely and and I use I use a regular expression for search but the result is
it's quite acceptable invites compress it it's 17 exact compress it 17 megabytes
yes around because it depends on which version you're looking at but it's
basically yes it's around this size yes could be could be yeah now sorry the
question aha this was a trick just to see if I can repeat yes the question was
was why not use a slash instead of star because a slash a slash made more sense
probably yes so but I I am yes we can we can probably
yeah oh good question no I just see happy people