[00:00.000 --> 00:15.720]  So, as a start, oh yes, I put who am I, so I'm Rodrigo Solio, so I'm FreeBSD power
[00:15.720 --> 00:28.880]  committer since 2014, what else, so yes, what speak it provides, I don't know if how
[00:28.880 --> 00:40.640]  many of you knows package provides, a few ones, okay, so sorry, oh yes, so this part
[00:40.640 --> 00:47.760]  of, so as you know, we have package, PKG, which is a FreeBSD command used to
[00:47.760 --> 00:53.880]  manage packages, and the package was designed to be extended with plugins, so
[00:53.880 --> 01:01.520]  you can route your own plugins to run special commands, handle hooks, replace
[01:01.520 --> 01:08.080]  existing commands, or access the package database just for statistics or have
[01:08.080 --> 01:14.520]  numbers or whatever, so package provides basically is a plugin you can use with
[01:14.520 --> 01:22.560]  packages, we extend package to do a special thing is matching a found the
[01:22.560 --> 01:29.440]  packages for a specific file, basically you have you are looking for which
[01:29.440 --> 01:37.120]  package I need to install to have this file in my computer, and usually is you
[01:37.120 --> 01:42.840]  you start looking on the internet, you Google a little bit to know exactly what
[01:42.840 --> 01:48.520]  the package you need, and because sometimes the packages doesn't provide the
[01:48.520 --> 01:55.880]  name of the files we have inside the command or so on, so basically it's a
[01:55.880 --> 02:01.760]  way to found, is a reverse way to find which package correspond to a specific
[02:01.760 --> 02:12.000]  file, it could be a library, an executable, or just a header like hfile, so this is an
[02:12.000 --> 02:19.400]  example, so you just start package provides and the libmpg123.iso and
[02:19.400 --> 02:28.080]  give you the list of packages who provide this, sorry for the contrast, it's not
[02:28.080 --> 02:34.000]  very good, so you have a you have a for this lib, you have a two
[02:34.000 --> 02:40.320]  far two packages, we have the mpg123 which is probably the main project who
[02:40.320 --> 02:45.440]  provides the far, but you see that zoom video conferencing also provide this
[02:45.440 --> 02:52.120]  library in another version probably for your own and basically it's a Linux
[02:52.120 --> 03:03.840]  compact library, so as you can guess package provide is a client, so you have
[03:03.840 --> 03:11.840]  in your computer but it's also a database who contains the full list of files you
[03:11.840 --> 03:18.120]  can install if you install all the packages existing in FreeBSD and this
[03:18.120 --> 03:26.680]  is a lot for just one release and one architectures we have one nine million
[03:26.680 --> 03:36.320]  files installable, so this is a huge database we have to build, so just I
[03:36.320 --> 03:45.240]  put here this is the numbers for the the latest version so the 13 and if you
[03:45.240 --> 03:51.160]  look at the different versions of FreeBSD and different architectures you see
[03:51.160 --> 03:58.000]  how many how many files are and basically this is based on the packages
[03:58.000 --> 04:04.600]  you can build on FreeBSD for this architecture and version, so you you
[04:04.600 --> 04:10.680]  can probably see which are the the good architectures, the architectures who
[04:10.680 --> 04:16.720]  works and you have a lot of applications and those architectures you who are a
[04:16.720 --> 04:24.000]  little bit not really used or we are missing a lot of tools who
[04:24.000 --> 04:30.880]  doesn't build for those architectures, you have also the new ones like the
[04:30.880 --> 04:37.440]  ARMv7 who appears suddenly and it has a big big number and you have the
[04:37.440 --> 04:46.800]  ARMv6 who is declining a little bit, things still works but as the
[04:46.800 --> 04:58.520]  numbers shows it's not that popular than ARMv7, so this is
[04:58.520 --> 05:08.320]  basically how everything works, I have here because I host the database server
[05:08.320 --> 05:14.840]  myself, so I have the the package provide server here, you have the clients who
[05:14.840 --> 05:24.240]  connects to upload the database and I use the package FreeBSD servers to sync
[05:24.240 --> 05:30.600]  and have the list of files because this is the the best and most efficient way
[05:30.600 --> 05:36.920]  to have the full list of files in a minimum of time because of course you
[05:36.920 --> 05:41.880]  you can guess it's it's not always possible to build all the packages just
[05:41.880 --> 05:50.320]  to have the list, it's kind of nightmare, so basically yes I put some
[05:50.320 --> 05:59.360]  sometimes about these three so I start it on 2017, I have multiple motivations, the
[05:59.360 --> 06:06.760]  first one was to stop doing grep on ports when I'm looking for a file, it's
[06:06.760 --> 06:13.960]  nothing more terrible than you try to compile a new a new project and say
[06:13.960 --> 06:20.160]  oh I'm missing the something H file and you start draping around and it takes
[06:20.160 --> 06:29.520]  hours just to have the full list of dependencies, so this way you can
[06:29.520 --> 06:36.760]  found it easily and have the exact number and also you can have a full
[06:36.760 --> 06:43.040]  list of the packages providing the same file so you can choose the
[06:43.040 --> 06:48.240]  right one who match for you, usually it's not the it's not the first one
[06:48.240 --> 06:54.600]  because some packages bring a lot of dependencies so you can choose
[06:54.600 --> 07:03.720]  the right one and my second motivation was to write a package plugin, at this
[07:03.720 --> 07:12.080]  time package just was a new thing, we have this plugin feature and as much
[07:12.080 --> 07:19.240]  as you know I'm probably the first plugin available for PkG and probably
[07:19.240 --> 07:29.600]  one of the who works, who is still working, I have real users but yes so
[07:29.600 --> 07:36.200]  yes PkG provide was introduced at the FOSDEM, the one year late one year
[07:36.200 --> 07:47.120]  after in 2018 and this is where the story starts, this is the daily connections
[07:47.120 --> 07:56.160]  of people who access the server to upload the database, looks not that much
[07:56.160 --> 08:02.200]  but I consider package provide like a consider package provide a developer
[08:02.200 --> 08:08.880]  tour so I don't expect regular people use it that much because it doesn't too
[08:08.880 --> 08:18.080]  much sense and also we have only people access only when they perform a
[08:18.080 --> 08:23.880]  package update, I don't upload the database every time so when you are doing
[08:23.880 --> 08:30.000]  an update, you have updates at this time the plugin looks if we have a
[08:30.000 --> 08:39.960]  new database so it downloads it, it's right to be oh so oh yes yes this is
[08:39.960 --> 08:44.880]  what happens when you host your own server from time to time you have a
[08:44.880 --> 08:55.160]  breakages and in this case was some kind of certificate issues, some
[08:55.160 --> 09:01.200]  numbers, the question is from where people come from and as you can see it
[09:01.200 --> 09:10.400]  matches a lot the free BSD developers base so you have people mostly USA,
[09:10.400 --> 09:16.640]  Canada, some people in South America, mostly in Europe, we have some people
[09:16.640 --> 09:23.280]  Asia, Australia and unfortunately we now have too much people born Africa who
[09:23.280 --> 09:31.800]  don't use people not like BSD there so if you take the numbers yes we have the
[09:31.800 --> 09:38.360]  big chunk of users are in the United States and you have Germany and
[09:38.360 --> 09:47.640]  France we are quite the same numbers and yes and after that you have a lot of
[09:47.640 --> 09:55.400]  countries around the world sometimes yes you have what's funny is we have a
[09:55.400 --> 10:02.240]  Russian Federation and Ukraine and both have the same number I have the same
[10:02.240 --> 10:08.400]  number of people coming from both countries so strange and from time to
[10:08.400 --> 10:16.560]  time you have some special guests so I for a couple of months I have people
[10:16.560 --> 10:24.920]  from St. Kitties and Nevis and if you don't know St. Kitties is a small island
[10:24.920 --> 10:31.200]  here just close to Cuba and it's basically just a place to put
[10:31.200 --> 10:38.920]  money and register your companies when you have things to hide so okay we have
[10:38.920 --> 10:50.640]  developers there just yes a quick look about the database as I say the database
[10:50.640 --> 11:01.120]  so we have at this time we have 58,000 of ports if you consider we have a
[11:01.120 --> 11:06.880]  three major releases and say architectures that brings you a lot of
[11:06.880 --> 11:13.960]  ports so as I say it's something you can build by your own because I don't have
[11:13.960 --> 11:22.600]  an inner space or CPU also to obtain the list of files it's not possible to
[11:22.600 --> 11:29.800]  trust PQGP list because those files are not totally complete sometimes you have
[11:29.800 --> 11:39.080]  replacements who don't during the build and I can just download as is the
[11:39.080 --> 11:44.960]  packages because it needs time and space to download the full list of packages
[11:44.960 --> 11:53.080]  and also have impact on mirror performances so I found a simple way
[11:53.080 --> 12:01.720]  is I try to abuse the package file format as you know packages are just
[12:01.720 --> 12:10.280]  archived who contains two times the manifest of the package and the
[12:10.280 --> 12:15.640]  second manifest which is a full manifest has the list of files provided for
[12:15.640 --> 12:24.480]  the package then comes all the files and the thing is you can just use the head
[12:24.480 --> 12:34.160]  HTTP request to only download the first part of the package and let's say we took
[12:34.160 --> 12:40.920]  we can extract the manifest because it works and if the JSON file is valid you
[12:40.920 --> 12:46.880]  are good if you if you have an error you just increase a little bit of size
[12:46.880 --> 12:54.360]  just remember it for the next time so this way you can just download the
[12:54.360 --> 13:01.080]  only the manifest part of the files and the good thing is the HTTP head is a
[13:01.080 --> 13:07.680]  totally standard method so we are not trying to abuse the HTTP server and we
[13:07.680 --> 13:13.200]  are not producing or breakages in connections we are saving people band
[13:13.200 --> 13:20.720]  with because we are we are not we are not using the the package to mirror just
[13:20.720 --> 13:31.040]  for us and we save also everybody's time yes so talking about the database itself
[13:31.040 --> 13:40.240]  at the beginning I I designed it as a just a four text file just because it's
[13:40.240 --> 13:46.920]  easy to read and search but it I have I have this problem is was too large is
[13:46.920 --> 14:01.440]  was 36 megabytes even compress okay so at the in 2018 at the USB beacon I talked
[14:01.440 --> 14:07.800]  with talking with mark SP about the the thing he suggests me I can use the
[14:07.800 --> 14:17.600]  locate file format locate is a UNIX standard tool who store all the is try
[14:17.600 --> 14:21.440]  to sort all the files you have in your computer so it's easy to found the place
[14:21.440 --> 14:31.920]  and use a Nagarin called Bgram to store the the data and this this format is
[14:31.920 --> 14:39.840]  pre-efficient because if we once once are generated and compress it I reduce by
[14:39.840 --> 14:47.040]  two to not do the size of the file so it's it's it's pretty convenient because
[14:47.040 --> 14:52.440]  instead of downloading for a couple of minutes the database we can have a just
[14:52.440 --> 14:57.320]  in one one minute and something it's pretty similar with the package itself
[14:57.320 --> 15:06.400]  so in time so the package provide us ages as I saw yes I show you you can just
[15:06.400 --> 15:12.280]  perform reverse searches you're looking for a file give you the name of the
[15:12.280 --> 15:20.040]  package but you can also use it to identify unexpected the files in the
[15:20.040 --> 15:26.440]  packages like core dumps things happen sometimes people commit and then the
[15:26.440 --> 15:36.280]  commit comes with binary files or dumps because of it happens and also you can
[15:36.280 --> 15:41.320]  use it to search for duplicate files when you have two packages installed the
[15:41.320 --> 15:49.440]  same far and conflicting for a long time I report this this information to to the
[15:49.440 --> 15:58.520]  developers and I I try to track exactly how much conflict we have and someone
[15:58.520 --> 16:04.840]  decide or Stefan Stefan is there decides to start looking at it and use the
[16:04.840 --> 16:11.360]  database the database I provide to to search for file conflicts and try to
[16:11.360 --> 16:20.120]  fix it to to to to avoid conflicts during a package install because package the
[16:20.120 --> 16:26.720]  tool can handle conflicts but the handle the conflicts the worst way it say okay
[16:26.720 --> 16:31.080]  you have a conflict so or you can't install the new package or you have to
[16:31.080 --> 16:37.120]  remove the old one so there's no way to and sometimes you you have and that
[16:37.120 --> 16:42.800]  there's a lot of funny things because we have binaries who have to say exactly
[16:42.800 --> 16:48.040]  the same name doing completely different things sometimes the game and the
[16:48.040 --> 17:02.480]  network manager and wow so yes so yeah and I still having six minutes so if you
[17:02.480 --> 17:20.720]  have questions please yeah the options no I am what I take is the real files
[17:20.720 --> 17:31.000]  available in in package servers so is the real thing you have if you install a
[17:31.000 --> 17:39.320]  package it's not something I'm sorry the question was if I scan for options and
[17:39.320 --> 17:48.440]  the answer is no I just try to track what's on the package servers and the
[17:48.440 --> 17:58.000]  real files install if you install it's not it's not use the custom if you have
[17:58.000 --> 18:03.200]  a custom package server it doesn't bring you those are the files you are
[18:03.200 --> 18:11.480]  installing yes it's just for standard insta-free bsd packages the way we build
[18:11.480 --> 18:25.440]  it yeah oh the first okay how long is to how long is take to scan the packages
[18:25.440 --> 18:34.440]  and build the index hello so the first the first con it can take about I think
[18:34.440 --> 18:41.880]  is I saw I didn't do for a long time but I think it's like half an hour 30 minutes
[18:41.880 --> 18:48.840]  something like that and once once I build it I can I can just update in a
[18:48.840 --> 18:55.920]  couple of seconds from one minute so it's that the the good thing is if I
[18:55.920 --> 19:04.040]  consider because I we have a file a manifesto with all the packages on the
[19:04.040 --> 19:09.720]  version so I can see which package change so I because I just look at the
[19:09.720 --> 19:16.920]  package who change a version and based on that and just recovering the the
[19:16.920 --> 19:22.800]  manifesto it takes yes it takes a cup I think of the maximum is like a five
[19:22.800 --> 19:33.440]  minutes for for for an architecture and the version yes
[19:34.640 --> 19:44.040]  the database I use the locate so it's the yes is the locate format so because
[19:44.040 --> 20:03.840]  because it's really is really convenient for this kind of usage yes yes I
[20:03.840 --> 20:25.680]  I store it that and yeah yes I am I include that because of the the the way
[20:25.680 --> 20:34.920]  big Ram works it takes just a line and just a line is it is a path so it's
[20:34.920 --> 20:39.720]  considered path and there's no extra data around so you can you can use so I
[20:39.720 --> 20:45.480]  just include the the the package name inside inside the path and use a
[20:45.480 --> 20:54.000]  special character yes also separator I in this case I use the the star character
[20:54.000 --> 21:02.080]  because I don't expect people to put a put a star in the package name yes
[21:02.080 --> 21:12.720]  absolutely and and I use I use a regular expression for search but the result is
[21:12.720 --> 21:26.400]  it's quite acceptable invites compress it it's 17 exact compress it 17 megabytes
[21:26.400 --> 21:32.600]  yes around because it depends on which version you're looking at but it's
[21:32.600 --> 21:49.520]  basically yes it's around this size yes could be could be yeah now sorry the
[21:49.520 --> 21:54.840]  question aha this was a trick just to see if I can repeat yes the question was
[21:54.840 --> 22:01.560]  was why not use a slash instead of star because a slash a slash made more sense
[22:01.560 --> 22:14.080]  probably yes so but I I am yes we can we can probably
[22:14.080 --> 22:29.200]  yeah oh good question no I just see happy people