[00:00.000 --> 00:12.640]  So this is a presentation, we're starting now actually a new set of presentations in
[00:12.640 --> 00:19.440]  the dev room, not about completely S-bombs but about information that gets inside S-bombs
[00:19.440 --> 00:21.640]  and you'll be hearing more about it.
[00:21.640 --> 00:28.440]  So this is a presentation about a system to capture detailed information about building
[00:28.440 --> 00:36.960]  and the work, I mean almost all of the work happened by thirties and so I was just, yeah,
[00:36.960 --> 00:41.480]  you know, explaining what has to be done, deciding the problem.
[00:41.480 --> 00:48.600]  So we'll be going through a very typical agenda, you know, what we're trying to solve,
[00:48.600 --> 00:55.640]  how we solve it, how we solve it and what was not solved and what we're going to do.
[00:55.640 --> 01:02.920]  This happened as a GSOC project, Google Summer of Project last year of the GIFOS organization
[01:02.920 --> 01:09.000]  which is an umbrella organization for open source in Greece and we have representatives
[01:09.000 --> 01:10.000]  there.
[01:10.000 --> 01:15.200]  Right, I'm not going to be reading the whole slide, you can read it and you can see it there.
[01:15.200 --> 01:20.680]  But they're an umbrella organization that they're looking over open source development
[01:20.680 --> 01:30.320]  in Greece and they are, have been participated to GSOC for many years now and one of the
[01:30.320 --> 01:34.400]  project was the wonderful that Thottis did, right.
[01:34.400 --> 01:41.640]  So let's start with the basics, what are we trying to solve, right?
[01:41.640 --> 01:45.640]  There are different names for what we're trying to solve, some people call it origin, some
[01:45.640 --> 01:52.000]  call it providence or pedigree or things, essentially if we have a binary file and I'm
[01:52.000 --> 01:58.440]  talking when I say binary I mean an executable or a library, not an image, right, or a PDF
[01:58.440 --> 02:06.840]  but an object file, right, how was this created, right, what are the sources that were used
[02:06.840 --> 02:13.160]  in order to create this binary, right, what was the process that was used in order to
[02:13.160 --> 02:22.760]  create that, essentially these two things, right, the goal is to have extra information
[02:22.760 --> 02:32.280]  and meta information about all this stuff and so in the end you know exactly what information
[02:32.280 --> 02:36.760]  is inside the binary, right, or was used to create it, right.
[02:36.760 --> 02:44.360]  So think of it very simplistically, you know, if I have a command that builds a binary like
[02:44.360 --> 02:52.240]  a compiler that gets a source file and generates an executable, I want to record, okay, there
[02:52.240 --> 03:01.800]  was this process called CC and it read the wonderful file in C and it produces an executable,
[03:01.800 --> 03:05.760]  right, that's what we want to solve.
[03:05.760 --> 03:12.120]  And then it gets tricky because as you can imagine, you know, okay, we're not obviously
[03:12.120 --> 03:16.480]  looking at only things that are specified in the command line, right.
[03:16.480 --> 03:24.360]  So when you do CC, the hello world dot C, maybe it includes the file, right, you see
[03:24.360 --> 03:30.240]  language has include, so and this might be significant, it will definitely be significant,
[03:30.240 --> 03:36.640]  right, so it might include, slash yourself, slash include, slash stdio dot h, so I need
[03:36.640 --> 03:43.320]  to record that this file was also used in order to produce this binary, right, and in
[03:43.320 --> 03:50.720]  the same way CC is just a program, I cannot just record the name because the name does
[03:50.720 --> 03:57.120]  not mean anything, this is definitely a file and maybe it uses other linked libraries in
[03:57.120 --> 04:05.000]  order to do that and when you produce the final executable, definitely it might include
[04:05.000 --> 04:08.240]  system libraries and you want to record all this, right.
[04:08.240 --> 04:12.480]  So we're not talking about, you know, parsing the command line and seeing the three files
[04:12.480 --> 04:19.240]  mentioned there, we want to record everything happening in there and obviously the command
[04:19.240 --> 04:26.920]  may not be explicit, it might just be make, right, and you want to record everything happening
[04:26.920 --> 04:39.120]  with this build command, right, is the problem clear, has anyone else, has this problem?
[04:39.120 --> 04:45.640]  So the project, the whole project was okay, let's build something like that and how do
[04:45.640 --> 04:50.040]  you create a project, you say what I want to achieve and this is a functional specification,
[04:50.040 --> 04:56.320]  right, so we want a minimal interface, I don't want to tell developers how you have to change
[04:56.320 --> 05:01.160]  the compiler in order to do that and I mean the first idea would be to change the compiler
[05:01.160 --> 05:09.680]  but yeah, let's try to make it minimal, so ideally no changes to the command at all,
[05:09.680 --> 05:17.000]  right, and I want to record the files being created or written, the files being read or
[05:17.000 --> 05:24.080]  the process is being run, so for each of the files I want to know the name, I want to know
[05:24.120 --> 05:29.800]  the full path because there might be difference and then because I care about the, actually
[05:29.800 --> 05:35.360]  the content, I want the hash because the same file can be in different places with the same
[05:35.360 --> 05:40.640]  hash so I have to know what it is and you know, extra nice information, the size of the file,
[05:40.640 --> 05:47.040]  mode of the file and stuff like that, I don't care, for every process that runs I want to know
[05:47.040 --> 05:52.960]  okay, what was the process, what were the arguments, what was the environment because as you probably
[05:53.000 --> 05:58.640]  know, every process reads may be different according to the environment that it finds,
[05:58.640 --> 06:07.360]  yeah okay, start and end times would be useful and stuff like that, right, so this is the
[06:07.360 --> 06:15.000]  information that, this is the function specification, this is our wonderful problem, right, and then
[06:15.000 --> 06:35.880]  40 sat down and worked, well no no wait, you'll take the mic, this is, well he's not engineer yet,
[06:35.880 --> 06:44.440]  he will become, can you hear me now, well to tackle this problem we created the command
[06:44.600 --> 06:49.080]  line tool named build recorder, which you can see right here, build underscore recorder,
[06:49.080 --> 06:55.320]  which records information about the build on Linux, use it, this is rather simple, all you have to do
[06:55.320 --> 07:03.240]  is prefix your build command, whatever that is, make gcc, java compiler, with the name of the
[07:03.240 --> 07:07.960]  executable, build recorder runs transparently in the background while your build is running,
[07:07.960 --> 07:12.200]  you don't need to change anything in terms of your configuration, your source files or even your
[07:12.280 --> 07:16.680]  build system, you can literally pick any project you like at this point in time,
[07:16.680 --> 07:21.160]  run with build recorder and it should run, if it doesn't please file a bug report,
[07:23.320 --> 07:36.040]  now as you can see build recorder, build recorder stores all of that information in an
[07:36.040 --> 07:40.600]  output file by default named the build does recorder dot out, but as you can see you can
[07:40.600 --> 07:46.920]  use the does so option to actually supply an alternative file name, but what does it actually
[07:46.920 --> 07:53.080]  record, well pretty much everything we talked about in the functional specification, for its file
[07:53.080 --> 07:59.080]  we have a list of attributes, its name, its size, its sex sum of its contents as well as its absolute
[07:59.080 --> 08:04.920]  path, similarly for each process we store its command line with all its arguments, a start and
[08:04.920 --> 08:11.800]  end time stamp as well as the environment, we also store a list of relationships, namely a
[08:11.800 --> 08:18.600]  process creates a process that is by forking or cloning or any of their variants, as well as a
[08:18.600 --> 08:25.000]  process opens a file for reading, a process opens a file for writing or some processes are associated
[08:25.000 --> 08:30.440]  with executables, for example if we were to run make, the system would probably run the file at
[08:31.240 --> 08:39.640]  slash user slash bin slash make, we want to record that as well, now all of this information is
[08:39.640 --> 08:45.800]  stored in the output file in RTF turtle format, class is being processed in file and all of the
[08:45.800 --> 08:53.960]  attributes as well as the relationships been the predicates, this is an example, for example we
[08:53.960 --> 09:02.520]  have a process ID 1 which is the initial compilation process, imagine gcc-fusy, it starts at the
[09:02.520 --> 09:09.000]  current time stamp, we have another process which is a preprocessor, the c preprocessor,
[09:09.000 --> 09:15.960]  it starts at another time stamp, we specify that the initial compiler actually created the preprocessor
[09:15.960 --> 09:27.560]  unit and then we have the fact that the we have a file f1 foo.c which should be specified that
[09:28.520 --> 09:36.680]  ah pid2 reads f1, the preprocessor opens the file foo.c which has size 100 and the random
[09:36.680 --> 09:44.120]  has of zeros, we have another f2 which is a temporary that the compiler might use,
[09:44.120 --> 09:51.160]  which as we can see the process number two actually writes on this, we have another process,
[09:51.800 --> 09:59.800]  the classical cc which merges those files together the object file, yeah that's a general idea I guess,
[10:01.880 --> 10:08.760]  well how is it implemented, how does it support all of those languages, well the basic idea is that
[10:08.760 --> 10:15.000]  we basically want to record all of the files and processes that a process uses and now if we
[10:15.000 --> 10:20.920]  think about it all processes and files are actually handled using system calls, so if we were to look
[10:20.920 --> 10:25.960]  at the system calls a process makes, we could see all of the files and processes that it uses,
[10:25.960 --> 10:30.920]  for example if we look at the open family of system calls, open create and its variants,
[10:30.920 --> 10:37.000]  we can easily extract the name as well as the access mode, on the same note for process all we
[10:37.000 --> 10:44.840]  have to do is trace, fork, clone and its variants as well as the execution system calls to actually
[10:44.840 --> 10:52.120]  see the process ID and the executable files, now from the information that we extracted from these
[10:52.120 --> 10:57.880]  two we can actually get even more information like the command line, the environment, a link to a file
[10:57.880 --> 11:05.640]  which from there we can get the size that has in the absolute path, yeah all of this happens
[11:06.360 --> 11:12.200]  on linux, so we basically just need a way to trace system calls, under linux this is a fairly
[11:12.200 --> 11:17.320]  straightforward problem, we use pitch-race which directly copy and paste it from the command line,
[11:17.320 --> 11:20.440]  it is primarily used to implement breakpoint debugging system calls tracing,
[11:24.280 --> 11:27.000]  now that's it, it's a very simple program,
[11:27.960 --> 11:39.240]  no, for the duration of our project, okay so photos run through the slides that
[11:40.120 --> 11:50.360]  did implement it, so we have lots of time now, so it's not perfect it has issues right, so let's
[11:50.440 --> 11:58.040]  start discussing, you got the major idea of how it works right, what are the issues, the main two
[11:58.040 --> 12:11.960]  big buckets of issues are the real complexity and performance right, so real complexity, remember
[12:11.960 --> 12:17.880]  that we said you know it's a compiler that just reads a file and creates an executable that was
[12:17.880 --> 12:25.080]  this wonderful diagram first, it's real life is not that simple right, so on the right hand side
[12:25.080 --> 12:31.720]  you see a more typical ideal again compiler, so when you have a compiler it actually calls
[12:31.720 --> 12:37.640]  three different processes right, you call the first one the first pass that reads the C file and
[12:37.640 --> 12:44.680]  creates an assembly file and then you call the assembler that reads the assembly and creates an
[12:44.680 --> 12:52.280]  object file and then you have the linker loader that reads the object file and creates an executable
[12:52.280 --> 13:00.120]  right, so this is a very abstracted and ideal world situation, we do not, real world is nothing
[13:00.120 --> 13:08.520]  like that, real world may be like that if you have you know done your compiler courses and you
[13:08.520 --> 13:13.640]  have seen the different passes of compiler and stuff like that and then you go to the real world
[13:13.640 --> 13:20.600]  where the hello world which is just you know print hello world dot c in current Linux is this one
[13:21.880 --> 13:27.720]  right, so I'm not going to be explaining every file in that one but this is just you know the three
[13:27.720 --> 13:39.640]  lines of main hello world print hello world and if you compile it with gcc explicit, so you see
[13:39.640 --> 13:46.200]  there are, what can I say, lots of include files been included, this is the first step of compiler
[13:46.200 --> 13:53.960]  right, the compiler first step right, so it you think that it will only read hello world dot c
[13:53.960 --> 14:00.040]  but you have a hash include in order to include the print f definition, so and this include
[14:00.040 --> 14:06.600]  unfortunately includes you know std def and libc header start dot h and lots of other different
[14:06.600 --> 14:15.640]  files and all these are files that have been read by the first process and then it creates
[14:16.840 --> 14:22.040]  something else which I'm not sure where it is somewhere or so it creates the temporary files,
[14:22.040 --> 14:29.320]  it creates the assembly there if you can see and then it creates the object files and in order to
[14:29.320 --> 14:39.080]  create the final executable lots of libraries have to be included right and this is the library
[14:39.080 --> 14:45.080]  is being included in the executable right and then we have the other set of things that in order to
[14:45.080 --> 14:51.960]  run cc one which is an executable so a file in the file system right, you have to load dynamically
[14:51.960 --> 14:57.880]  a number of other libraries that this executable depends on, so you have to record all these as
[14:57.880 --> 15:06.840]  well because again you have to have an accurate record of everything being used right, so yeah
[15:06.840 --> 15:17.800]  even the hello world example is complex right and you create and record lots of files and processes
[15:18.440 --> 15:25.400]  for that one which means oh and a lot of them are going to be reused again and again and again
[15:25.400 --> 15:31.880]  right so when you had for example to compile two different files right they will both include for
[15:31.880 --> 15:40.360]  example stdi.h right and ideally you don't want to redo, you don't want to have another record
[15:40.360 --> 15:46.520]  another box here for the second instance of the same file on the other hand it might not be
[15:46.520 --> 15:50.680]  the second instance of the same file because something might have changed right while you're
[15:50.680 --> 15:56.440]  running somebody else is installing a new compiler and it messes up all your binaries right and it's
[15:56.440 --> 16:03.480]  different binaries in the first run on the second run and all this stuff so this brings us to having
[16:03.480 --> 16:15.320]  to do it efficiently, so you're doing performance. Well performance isn't great at the current moment
[16:15.320 --> 16:23.240]  I mean we have to stop and restart the process multiple times using ptrace like you stop the
[16:23.240 --> 16:32.040]  process you read an entire file from disk you hash it thank you and then you restart the process
[16:32.040 --> 16:40.360]  this is relatively expensive as you can imagine the current profiling shows like when I built one
[16:40.360 --> 16:47.080]  and my system normally took nine minutes with build recorded it took 36 minutes so yeah it depends
[16:47.080 --> 16:53.480]  on your hardware your hard disk pretty much but the good news are that there's a lot of room from
[16:53.480 --> 16:58.520]  improvements because first of all we haven't optimized anything we're using plain arrays
[16:58.520 --> 17:05.240]  we will be switching them with lookup tables so we expect massive performance gain from this
[17:05.240 --> 17:09.000]  in fact when I tried the hash mapping limitation we dropped this to 22 minutes
[17:09.880 --> 17:17.080]  so that's optimistic and another thing to mention is that ptrace actually makes a
[17:17.080 --> 17:23.880]  multi-threaded process run as a single thread which is an issue so if you run for example make
[17:23.880 --> 17:32.200]  dustj8 you won't actually get the performance benefits of multiple threads we have plans to
[17:32.200 --> 17:37.720]  change that as well lots of changes need to be made in fact we proposed it as another project for
[17:37.720 --> 17:45.000]  GSOC for this summer and yeah that's it pretty much we hope to see an improvement we can't really
[17:45.000 --> 17:51.320]  tell how what at the end the overhead shall be like how much we can improve build recorder but
[17:51.320 --> 18:00.520]  we will get there now regarding future work pretty much we approve performance we plan to handle more
[18:00.520 --> 18:06.680]  programming systems I mean you can use it with any programming language you want but if I was to
[18:06.680 --> 18:13.320]  for example compile a project in Java using a build system like Maven Maven has web dependencies
[18:13.320 --> 18:19.240]  it downloads packages from repositories so ideally we would like to also record those
[18:19.240 --> 18:27.000]  repositories those URLs this is another proposed project for the next GSOC and next we have porting
[18:27.000 --> 18:34.840]  it to non-linux systems like other unixes net bsd free bsd the list goes on the one thing that
[18:34.840 --> 18:39.800]  wasn't mentioned is that build recorder at the current time only works with Linux kernels of
[18:39.800 --> 18:50.520]  version 5.3 plus reasons being reason being that we are using some system calls that make it run
[18:50.520 --> 18:55.160]  on every architecture so you can build it on a Raspberry Pi or any architecture of your like
[18:55.560 --> 19:02.680]  any architecture that Linux supports in expense of not having support for prior versions
[19:04.680 --> 19:14.280]  that was us thank you this is the url yes but we also have a QR code if you do not trust your
[19:14.280 --> 19:20.200]  code you don't know you're going to be a re-crow right yeah this is not the same url
[19:23.000 --> 19:25.240]  so we're plenty enough time questions
[19:25.240 --> 19:46.840]  right so the question was how do you know when to hash the file because when you open it for
[19:46.840 --> 19:53.560]  writing you have to know when it the modification ends and then you should hash the result that's
[19:53.560 --> 20:00.840]  correct and the answer is well we have the file when we find the closed system call
[20:02.120 --> 20:05.640]  you wait for all modifications to happen and then upon the closed system call
[20:06.440 --> 20:12.040]  the process is stopped so all the stuff is in there the virtual memory and the list goes on so you
[20:12.040 --> 20:17.880]  have the file at that time we also have the file when we open the file to see if it was seen before
[20:18.840 --> 20:25.400]  so we can have a lot of nice graph in fact the graph is just a dependency graph in a
[20:25.400 --> 20:28.360]  like a semantic web as one makes sense
[20:39.400 --> 20:45.480]  nice so the question was how do we consume this we produce all this information and how is this
[20:45.480 --> 20:51.880]  consumed and the answer is this is a build recorder it just records here's the information we
[20:52.760 --> 20:59.880]  we have we do not consume it at all it might be used to create to enter it into an s-bomb or
[20:59.880 --> 21:04.840]  something like that that's completely out of this this is just recording all this information
[21:05.400 --> 21:06.920]  it's out of the scope of this project
[21:07.640 --> 21:10.520]  yeah let's see sorry
[21:15.080 --> 21:20.280]  have we tried converting the data to svdx similar question to that one no we just record
[21:20.280 --> 21:25.880]  build this is a build recorder right another tool might read this wonderful output and create
[21:25.880 --> 21:29.400]  whatever they want it's way on the back
[21:34.840 --> 21:41.640]  the question is have we explored ebb ebbf ebbf whatever you you know what i'm talking about
[21:41.640 --> 21:47.720]  yet the answer is no no not yet
[21:56.440 --> 22:01.720]  sorry
[22:05.400 --> 22:10.680]  oh yeah sure so the question is what's the problem with maven that we said that it's going to be
[22:10.680 --> 22:17.560]  handled in the future yeah on this level maven works the same right so we can't record the
[22:17.560 --> 22:26.120]  information but we will just be recording hey a jar file was just been used we would like ideally
[22:26.120 --> 22:32.600]  to record because maven already downloaded it but we're just tracing the file system system calls
[22:32.600 --> 22:37.960]  right so ideally would like also to record the information hey we're downloading from this location
[22:37.960 --> 22:43.800]  this jar file we put it there and then we use it right on the level that we're using it we can
[22:43.800 --> 22:48.840]  record it right now sorry go ahead so i missed you sorry
[22:57.240 --> 23:03.080]  the question is do we distinguish between modifications to a file or completely new
[23:03.080 --> 23:09.400]  created file and written and the answer is well if we did that the performance would be even worse
[23:10.200 --> 23:15.960]  you can actually do it you can actually add another handler that checks for write system calls
[23:15.960 --> 23:20.600]  and has the file every time but imagine if you had to has the file every time someone calls right
[23:22.200 --> 23:23.000]  i don't want to
[23:23.880 --> 23:48.600]  okay the comment was that there was a
[23:48.600 --> 23:56.280]  yesterday talk in the golden room that they showed using ebpf instead of ptrace for a
[23:56.280 --> 24:02.200]  similar work and performance was great something to explore yes go ahead
[24:02.440 --> 24:22.440]  the question is how does this compare to scan code trace code that supposed
[24:22.440 --> 24:29.480]  doing the same thing we have not measured it for that i didn't even know that you know sorry
[24:29.480 --> 24:36.920]  oh if it runs for this race it's yeah it's the same as trace race we don't
[24:39.480 --> 24:40.520]  any other questions
[24:40.520 --> 24:57.720]  wow the question was what does it happen when you run it
[24:58.760 --> 25:05.880]  while building a container you mean or so again remember it just records
[25:06.680 --> 25:14.920]  it just records all the system calls right into the disk right so when you run it in the
[25:14.920 --> 25:22.360]  while you do docker build it will record all the files being used in order so if you do copy
[25:22.360 --> 25:29.560]  things it will record everything being copied into the layer right stuff like that yeah go ahead
[25:30.680 --> 25:35.400]  to be honest the command doesn't have to be a build command you can literally plug in anything
[25:35.880 --> 25:41.800]  in there ls or whatever i mean we are not supporting it we can promise that we will
[25:41.800 --> 25:49.000]  be supporting it in the future but you can do that so anything you can imagine it will run
[25:49.640 --> 25:55.160]  docker it will probably record that docker docker was called docker will still run all those
[25:55.160 --> 25:59.640]  dependencies it will still try to compile it will still link against all those libraries which
[26:00.360 --> 26:09.960]  is anyways so yeah any thoughts about going off off linux to uh windows
[26:14.840 --> 26:24.120]  we are at an open source conference repeat the question ah yes they asked if we're planning to
[26:24.200 --> 26:28.760]  implement something like this on windows well no
[26:31.560 --> 26:37.400]  first of all first of all it's hard the idea the idea would be great if i have developers
[26:37.400 --> 26:44.920]  who can do that and know the corresponding things magic to do uh in windows pr's welcome
[26:45.640 --> 26:50.680]  it's an entirely different process you don't have to all the system goals and other unixes
[26:50.760 --> 27:03.000]  we can work with that but there must be something on windows but i don't know
[27:07.480 --> 27:16.680]  no okay thank you
[27:20.680 --> 27:21.180]  you