So I'm Bruno Latulier and I'm working at EDF which is an electricity utility company
in France and we are doing a lot of numerical simulation and of course we need to have a
good verification and validation process and inside the verification process we have to take
care about the floating point error. So I will give more detail about the floating point but I
will be really short because I think almost everybody knows about it. We have in the when we
are doing numerical simulation we design our algorithm with real number is the wonderful
world of the mathematics but we have limited precision and so we have to use float or double
in our code and it means that some usual in our code we have to do one thing and this small
one thing because in double it's 10 to the power minus 16 it's really small this small one thing can
have a huge impact for the final result of for simulation. So we need to be able to
to have an estimation of the difference of the floating point computation and
and the result we expect in the in the mathematical world and usually the developer that's our problem
but usually the developer are able to see that there is a problem because when you modify your
compiler option when you want to add parallelism you change the order of parenthesis and you have
different results and so you see you have a problem but the problem is there and so we need a tool
to be able to do this error estimation for real industrial complex application and it's the
tool we called Vero which has this objective so I'm sorry but I need one slide of mathematics
so it's quite easy we use stochastic arithmetic so usually when we want to debug
we we don't like stochastic but there we are we want to we will use the stochasticity to debug
so we replace the operation by the same operation but with stochastically random
stochastically rounding so I tacadize and it's to the left or to the right with a defined probabilities
I'm doing that for all the operations in my program so it's like a galton ball
each operation I go to the left or to the right and at the end of the program I have kind of
distribution and I use the support of this distribution with the formula there to compute
the number of significant bits or the number of significant digits on this small program one divided
by three followed by a multiplication by three this is the normal execution of the program with
one to the nearest and after I put three execution with random rounding and I use all this result
with the formula to see that I have almost two significant digits so it looks like really easy
like this but if you have to modify all your program all your floating point operation in your
code it will be really tough so the idea is to use valguine a dynamic binary instrumentation
which is help a lot to develop this kind of tool valguine will give me an intermediate
representation and I can just modify the intermediate representation so I don't I'm not able to write
one line of assembly and I will do this kind of tool so valguine is really powerful and so when
there is an operation an integer operation I can replace I can use the same when there is a floating
point operation I can add counter that's easy and I can call my own implementation of the
floating point operation and in this operation I can add the stochastic part of the operation
for the user point of view I need to run the code several times that's the bad part
I need to extract the value of interest I'm computing something I have to know what I want to
compute what is the result the good part it works for all languages so C++, Fortran, Python
yeah and it works with external library when we don't have the source from the valguine developer
point of view it means I have to replace all floating point operation what is nice I there is
no need of shadow memory so it's quite fast and what is really different compared to the other
tools I want to modify the result and so this is the difficult part so I give you a small example
which is called the Mueller suite so it's I compute a suite with recurrence and add some verbo
CT to be like or simulation so there is this kind of result this is the execution with
rounding to the nearest and there is address it's stupid but it will it's it's only a way to
to present something later but it's stupid but I see really often this kind of thing we don't
control or user and so now I run several times with veroo and in red there is all the all the
output which is different so I can see there the result is completely wrong
so when if I say to my colleagues you have a tricky floating point bug somewhere in your
two millions line of code you will get a lot of friends so I need to do something and with one
colleague we we developed something called delta debug which is a trial and error search algorithm
so the user has to provide two script the first one is how I call my program with veroo so it's
very simple because it's only a prefix command where the valguin prefix command and another script to
say if the result is good or not and to say if the result is good or not we are doing a comparison
with the result to the nearest we don't know if the reference is a good one we only know that
there is a difference as is the difference we want to explain and and then we are we are with
this command we we have to say the number of samples we need and at the end you will get the result
there is problem in these two lines of code that's really nice it corresponds to the two
divisions in my example from the valguin developer point of view it means I need to generate the
search space I have to know which line or which symbol in my program where there is floating point
operation so this is the first part and the second part I have to be able to run the program
with a specific configuration I mean a set of function of line which are instrumented and a
set which are where the line are not instrumented so I have to have to introduce
discussion between this tool and valguin but it's not too difficult to do
and it works well on real application but now I will present something which is more experimental
I really go there
I with with the two lines of code sometimes you do you do not have the right information what you
want sometimes is to know the problem happened in the first iteration not in the last one
and this corresponds to temporal localization and so for that I need to to modify the search space
and I use in fact the the output of my program and I can't use directly the output because
from one line to the other I need to to have the same key of all in all execution
so it means I have to wildcard all floating point in the form for the search for the search space
and especially when someone print addresses I have to wildcard addresses because it will be
different in all in in different one and for the users there is nothing else to to modify in it the
result is it happened in begin it and and it is also it's a two first iteration the problem
happened only in the two first iteration so the standard output match without temporal context
I only use the the fact that the user prints the number of iteration
that's very important the user has to pay attention to the bufferization
if you put if you print everything at the end of your program it's unuseful the empty line can
be ignored and I can modify the the output by a filter script and the two last element used
together we are able to to work group iteration we are able to do a lot of things
and from the valgoin developer I have to define a five format to define the interaction between
the IO so this is the standard output or even a file and veroo like client request the idea is to
be able to call client request activated by the IO so my conclusion with veroo we are able to estimate
floating point error it works well we are able to search the origin of floating point error
with delta debuil it works well we are able to search mixed precision configuration it works
well but not so much and we are also able to search where error are amplified
it works sometimes and in my roadmap I want to to be able to to work on new architecture
especially arm 64 I want to add new search space like the bike trace because if someone has
encapsulated the addition it will give me that the addition is unstable nice
and the last point and it will be probably the the most difficult research part is to be able to
to to do error amplification localization without false positive that's the that's the key point
and for the real conclusion it's on github there is documentation there with paper and so if you
want to use it I will be happy
so I imagine quite a few runs are needed because I guess for each floating point operation
you have multiple combinations and can you give some
numbers from experience how many you run one needs to track down bugs
yeah the the question is how many samples we need to be able to do an error estimation
it depends off the accuracy you want if the if your code is unstable and
only to run run with the nearest the first one you have difference you have a bug
so to have a problem it's a really small number if you want to prove that there is no bug
statistically you will need to increase the number of sample
and we have a paper I did there is a paper to say how many number you you need with
with the confidence interval and everything but in practice it depends off the running time
because it's it's always it's always the first question and now we have done the work to be able
to have a number of samples and with theory and nobody use it the reason is is the computational
time which is important
in the interfloor project we are we have collaboration with colleagues of Versailles
near Paris and they are doing all the almost the same work with LLVM infrastructure
and so we are working together and it's a little bit faster
but in fact it's more convenient to use from the binary point
okay question did you at any point contemplate instead of using the stochastic method of
figuring errors actually having your valgrain model of the floating-wave instructions use
interval math that is represent bounds and then propagate the errors then you wouldn't have to
rerun anything you just calculate with an upper and lower bound for each value and
the problem with interval arithmetic what is now first what is really nice with interval
arithmetic right the question is is it better to use interval arithmetic instead of stochastic
arithmetic what is nice with interval arithmetic it don't lie never it say it will say the truth
and it's really nice but the problem is on real industrial application it give the result is
between minus infinity and plus infinity and it's true so that's the problem there is a lot of
false positive in fact I'm I really use I use interval arithmetic when I discover a problem
like that with this tool I'm I extract the problem I work on a small proxy app where I
I'm able to to run interval arithmetic but with with multi precision to increase the precision in
order to reduce the size of the interval as there it's it's really nice because I've I've access to
the guantt's tool but on real industrial application it's it's too difficult
it it works with acc avex avex too but I there was limitation with avex 512 because
it's not implemented in valguine
so this is a question about I think the mathematical library
there is two way to do mathematical to to call mathematical function the first way is sometimes
there is hardware and now I'm able to format wire to instrument fma and sqrt
and the other way is to call the dynamic library of mathematical function and there is way tough
because all the the developer of the mathematical library really know floating point operation
and they are taking into account the fact that they they use one to the nearest operation and so
if you use veroo with stochastic wounding on the mathematical library you can get
stake fault you can get a lot of you you have a lot of problem it means that I have to exclude
from the instrumentation the mathematical library and I have to re-implement myself
the all the mathematical library to add the stochastic part
uh that's quite tricky and uh and in fact I'm using some some thing which are a little bit
limits because I use a reference with the quad mat and uh yeah it there it's okay but if I speak to
to to to specialist of floating point they will kill me
officially we have one minute for one last question but this is the last one
I didn't try so you know what you get
it so I worked in a project once where people had started using this minus f unsafe math in
gcc which basically creates such issues and but they weren't it was a trade-off that they did
performance but they wanted some more reproducibility would this tool work to kind of narrow it down
to certain code paths where you might want to disable the fun safe math
yeah uh the question is uh what what is the the liberty of what kind of freedom can I give to
the compiler to uh to optimize my code in terms of floating point if I understand well and uh
and in fact it's an open question it can help in in in the in that sense um I'm able to um to say
where the code is sensitive but it's related to my test case because it's it's uh I'm not able to
say anything of about the code in general it's the code plus the data set so to help a compiler
it's really tough because the compiler needs to be able to run for its kind of data set
and and the other part is uh or the question is uh which kind of option should we use
for for the for the compiler what I see is a lot of people want to be reproducible and so they use
all zero option and so the only thing they are doing is to be able to reproduce the wrong result
because if you don't know why all zero is better than all three there is a good probability that all
three uh will give you a better result because uh when you are doing a summation you will regroup
it for uh for with the three uh when uh there is fmi there is one error one one
ending error less so it's if if I have an advice you use all the compiler option you can and except
if you know what you what you do uh there are small parts in veroo where I have to take care about uh
floating point error really carefully if you if you use uh error free transformation you have to
take care about floating point error uh so I don't know if you know what is error free transformation
no nobody knows the error free transformation so it's it's a way to when we do a floating
point operation we are able to compute the error on the floating point of of this error
usually is we can represent this number and so we can compute the error and if you
say to a compiler uh the fast math option you will say this error is zero because it's
mathematically is zero and uh it will skip it so if you want to do um tricky algorithm with this
kind of thing uh you have to protect it uh you have to protect your part in fact this part
are used in twasic to be sure that the compiler is not able to to skip it if I make there is a
misconception in floating point real that people want accuracy and reproducibility
and they think they are the same but they are completely different
and you can be inaccurate but they were reproducible or the other way around
and actually in this case the inaccuracy wasn't the problem uh it was more the reproducibility
because the the accuracy wasn't that important that's why people that probably at some point
decided okay I can do so much more in the limit of the amount of time in real time context but let's
see
is that accurate
no
coronavirus
system
Valgrind is not natively possible to run like a ELC or some other end-gain system?
I need Valgrind. In fact, I have less portability than Valgrind because I work for x8664.
This works. I'm working for ARM and this is tough.
This is really tough.
I will need to do patch inside Valgrind for that.
And it's not my... I'm not really confident to do that.
But for x86 it was not easy but it was okay.
Is there no solution for all PLC systems?
Not yet.
That question. Do you have plans to include this into the upstream version of Valgrind?
At some point.
It could be nice. I will like that.
In fact, I think there is still a lot of work because I have only one architecture.
And in the test infrastructure it's quite different because in the Valgrind architecture
we need to do only once a test.
And I have to run several times to be able to compute.
And so I have a completely, I have a separated test infrastructure.
So it couldn't be tested as part of the modern world?
Yeah. It would be difficult I think.
There is work and I think I will need to discuss with the Valgrind team.
It's why I'm there.
I want to begin the discussion with the Valgrind team.
Do you have something about the performance panel?
I was expecting it was the first question.
So with the nearest it correspond to the instrumentation.
And I'm doing a dirty call and I'm doing the same operation.
With float it was not really optimized.
I mainly work with double.
And so it's quite acceptable.
But it's not, the program is a stencil with FMA.
So it's more difficult than a lot of other code.
And if you are running with a code where there is only IO it's really fast.
So but if you want to work with a Blast tree it's worth a lot.
So the one I presented it's only the random part.
And there is a lot of other kind to do randomness between upward and downward.
And the other kind is to over some crazy false positive.
So and FMA only is what I will get in Valgrind when I will be able to modify Valgrind.
Because in Valgrind there is a there is a
the FMA is implemented with the software.
With the software and so there is a lot of operation.
So we can reduce the time of the tool none.
Tool none is yeah I'm faster than the tool none.
So it means that there is a problem.
So I discovered this last week so I'm not able to correct everywhere.
And for me it's important even if I because when I'm doing Delta debug I do not instrument.
I do not instrument some part with FMA and so it's costly.
So I really need to to to modify Valgrind to reduce the performance cost.
Because it's really painful to say I'm doing nothing and it's more costly than to modify the 14.0 behavior.