Hi, everyone. Well, I'm very impressed to have such a large audience for such a small
tool. But, well, I'm Beatrice. I work at the French Media Lab. And today I'm going to
present PIMI, which is a tool to study image propagation. The Media Lab is a lab where
social scientists try to, among other things, study the traces that people leave online.
And for now, they are quite well equipped with tools to study text. But when they ask
me, OK, how can I study mean propagation? I'm still struggling to give them answers.
So what does it mean to study mean propagation? It means, OK, being able to recognize that
some parts of an image are copied or partially copied. So what this tool does, it's very
simple. It's able to create clusters of images and group together images that are total or
partial copies of each other. It's able to deal with image transformation, so if the
image is cropped or zoomed. And it's able to adapt to copper's characteristics. So
it will try to make the best of your data sets, depending on the number of images you
have or the type of images you have. What PIMI is not able to do is to cluster semantically
similar images. So it's not the tool that you are going to use if you want to create
clusters of cats and clusters of dogs or, I don't know, find images of violence versus
images of peace. And it's not able to do some face recognition. So again, you will not
be able to make some clusters of pictures of Elizabeth II versus clusters of images
of Emmanuel Macron. What you could imagine doing, and we could imagine also work together
if you are a researcher working on those subjects, is to study the propagation of MIMA on social
networks, as I was saying. But also you could study the usage of the press agency photos
in a press corpus or stock photos as well. You could also study the dissemination of
fake news based on image montage. Or you could study the editorial choices between different
media, depending on whether they use the same images or not. So let me do a quick demo of
how it looks for now. It's not on the screen. Okay, let's forget about that.
I'm very sorry.
Well, I'll try to make it work. Okay, well, it's still not showing totally all clusters.
So we create clusters of images. So this is a data set that is created by the French
Inria and that is presenting some degradation on images. So they take an original picture
and they apply some filters or they crop the images to see if they are able to group the
images together. So we can see that we have pretty correct results on that data set. And
this is our results on some images that we collected ourselves on Twitter using Elon
Musk as a query. And so we try to clusters those images. So as you can see, we have images
of Elon Musk. We are able to group together some images that are crops of others. So this
is probably the source image of the montage that has been done here. But we can also see
that we have some problems with the tool. For example, here we have a cluster with two
images that have been assembled together and we create a cluster of actually two images.
But well, that's the state of the tool for now. And now I try to come back to my slides.
Okay, so how does it work? For people who work in computer vision, I'm going probably
to say some things that are quite basic, but I'll try to make it clear for people who
do not do computer vision. So it is not based on colors at all. It's used like the grayscale
of images. And it tries to detect points of interest on a picture. And then it uses these
local key points as vectors. And then those vectors are indexed in a database that is
able to perform some very quick similarity search.
of the tool. As I say, there is that problem of parts of images that create clusters that
are bigger than they should be. So our plan is to be able to detect images that are actually
those links between two clusters. So to be able to detect that this image is actually
containing two images and to be able to deal with part of images. And also what we would
like to do is to show images in their context, to be able to show the tweets that contains
those images or Instagram posts, et cetera. Or at least to show additional metadata for
the users. And also we would like to show you the graph of image similarities so that
the clusters that are resulting from that graph are not interpretable. And to improve
our tool, we need your use cases because for now we have those two, three databases. But
we would be very glad to do some partnerships with other researchers to improve the tool.
Thank you very much for your attention. If you want to look at the slides, we have the
references to all the images used and to the papers of the algorithms used by Pini. I'm
open for questions. We had a bit of trouble with the sound stream, but it's back on now.
So yeah, you should repeat the question. Okay, I'll do. Yes. So thank you very much
for that. We'll try to find similarities. Oh, sorry. I have to repeat the question.
So the question was, if I understand well, how to reproduce that use case, not on images,
but on other types of documents that would be, I guess, some features. 3D counterparts.
And I'd say, well, as long as you can, like, represent your data in the shape of vectors,
then you're ready to use face to, like, do some, some search for nearest neighbor in
your database. And then you can go for the whole pipeline, create some graphs, find communities
in the graph, and go for it. But I'm not sure Pini is your tool, but, but, well, the architecture
of Pini could be, of course, a model. Yes. Is there any project current or you're completely
ongoing that Media Labs has used before, or is it still largely in development? It is
largely in the development. Sorry, I repeat the question. So are there some projects at
the Media Lab that are currently using Pini? And the response is no. Yes. Sorry, can you
consider any other ways that can be considered? Yes. Have you considered other ways of presenting
picture similarity or using picture similarity, or the types of image similarity, if I'm
here in the Sunwell? Well, I'd say that that was what I was saying in my second slide. There
are other types of image similarity, for example, semantical similarity. And, well, maybe in
a few months, if we have like a robust architecture, we could maybe include some other types of
vectorization of images. But for now, well, there are already tools that do that. Like,
there is something called Clip Server that helps you find similar images from clip vectors
that are like semantical vectors. So you could use that tool. It's great. Yes. Yes.
So the question is, is the tool really able to distinguish the thing that is of interest
to us, the fact that we are talking about a dog? So the tool is only able to find partial
copies in an image. So the tool would probably be able to say that all those images contain
the same parts of face of a dog. So it would probably be able to group all those images
together. The problem is that if there are other images in the database that contain
the rest of the images, then they would probably also be grouped in the same cluster. So that's
why what we are currently doing about parts of images would let us improve the cluster
so that it's purified from the rest of the images. And we could have a cluster of the
face of that specific dog and then a cluster of that taco in the second cluster. Yes.
What kind of clusterization do you use on the graph? Well, for now, we have the best
result with, excuse me, what kind of clusterization do you use on the graph? For now, we have
our best result using pure connected components. So actually, the specification we do on the
graph to reduce the number of links between images is enough to have separated connected
components in the graph. And so we take each connected component and it's our cluster.
What we would like to do is to try to mix with some Luvain community detection, but actually
for now, it's not the thing that works best. Yes.
I'm not sure I understand the question. Can you try to rephrase it? Okay. What things
are you looking at to improve the model? Well, there are many things we are looking
at. For now, mainly, we look at techniques to do a better graph specification in order
to find more coherent clusters. We are not so much working on the local descriptors part
of the tool for now. Have you considered using the direct link
to the Twitter images or social media images online? Did I repeat everything? Well, yes.
We would like people to be able to see images in their context because, actually, they won't
understand what's happening if they just have images. They need to see, okay, why was this
image published? Who answered, et cetera. This would probably mean that we need to add
at least the links to the pulse or maybe some kind of visualization of it.
We have a bit of time here. Any more questions? We can take one or two. If not, we can switch
quietly. All the next questions.
Thank you.