Okay, I think we can start already. Hi everybody, I'm Diego Barreiro. I'm one of the open source
contributors to the MIT App Inventor project and today I'm going to be talking about App
Inventor and how we can introduce artificial intelligence to kids using this platform.
But before getting into it, I would like to introduce myself a little bit more. I first
started coding with App Inventor when I was 14 years old. It was 2013 at that time and
I basically wanted to build an app. I didn't know anything about coding and a high school
teacher showed me this amazing platform. So I just spent the next couple of years building
up with App Inventor and eventually I switched it to Java coding and I was able to contribute
to the project later on as I'm doing right now. So what is MIT App Inventor? MIT App
Inventor is an online platform that enables anybody to build any kind of application without
having to learn any programming language like Java or coding that is nowadays the most popular
ones. This is the interface and it has the mock font on the center and on the left side
we can have the components, the elements that will make the app like buttons, labels, text
boxes, text areas, like any kind of interaction that the user will have with the app. And
then we can customize the properties like colors, text fonts, text sizes, whatever on
this panel so we can make the app look as we wish for the final user. And how does the
logic work? Well, most of you may know about Scratch. App Inventor works somehow like Scratch
using this block language. So let's say that we want to play a sound when the app opens.
We will be using a block that says when the screen one has opened, we want to play a specific
sound later on. And that's how we can just make any kind of application. MIT App Inventor
allows existing Android developers and Android developers to introduce new components using
extensions. And we will be using today one of those extensions that was developed by a
research project at MIT that enables to classify images on different groups using artificial
intelligence. And to give some numbers of App Inventor, it was tested in 2008 as a Google
project. And then a few years later it eventually was transferred to MIT. Right now it has gathered
over 18 million users since it was created, since it was transferred to MIT with nearly
85 million apps that have been developed. And on a monthly basis we get roughly around
one million users. And in terms of open source contributions, we have seen 164 different contributors
to the project on GitHub. So today I'm not going to be giving the classic
talk. I'm going to be showing a tutorial and people in the audience and at home can just
follow this tutorial by visiting the following link on how to build an app. And what we will
be doing today is building the Pickaboo app. Pickaboo is a game that is usually played
with babies that when you see the baby, the baby loves and when you hide yourself, it
just cries. So to show the final result, this will be the final result. Let me just switch
to my phone. I'm going to be mirroring this phone. So I open the final app and I'm going
to start using the camera. I can see, here I am. I'm looking at the baby that is happy.
If I hide myself, it just cries. So let's get into it.
So how we can use MIT App Inventor 2. The standard instance of App Inventor is hosted
at aiu.appinventor.mit.eru, but that requires an account. So MIT has created this specific
instance called.appinventor.mit that allows you to create these projects without any existing
account. We will be using an anonymous account so that can be cleaned up later after finishing.
And we will receive a code like the following one. Now this is blurred, but you will be able
to see a code. And on the previous screen, if you want to recover the project later on,
you can just paste the code that you previously get right here. And once that is done, you
will be able to access an anonymous account as you can see over here and you can just
start creating the app. So let's get into it.
So we can visit fosdm23.appinventor.mit.eru and we can click on this link. This link
is basically the code App Inventor instance and we are loading a template project for
the pickable project. So I'm going to click on this one. So I click on continue without
an account. And just wait a few seconds for the project to download from the repository
and here it is. That was faster than last night. I can see the code here so I can just
copy paste the code to access this instance later on. And as this is a tutorial, we can
see on the left side that we are seeing a description of what we are trying to build
with a detailed step by step guide. And this is the instance of the project. I can see
the happy baby and then the sad babies hidden here. Okay, so let me just continue the presentation.
The next step is turning the classifier. I'm not going to get too deep into the machine
learning and how it works. I'm just going to be providing a very high level overview
of how this works. So we will be using an image classification system that consists in creating
two groups of images. We will be creating one group of images that is the face of
myself looking at the baby and another group of images that is me hiding from the baby
so we can show the sad face. So how does this work? We can visit this website, classifier.appinventor.mit.edu
to train this model. Just as a side note, this instance only needs internet to load
once. To train the model, that all happens in your browser so no servers are involved,
no images are transferred outside your desktop. And this website is also open source so you
can just check. There is the link on the FOSDEM23 website. So we visit the website and we start
first creating the images group. So in this case we will be creating one image group for me so I'm
looking at the baby and not me I'm hiding from the baby. If for example we are in a biology class
and we want to classify trees, we will be creating one group of images for each kind of tree so we
can recognize them later on. Then we will turn on the camera to take a photo of myself setting
the group of images that I'm going to be saving this. So if I'm looking at the camera, it's going
to be the not me group. If I'm looking at the camera, it's going to be the me group. This is the same
for the not me. And once that is done, when we have a reasonable amount of images for each group,
which should be around 5 or 10 images for each group, we can train the model. As again, this
training happens in your computer so no images transferred outside of your computer. We can
then test the model to with new images to make sure that we have properly trained the model and
can identify ourselves. And once that is done, we can export the model and load it into App Inventor.
So I'm going to be doing that very quickly. Go to foster203appinventor.mit.edu. I open the
classifier instance. It's going to ask for permission to use the webcam. There it is,
I just accepted it before. So the first step is creating the labels. First the me label, enter,
and next the not me label. And well, the light is quite hard. I think it was...
Take a few more images of me like looking to different places so I can train better the model.
One more, one more. So seven images should be fine for this demo to not take too much time.
And now for the not me, I'm going to take like one photo of me not being there, basically.
I'm going to be using the right hand to hide myself so I can put this one in front of my eyes,
turn it upside down, diagonal, like more images, the other hand as well, like that.
So that should be enough for now. And once that is done, we can just hit the train button
to train this model. It will be built on your local machine without sending anywhere.
Okay, this was faster last night.
Seems like the time that we saved from the loading the project, we lost it here. So now it's training.
Yeah, it's my laptop. So this is a React app that has been open sourced and the only internet
required is to just get the initial webpage later on. We can just disconnect and it will
work perfectly. It's just offline training. If you really want to build it fully offline,
you can just launch the React app locally and it will work. So now the model is built and to
test it, I'm going to be looking at the camera as I did before. I captured the image and I can see
that there is a 99.42 confidence that I'm looking at the camera. If I take a photo of myself hiding
from it, there is a 99.33 confidence that I'm not looking at the baby. So once we have validated
the model properly, we can export it to the app. And we will get this model.mdl file for AppInventor.
So let's go to the presentation. And once we have the model, it's time to code the app using
blocks. I'm not going to go through the slides anymore for people at home. If you have internet
problems or the streaming is down, feel free to follow the slides. It's a step-by-step guide.
But for here, I'm going to be showing the tutorial live. So let's go back to the
project that we just loaded. And right here, we can see a quick description of the project of what
we are going to do. A set up of the computer of how to connect to the MIT instance to the app.
I'm going to show that at the end of the presentation. And we have here the pickable example.
This is the final result. This is one of the MIT curriculum developers that made the original
tutorial. And yeah, basically it says that we will be using the personally much classified
extension that was developed by that research of MIT. And the first step is loading, turning the
model. And then we have to upload the model. To upload the model, we go to this section over here
on the media. We select the just downloaded model file. It should be here. And now it's
uploaded. It's in the asset file of the app. And we can just change the model of the image
property, personal image classifier to the now loaded new model. And we have just loaded the
model properly. To give an overview of how the app is going to look like, there is going to be
this status label that will tell the user when the app is ready to work. Now it says loading,
because it's the initial state. It will let it go to the ready state and it will just identify
the faces. We have these two bars that will be showing which percentage of confidence we are
that we are looking at the baby or not. This is going to be the live image from the camera that
I just showed before. And these are the interaction buttons to start the classification,
to toggle the camera from the front to the back camera. And we have here the happy baby in this
case. So uploading the turning model. This is the sequence of events that I was just talking about.
First we start the app. The app will show to ready as soon as the classifier is ready to
start working with the app. The user will press the start button and then the personal image
classifier extension will keep classifying the live stream video from the camera continuously.
And once they have identified the result, if there is a higher confidence that the me
model is working, we will show the smiley face, otherwise the baby will just start crying.
So in this template, there is already a set of blogs that are available to speed up the process.
And we can go here that we see the one personal image classifier error. So that means that if for
any reason the personal image classifier shows an error, maybe because there are some missing
things on the phone or whatever, we will set the status label text to the actual error that we
return from the image classifier. Once the image classifier is ready, we will enable the start
button as well as the toggle camera button. We will set the status label text to be ready so the
user knows that they can start using the app. And we will set the text boxes of each classification
group to the previously defined labels me and not me in this case. If the user presses the toggle
camera button, we will be changing from the front to the back camera just every time that they press
so we can use the front selfie camera or the back normal camera. And once the user presses the
start button, if the personal image classifier is already classifying an image, we will just
stop it and we will show the start button with the start text. Otherwise, we have to start the
classification. So to do so, we just invoke the start continuous classification method and we
change the text to stop because we will be changing the the button interactions. And that's the quick
overview of the code that is already available in the in the app. So how does the image classification
work in MIT Preventor? Well, we have this big block that is the when personal image classifier
has received a classification succeeded. We will receive this result variable. This result variable
is a dictionary that to just give a little high level overview of what is a dictionary. It's a
key value list of elements. So if we have two different groups, me and not me, we will receive
me equals a specific value, not me equals a specific value. If we have three groups, we have
one, two, and three that they each equal to a specific values. This is a little example of how
it looks like. So we have key father equals this value, key model equals this value, then equals
this value, etc. For the image classifier, a specific example, we will have something like
this. We have me with this value and not me with this other specific value.
So how we can retrieve a value in the dictionaries area, in the dictionaries block area, we have
get value for a specific key. And we will be doing something similar to this. So we have the
original dictionary here. We are building it in this area, make a dictionary me and not me. And
we will be getting the value of the group that we want to use right now. In this case, it's the
me example. If we want to take the not me, we just have to change this label to not me. And if
we are using the wrong model because the groups are not the same, we just return a zero because
we cannot classify that group. So let's get into it. By default, the tutorial will provide this
block that is some variable, some me confidence level, and we have to complete them using this
block. So to do so, we will take the get for key in dictionary block. We join it to the me
confidence block. We remove, nice, get value for the key. And we will take from the text blocks
an empty text block to touch it right here. And we can type me. So we can get the me group
into the me confidence variable. The dictionary is the result. We can just attach it here.
And if not found, we will just returning an empty zero value. And for the not me confidence,
it's basically the same. So we can copy paste the blocks. We attach them to the not me confidence
area. And we just have to prepend a note in front of the me. And now we have just defined
that me confidence variable that can be accessed like that. We'll have the percentage of confidence
that we are looking at the baby. And the not me confidence, it's the opposite. It's how
confidence we are that we are not looking at the baby. The next step, the interesting variables.
And now we just have to recap what do we have to do in the app. So in the app, we have to first
update these labels here with the percentage. And we have to update these color bars with the
correct confidence levels. We can do that by going to these components, to these two horizontal
arrangements. And we have percentage one, bar graph one, percentage two and bar graph two.
Percentage one, we can update the text to the percentage that we are showing.
One second. There it is. So the value that we return from the dictionary goes from zero to one,
but we want to return a percentage which goes from zero to 100. So we will take this me confidence
value and we will multiply it by 100. So we can get the zero to 100 range. We just join it right
here with the math number and we multiply it by 100, 100. But we will be missing the percentage
sign. To get the percentage sign, we can use the text blocks with the join block. So we can join two
text together and we can just create a new percentage symbol like this using the percentage
symbol. And this is for the percentage labels. For the bar graph labels, we will be pledging with
the length of the actual graph, bar graph. To do so, we have the width percent block that can
modify the width according to a percentage. And we already have defined the percentage right here,
so we can just copy paste these blocks and attach them to the width percent. And this is for the
me group. For the not me group, we can copy paste the percentage one, which changes to percentage
two, and we change the me confidence value to the not me confidence value. And for the bar graph,
it's going to be bar graph two right here. And me confidence changes to not me confidence.
And with that, we already have all the sequence of events for the labels updates.
We can just go to the next step and confirm that we have defined it correctly, which is the same
result. The next step is the fancy image change that if we think that we are looking at the baby,
we will show the happy face, otherwise we just show the crying face. We will be using the if
then logic. So go to the control blocks and we just take the if then otherwise block. We append
it here. And what will we do is we will be comparing the me confidence value to the not
me confidence value to know if we are looking at the baby or not. We go to the math blocks,
we pick this comparator block, attach them to the if statement, and we are going to be changing the
comparison to higher or equal because we will not, we don't worry about the equal in this case,
we just want the higher or equal. We take again the me confidence variable, we compare it here,
and we take the not me confidence value and we compare it right here. Then we will be updating
the background of the of the app, which is available in the screen one. We take the background color
block, we attach it here, and the tutorial already provides the example colors. So I'm just going
to be dragging this right below so I can have them more easily accessible right here. And I can
just join it here. And for the baby images, we have two images available here, happy baby and
sad baby. So if we think that we are looking at the baby, we show the happy baby. So we use the
visible block. And if not, we just hide, sorry, if we think that we're looking at the baby,
we hide the we hide the side baby face. We go to the logic blocks, we take the true so we can set
to true to visible, we can set visible to true, and we set visible to false for the sad baby
like that. And we just join it. For the case of me confidence being higher than not me confidence,
for the opposite case, when we are not looking at the baby, we just change the background color to
this pink color. We hide the happy baby face like that. And we show the sad baby face
just like this. And now the app is finished. So here we can just check the final code,
which is exactly the same as we have right here. There are other possibilities like we can just
implement a classifier using a different person. But to show how this works, we can use the MIT
company map that is available on the Play Store. Let me just show my phone again. Here it is. So
you can just go to the Play Store and go to MIT App Inventor, search MIT App Inventor and you have
right here the company. You can open it. Yeah, you can just ignore this warning. It works without
Wi-Fi. Continue without Wi-Fi. And over here you can connect the AI companion. And now I can scan
the QR code like this. I'm sorry, it just disappeared. It takes a few seconds to connect.
Let's see if it was faster than tonight.
Now it's loading the extension, the personal classifier extension into my phone.
Like this works with Wi-Fi, mobile network. It doesn't have to be connected. It's just connected
because I'm just mirroring the screen through that cable. And I see here the layout of the app.
I can see that it shows ready. So I can just toggle the camera to be the from one.
And I can look at the camera and I'm just going to be start. And there is a higher confidence
that I'm looking at the Wi-Fi. Just put a hand in front of it. It's just crying.
And yeah, that's it. Later, if you want to build any other apps, you can export it
to APK files. So you can start it on your phone or 200 app bundles if you want to distribute it
through Play Store. But yeah, this is just a very high-level introduction to artificial
intelligence in Inventor. You can just build any kind of classifier, for example, to classify trees,
flowers, to classify even people. For example, for a faculty, if you want to build an app that
recognizes people in your class as a game, you can just use Inventor and build any kind of app.
Thank you so much and hope that it was useful for everybody.
Thank you. Any questions?
Do you mentor a technobation team? No.
Sorry, I'm a software engineer. I just contributed to Inventor. I started as building apps and then
I transitioned to open source. I participated in Google Summer of Code, like this option to export
a 100 app bundles was my project in 2020 for Google Summer of Code. But yeah, I'm more like,
more technical than actually teaching to kids.
Any other questions? What's your experience with the relation between the number of pictures you
have to submit to your classifier and your accuracy? That's a very good question. So for the linear
example, he was asking like, what's the experience with the amount of images that we are going to
be using for the classifier? So I haven't really tested it right now. But we have seen that if we
go higher than 10 images for each class, for each group of images, we'll have really good results.
In this case, because I was just turning very fast and using just a few number of images,
you can see that the confidence levels were a little bit like 80, 20. But if we provide more
than 10 images for each class, we should be able to get around over 90, 95% of confidence for each
number. I'm not sure if there are any questions from the chat. Let me just check.
What do you capture? Is that just for your face or did the learning, what was learned?
It recognizes, it depends on what you are training, because in this case, we are just
providing a very specific gestures. It's training my face like any face looking at the camera,
or a hand in front of the face. By default, the model that is available, that is here,
this model is turned by Salim. It's the example guy that is at the beginning of the tutorial.
And I just tested it last night, and it worked with me because it recognizes the gestures,
not the faces. If instead we train recognizing people, we will all be looking in the same way
at the camera. So it will just go for specific facial, how do you say, facial features.
In this case, in this case, it will work. You can just try if you want. We can try.
Yeah, it should work.
Can you, it's going to be a little bit tough, but Mark, can you,
can you just try with Mark, for example? Toggle camera. Toggle camera.
Yeah, and just try with, and start. Press start. It's a happy face, so if you put a hand in front,
it's a sad face. It's recognizing the gestures.
So can you also train it to recognize specific people?
Yeah, it can be trained, but in this case, because the higher difference was the hand,
it's just looking for the hand in model. But if you don't show the hand, it will look for faces.
Yeah, it's a, it can be a fit because it's just, you can just use this website and
fill any kind of models. The only restriction is that it has to be an MLD file,
but yeah, it can classify any model basically. No problem. Any other questions?
Well, I think we can leave it here. Thank you so much.