All right, so welcome everyone.
Good morning.
I'm happy to be here.
It's 3 AM back home.
So I'm really looking forward to talking to you about MIT app
inventor.
And I'm being joined.
My name is Evan Patton.
I'm the lead software engineer for this project at MIT.
And I'm being joined by Diego Bishwas, who
are contributors to our open source project.
And they're going to share a little bit
of their experiences working with MIT app inventor.
And just to get started, for those of you
who don't know what MIT app inventor is,
it's a web-based platform for building your own mobile phone
applications.
So you have a kind of drag and drop.
What you see is what you get interface for designing your app.
Then we have a block environment for coding your application.
So in this particular example, we're
coding up a Bluetooth connection to collect some data
from a microbit.
And then you can see that app running on your Android or iOS
device.
As we see here, we get some data that we can then
graph on our phone.
So app inventor has been around for a number of years.
It was started at Google in 2008.
And then it was moved to MIT in open source
under the Apache license in 2011.
We have almost 21 million users.
And probably by the end of this month,
we will hit over 100 million projects
that people have created with app inventor.
And at our peak usage, which is typically in May,
we have about 1.4 million users a month.
So contributions to our project go a long way.
And the mission of app inventor is what
we call computational action.
So the idea is that you can really
use a tool to build a meaningful application for solving
a problem in your community or in your family life
or in your country even.
And we've got a number of examples
of what that looks like.
So back in 2014, there was this program
called the Innovative App Challenge.
And some students built an app to help their blind friend
here navigate his school in Texas.
And they won this competition.
And they even got to meet President Obama at the time.
And this was an app that they built with app inventor
to try to improve somebody else's life.
This is a similar example from Kentucky,
where this young lady here, her grandfather,
was suffering from Alzheimer's and was
forgetting to take his medication.
So she built this app called Farm Alarm
to help him remember to take his medications.
And you can see an example of what that looks like on the right
there.
And it's not just obviously restricted to the United States.
We're used worldwide.
These are the Darvi girls in India.
And so they made an app that helped
them to schedule time to go and fetch water at the community
well to try to make it easier to schedule their lives
around this very important thing that they
need to do for their families.
And then similarly, a little bit closer
to where we are in Moldova, a group of girls
made an app to help track water quality.
And so they were able to collect a bunch of data points
and visualize them on a map.
And you could say, the quality of this water is high,
medium, or low grade.
So that way, you would avoid going to places
where you wouldn't be able to get clean water.
And so one of the reasons why we can do this
is because one, people in the community, volunteers,
have gone ahead and helped translate App In Mentor
into a number of different languages.
We're up to 20 languages now.
And maybe you would be willing to help us make that even higher.
And so you can go into App In Mentor
and you can code an app in your own language
because the entire interface is internationalized.
We have a website up in the top right there,
AppInMentor.mit.edu, where you can go and access
a number of tutorials.
And we've got them in a number of different themes.
So obviously, there's a lot of beginner stuff.
But we're doing a lot of advanced things, too,
where you can learn more about data science and visualization,
do some work with internet of things
and connecting to nearby devices like Arduino and Microbit.
And we have a whole suite of different examples
of using artificial intelligence, including things
like chat.
GPT.
And in the case of using the Android version,
we also have a number of different extensions
that you can use, or you can develop your own, again,
because App In Mentor is open source.
It makes it very easy to extend it.
And we have, at the end, a link to the slideshow.
So you should be able to access all this content as well.
As I mentioned, we are open source under the Apache license.
You can get to us on GitHub at mit-cml.
And we've had over 180 contributors to the project
over the last decade.
Now, I'm going to hand it over to Vishwas.
He's going to give you a little example of how App In Mentor works.
So you can see it.
And if you like or you're available,
I don't know where's Peter.
Do we have any more slots available for the workshop?
Do you know?
Five.
Five?
OK, so if you would like to learn more and actually try
building your own app, we have a workshop that's
happening at 12.30 in J Building.
Do you need an instance for children between 7 and 7?
Yes.
7 to 17.
Right, OK.
Great.
So now, thank you, Evan, for an introduction to App In Mentor.
And let's actually go and dive into it.
No.
Is that good?
Yeah, OK.
So you can follow along by going to code.appinventor.mit.edu.
You don't have to have a Google account for anything.
You can just log in and start doubling with Android apps.
The example I'll get started with is Hello, Per, because it's
trivial to follow along with.
And it's also a fun app.
The premise is simple.
It's a picture of a cat.
And when you click on it, it says meow.
And we use this example a lot of the time because it's easy to
get started with.
And as I said earlier, it's a fun little app to play with.
So as an intro to the interface, on the left, you
see there's several components.
And components are things you can drag and drop into the app.
So let's say if you want to do that, you can do that.
Let's say if I want to add a button, you can drag and drop
it here.
I can get rid of it from here.
And so on.
And there's loads of components, stuff you can see on
the screen, stuff you use to put together things, multiple
things that you can see on the screen, stuff to access like
the camera or play sounds, translate text.
If you're making a game, there's components to do that as
well.
And then you can see how there's a variety of components
to play around with.
If you have a Lego Mindstorm, there's components for
Mindstorms and a few other ones that Evan will talk about
in a bit.
And on the right, well, let's get to the main part.
That's the phone itself.
The great thing about app inventories, you can see what
your app's going to look like.
And you can move things around.
You can add stuff.
You can edit stuff.
So this is a button with a picture of a cat as its image.
As you can see here, I can set it to none.
And it goes back to nothing.
And I can set it to the cat again.
So it's instant.
And it's really easy to follow along with.
Another great thing about app inventor is to the left, you
can see something called the emulator open.
If I connect to that, it might take a moment.
But let's be patient.
Right, so while it's booting up, the emulator is great
because it lets you view your app on your phone.
Well, it's on the laptop now.
But you can also install the app inventor companion from the
Play Store or the App Store.
And you will see your app live.
And any changes you make will be reflected in real time,
which is great for prototyping.
There you go.
Now, if I add something else, it will drop here.
And I can interact with it like I do any other app.
Now that's how you design the app.
Let's try and get some functionality in.
So we want the cat to meow when the button is pressed.
And the way we do that is by using blocks.
Now, at the top right, you see there's two parts to App
Inventor.
One is the designer, where you design stuff.
And the blocks area is where you actually build the
functionality of the app.
So HelloPur is very simple.
Earlier, we saw that the cat was basically a button.
So when the button was clicked, you call Play on the sound.
And what is sound, though?
You'll see there's nothing called sound visible on the
phone screen.
But below, there is a non-visible component called
sound.
And that's a non-visible component is exactly what it
sounds like.
It doesn't show up on the screen, but it lets you do
things that are important, like playing a sound.
Now, what sound do I want to play?
I can set it from the source, which is meow.
And if I want to know what meow is, there's the media
panel.
And I can click on meow.
I don't know if this is going to show up, but I can preview
it.
OK, plays on here.
Great.
So meow is the sound that we want to play.
And OK, there you go.
And when the button was clicked, just say, OK, I want to
play the sound.
And also, vibrate the phone for half a second, so 500
milliseconds.
And if I do that, OK, it says meow on here.
I'll turn it up.
Sure.
Let me also hold my mic here.
There you go.
All right.
Let's also maybe try and make a change to the app while
we're still here.
I will use the Notifier component just to show
notifications.
And what I want to say is when the button was clicked, I
want to show the text meow as well.
And that's also fairly straightforward.
All I do is so you can see all the blocks available by
clicking here.
There's lots to work with.
Yeah, you can pretty much do anything you want in your app
by using a combination of these blocks.
It's just like building with Lego.
You start real small, and then add stuff on to it and make
your whole app.
For now, all I want to say is I want to show an alert.
And we'll add this to the bottom.
So after the sound has played and the phone has vibrated, I
want to show an alert.
And I can go to the text category, and I will grab an
empty text box and connect it here.
So you can see how it all joins together nicely.
And I want to say meow.
If I try it out, there you go.
So as you can see, it's really easy to get started.
It's really easy to start making changes.
And overall, at least in my opinion, it's great to
actually get into how Android apps are made and make one for
yourself.
All right.
OK, thank you very much, Voswas.
So I briefly wanted to talk about the source organization
of app inventors, considering that many of you might be
willing to contribute to the project.
And therefore, it's helpful to kind of understand how the
project is put together.
So as Voswas mentioned, there are a lot of components in
the system.
We have 96 different things you can use for app inventor
spread across 13 different categories.
All of these are written in Java, and they link to the
native Android libraries.
This is not some sort of emulated environment.
You're actually running real code on a real phone when
you're using it on your own device.
And as we mentioned before, you have extensions where if we
haven't implemented something, you can implement it
yourself and add it into app inventor without even having to
build the whole source tree.
We have iOS, which is a more recent addition to the
project.
We open sourced it last year.
And currently, of the 96 components, 63 have been
implemented for iOS.
We have, in the past, had Google Summer of Code projects
to work on this.
We're hoping to have more of them again this year.
So if you're interested in programs like Google Summer
of Code, feel free to come after and talk to me about that.
And all of these, again, because it's native code, are
primarily written in Swift.
I'm going to skip this.
Obviously, we saw how components work in the demo.
We use Blockly.
And I believe we have Christopher Allen here.
He's going to speak about Blockly a little bit later in
the day.
But Blockly is this wonderful project that allows you to
build your own environments for coding using a block
language.
And so you provide the back end.
I think they have five languages they currently
support.
We have our own language that we've developed for App
Inventor that we generate.
And this basically is responsible for creating,
taking your blocks, turning it into code that then executes
on the device.
And then we use App Engine to host the front end.
So like the editor you saw when it's running in the
browser, the client is written primarily in Java using
GWT.
And then the back end is talking to a bunch of
different services, including what we call the Build
Server.
And the Build Server is responsible for taking your
application and actually converting it into a real
running app.
And we saw the companion version where Visuals is able
to make changes in real time and see how that works.
But when he closes the emulator, he loses that work.
Well, he loses the running version of the work.
Obviously, the project's still stored on the server.
But you can compile your application as an APK.
Or it's not quite released yet, but we have a beta version
that allows you to build for iOS as well.
And then you can actually bundle your app and then run it
on your phone, have it there.
I've actually got a copy of HelloPer that we just did on
my iPhone, for example.
And so that allows you to kind of take your project with
you.
And we run 21 Build Servers.
And we can support 168 simultaneous builds in the
cloud for supporting app and metric users.
Now, the project structure is fairly complicated.
You saw a lot of these things in the examples already.
Assets, where you have your different media files.
And additionally, if you have extensions, you've got
extensions or considered assets of the project.
You've got your source files, which are your blocks and your
design, and then project properties.
And so internally, App Inventor represents us all using a zip
file.
You can poke around and try it out if you want to learn more
about the internal structure.
But it's very helpful if you are working on the App Inventor
code to know how the files are laid out so you can edit them
and try things out at the source level.
Now, I put into slides here.
Is anyone interested?
How many people here are contributing to open source
projects already, or are interested in contributing to
open source projects?
All right, we've got quite a few.
So if you're interested in contributing to App Inventor,
we do have some example instructions for how to do that.
Obviously, because this is a project that's building for
multiple platforms.
We've got web.
We've got Android.
We've got iOS.
It's a little bit more complicated than a regular
process where maybe you're just building for web.
And so I do have this here.
I'm not going to go into much detail on it, just because it's
got quite a few steps.
But as I said before, we'll have a link to the slides at the
end so you can go back and reference these.
So let's talk about contributing to the project,
which is really why we do open source.
We want to be able to contribute to these projects and
make them better.
So one way you can contribute is you can help us with
translations.
So as I mentioned earlier, we support 20 languages already.
But these are all contributed by volunteers.
And you can sign up.
We use WebLate.
So weblate.appinventor.mit.edu.
And then you can email my colleague Susan, who will
set you up with a specific language.
You just need to let her know your username and which
language you plan to contribute to.
The second way you can contribute to App Inventor
is through GitHub.
So we actually have, again, because we have this
complicated build process, we have two different branches
we do development on.
So the first one is called UCR.
UCR represents all of the changes to components that are
going to run on the phone.
So that's if you're making changes to the Android
piece of the software, or changes to the iOS portion of
the software.
And the reason we do this is because when we do a release
of a new component, we have to put it through the app store
process.
We have to put it into the Play Store process, which can
sometimes take a couple of days.
And so we tend to group the component releases into larger
chunks to deal with the approval process there.
All of the changes go to master.
And then we build from master pretty regularly and deploy
usually once a month.
Or if there's immediate bug fixes that need to go out,
we'll do it as frequently as needed.
And then when we do a component release, UCR gets merged
into master, and then we release the whole bundle
altogether.
We have issues that are marked help wanted on GitHub.
So if you are new to contributing to open source, we
have some smaller issues that are easier to sink your teeth
into and get started.
And we tag everything, which branch to start from and all
of that.
And we will help you if you have any problems making
pull requests.
And I mentioned before, and I want to invite Vishwas and
Diego to really talk about their experiences with Google
Summer of Code.
We have participated in Google Summer of Code for a number
of years now.
I don't know the exact number, but I think it's six or seven.
We are currently waiting on the 2024 decision, but we're
hopeful.
And if you'd like to learn more about contributing through
Google Summer of Code, we have a community site where
people can ask questions about App Inventor and there's a
whole section dedicated to folks interested in Google
Summer of Code.
So we definitely would encourage you if you are
eligible for Google Summer of Code to consider applying for
App Inventor.
And then Vishwas, your first one.
Thank you.
So a bit of background about me before I go into why I did
Google Summer of Code for App Inventor.
I started dabbling with App Inventor when I was 12, 13, I
think, just starting small with probably the
Hello Pro App as well.
And yes, slowly got into making extensions for it in Java
and eventually got around to actually making changes to the
code base itself.
So it's been a long journey.
But I think over the years, I've come to be able to mess
around with the code base enough to actually make a few
meaningful changes, at least.
And one of those was my Google Summer of Code project, which
was in 2021, which was to change some of the internals of
App Inventor to make it more modular.
So at the moment, if I go back to the demo, you'll see that,
well, it's not really that visible.
But the whole thing is a monolith.
It's one massive bundle that's generated.
And as it might be evident, it's quite hard to maintain.
It's hard to make changes to.
And it's also not really that responsive.
So it doesn't work great on mobile.
So my project was to modernize it a little, make it look a bit
prettier as well, I think, bring it to this decade.
And also make it such that it's going to be easy moving forward
to make further changes to the UI.
And that was back in 2021.
And in 2023, well, last year, I also mentored a project that
was to add a new UI to collate all project-related properties
into a dialog.
So if I go back to App Inventor here, this should be merged,
I think.
So there's some properties.
So let's say buttons have properties like font size and
background color.
But what about properties of the app itself?
And initially, they were in screen one, which in hindsight
probably doesn't make a lot of sense.
But now they have been isolated into a separate dialog.
So this was work that Aaron Modi did as part of his Google
Summer Code project, which I mentored last year.
I think what I want to get across is really easy to start
contributing and really easy to sink your teeth into whatever
you fancy.
And go from there and become one of the main contributors
to the project.
So the barrier to entry is incredibly low.
And I encourage all of you to start maybe just look at the
source code if you can, and then go from there.
I would also like to invite Diego.
He's also done a Google Summer Code project and also
mentored projects himself.
So yeah.
Thank you.
This was a good morning, everyone.
So my journey is kind of similar to this was I started
programming with App Inventor back in 2014 when I had an idea
but I didn't know how to code it.
So one of my university, high school professors told me, oh,
you can use App Inventor and build the app without coding.
And that's how I got into App Inventor.
Then 2015, 2016, I started contributing to the project.
And in 2020, I was also a Google Summer Code student when I
was developing a new format to export the app, which is now
available for App Inventor.
So when we are building the app, we have, sorry, yeah.
Oh, it's not?
OK.
Yeah.
So when we are exporting the apps, we usually have the
typical Android APK file, which is the standard one.
But then Google added the Android App Bundle distribution
format for Google Play Store.
And I have basically implemented these new formats so
people can distribute the apps through this new format into
the Play Store, which is usually more optimized for the
Google Play Store.
Thanks, Evan.
Basically, just to give another view of the difference in
a standard APK file, basically all the assets, all the
kind of languages that architectures are bundled into
the same file with an AAB file.
Instead, the Google Play Store will automatically pick the
ones that are needed for the mobile phone.
And this overall size is going to be way, way smaller.
It's basically Google chooses which assets to pack and
distribute to your specific mobile phone.
And as part of this project, I was also refactoring the
computer, because it was also a monolith of 3,000 lines of
code.
So just to give another view of the changes, this is how we
basically invoked the compiler before the best passing
tens of parameters into the same constructor, which is
kind of risky sometimes if we mess up the order.
So this is the final solution.
It may seem a bit more complex, a bit more step through.
But in the end, it's easier to understand, because all the
parameters are properly named.
And thinking about iOS, the compiler now has a step-based
process.
So for the iOS later on, when it's going to be available,
the build server, this same format of adding new steps
will be conditionally added for the iOS build procedure.
And last year, Zimas Beach was also a mentor.
In MIT, I was mentoring Drubstree Vastava, where he
was implementing some versions for the Android components
that were not available for the iOS.
So that's why last year, because it was open source, we
were able to include this iOS support for our Google
Summer of Code project.
And as Alishia mentioned in Google Summer of Code and
Google Blockly, so we had a student in 2020, Becca Westberg,
that was starting to contribute to a pin mentor, adding some
enumeration blocks for a pin mentor.
And now Becca is basically working and contributing very
actively to the Google Blockly project.
And now, Evan can give more overview on the teaching and
mentoring from App Inventor.
Yes, so we've talked a lot about App Inventor in terms of
the technical side of things, but I did want to offer as
well that there is a lot that's available for you if
you're an educator or maybe a parent who's interested in
starting an after school club or even students.
I mean, we've had a number of students who have come to
MIT as undergraduates and will come work on App
Inventor.
And they say, oh yeah, I started a high school club
because we didn't have programming classes and I
wanted a way of learning to program.
And my friends, we wanted to learn together.
And we found App Inventor and it was great.
So we have on our website a number of curricula, both at
the middle school and high school level.
So kind of looking at ages maybe kind of 12 and up or 11
and up that are available.
It's mostly in English, but we do have some in Spanish.
And of course, again, we welcome very greatly
contributors who would be interested in helping us make
that material available in more languages.
There are a number of programs out there.
So many of the examples that I talked about earlier came
from a program called Technivation, which is a
program that helps mentor young women to have them work in
groups.
They learn how to do things like develop a business plan and
build an application to try and address a need in the
community.
And they have a big competition, a bunch of regional
competitions and there's a worldwide competition where
the winners go, I think usually to London, and they
present their app ideas.
And so that's a wonderful way if you would like to mentor
those types of programs, I can get you in touch with people.
I know folks like Peter have worked with groups like
Coder Dojo to try to make these programs available for
students in Europe.
And there are many others in all parts of the world.
And we have an app and mentor foundation now which runs a
seasonal apathon, we call it, where you build your own app
and compete.
And these are happening all around the world now.
OK, let's continue.
So the last thing I want to kind of leave you with is a
vision of the future and some of the work that we're doing.
Obviously, we're producing this project that is used by
millions of people, but we're also a research group and we're
trying to think about what does the future look like in terms
of computing and in terms of programming.
And so we have this new project we're working on, built on
app and mentor called Aptly.
And the idea behind Aptly is we're trying to take the ease
of developing an app with app and mentor and combine it with
the kind of explosion of large language models that we've
seen over the last two years.
And so the idea is that you can provide a description of an
application in natural language and then it uses GPT-4 to
generate a textual version of the program.
And we are then able to parse that into an
app and mentor project.
And so in our mission of trying to encourage computational
action, our hope here is that maybe you don't know how to
program yet, but you still have a really good idea of how you
can solve a problem in your community.
So could you just describe the solution to the computer and
see if it could make something that would work for you?
And then you can actually, this is just the first step in the
process.
You can collaboratively edit the application with the AI, make
changes, and then instruct it based on your changes, other
things you want it to do, and so on and so forth.
And if you're interested in learning about this, there's
a link there, appinventor.mit.edu slash aptly.
We've got videos and other things of how this works.
And we're continuing to do research on this.
So when we first announced this about two years ago now, I
think, Peter asked if it only worked in English.
And I said to him, I don't know, to be honest with you.
And so he gave me this example in Dutch.
And this is the app that it made on the right-hand side.
And of course, we thought this was really interesting because
obviously, many people are used to writing programs in
English.
And in fact, one of the times I tried making an app with
aptly, GPT came back and said, well, you can't make an app
in Italian because Italian is not for programming languages.
English is for programming languages, which I found to
be a bit funny.
And then I kept instructing it to do it, and eventually I did
it.
But I did think that, but it's not true.
Obviously, I don't understand Dutch.
But the LLM certainly seems to have some understanding of it.
I was doing my particular test in Italian when I did it.
And it kept telling me in Italian it couldn't do it
until I complained enough.
But it can take these things.
So you don't even have to know English anymore, potentially,
to develop an app.
The system can do it for you because it has these latent
understandings about what it means to build a program.
And it's seen examples of other languages, so it can
figure out the mapping of that.
I have noticed that it doesn't get it quite right.
There is a bug in this program, because this is saying it
wants an index, and selection index would be the
appropriate property here.
But we'll gloss over the fact that it made a semantic error.
That's OK.
And then the other thing we've been trying to do is trying to
teach it about concepts like games.
So I gave it these two prompts to try to start making a little
game world.
So I wanted a 2D platformer.
So you see I got some platforms at different levels.
And then I said, OK, well, maybe I want my main character
to be this little lizard with a space helmet on.
And so you can see him up here on the top left.
And so it's really quite fun to play around with.
And we're continuing to do research on this.
So it hasn't been open source, but we're looking to open
source it soon and try to get more people playing around
with these types of tools and trying to understand really
how we can take these large language models and combine
them with something that's as easy to use as App Mentor to
make some really interesting programs.
And so with that, I just would like to say thank you all for
coming here bright and early on a Sunday morning to listen to
us talk about App Mentor.
We have a number of different links, as I mentioned.
You can access these slides at fosdm24.appmentormit.edu.
And then, of course, you have access to the sources and all
of the resources that we've made available worldwide
for this project.
And if you have any other questions, you can also feel
free to contact me by email or by email.
Or we can talk for a little bit now if you have any
questions from the audience.
Yes.
They can make the app any language.
How technical does it have to be?
Or how do you express the app to use like a few terms somehow?
Or do you need to know a little bit about what it has been
or what it's known as?
It varies.
Obviously, the more technical you get, the better it tends to
be in terms of what it does.
But one of the fun things that we've been really playing around
with is kind of giving it things that are purposefully
ambiguous to see how it responds.
So one of the early examples I would do is I would start with
Hello, Per, which we saw earlier.
And then I would instruct it, when I click on the cat,
make it bigger.
Right?
There are lots of different ways you could interpret that
statement.
And in fact, if we run this multiple times, we get different
code outputs.
So in one example, it would add 50 pixels to the width and the
height.
In a different one, it would multiply it by two and things
like that.
But it does seem to associate, OK, it knows cat and it can see
that there's a button that is associated with the thing called
kitty.png.
And I imagine that's how it figures it out.
Because we also have a version where we have both cats and dogs
and then try to get it to understand the difference between cats
and dogs.
And then, of course, bigger, it seems to have many different
interpretations of what bigger could be.
Another fun one that I've been doing, which is actually really
cool, and I don't know how it does this, I make an app where I
have it with the add and remove button to add some content to a
list.
And then I say, make the colors of the buttons match their
purpose.
Right?
And because one button is called add and one button is called
remove, it makes the add one green and the remove one red.
And then I say, well, this is not colorblind friendly.
Make the colors be more colorblind friendly and then adjust
the colors, which is particularly interesting because the entire
interface is text based.
Right?
So there's no color here, but it seems to have some notion about
colorblindness and how to change things to make things more
colorblind friendly.
And so then it comes up with more muted colors or changes, you
know, some properties of the colors.
And it's really quite fascinating to see.
Actually, I wasn't planning to do a live demo of this, but let's
try it.
Oh, there's another question.
Sorry.
No, in the chat.
Oh, in the chat.
Someone can ask a question.
Why do we have to create a Google Cloud to use MIT App
Inventor?
So that's why I was suggesting, so if you use the version of
code, sorry, we'll go back to the app thing in a second, but I
do want to answer this person's question.
So in code.appInventor, when you come to the server, I got a
log out.
Yes, yes, yes.
When you come to the server for code, there will be this
continue without an account button.
And if you click on that, you can bypass the need for the
Google login.
And what it does instead is it generates a random forward
code.
And you just save that.
That's basically your password to access your account.
And so you write that forward code down.
And then when you come back, we just saw it.
There's a set of four boxes where you put those words back in.
So you don't need to have a Google account anymore to use this
particular service.
There are other services we offer where you do need a Google
account.
And right now, that's just for identity purposes.
So we have a way of linking your projects to you.
In this particular case, if you use this anonymous thing and you
forget those four words, we cannot help you because we have
no way of identifying you or your account.
And so that's really important to save those four words if you
use this anonymous version.
But yes, there is a way of using app Inventor without the
Google account.
So that's how you do that.
Yes.
One other question about phone generation again.
Assuming that say you know what you're doing and you're just
using it as a help, is it possible to focus on parts of
the generated app and actually enhance it in one fashion or
another without regaining the company?
Yes.
So this is our hello per example.
So let me just open that up so we have a good starting point.
Oh, yeah.
You can make it edit the pictures and stuff.
I forgot to mention that.
So here's our example.
Yeah, just the hello per.
And then there's this little pencil icon down here in the
bottom right.
And that brings up the editing thing.
So I'm just going to come in here and say, let's do the
example I said earlier.
This is now a picture of a dog.
Let's say when I click the dog, make it bigger.
And the next version also has a microphone input, so I
wouldn't have to be typing that.
And so, again, we can watch.
App Inventor also because we do a lot of logging in the
console.
So if you want to see a fun way of how it all works.
So now, so see, here's something that's done that's
interesting.
So I thought that it could create two button handlers.
Now, people who have used App Inventor know that this is a
no-no.
So I'm just going to combine those two.
But this is part of the interesting part of working with
the AI is it doesn't always get things right.
So I've gone ahead and I've taken it.
In this case, it's decided that making something bigger
implies adding 10 pixels to each of the dimensions.
But it's come up with that code.
And then we're able to take the textual representation of
that code and turn it into the block representation so that
you can see how it works.
I think it also made the button smaller in the designer
compared to previously.
Let me move this over to the side.
And then, of course, so we could say, what do we want the
button to be a picture of?
It could be, I wonder if it knows about FOSDEM.
Could we do something about FOSDEM?
Okay.
Let's see if it knows about FOSDEM.
That could be fun.
This is also one of the interesting things about this
technology, right?
And I'm sure people here have played with GPT-4, any of those
kind of generative AI systems.
So you know that occasionally it doesn't kind of either come up
with what you want or it does something completely random.
In other cases, it's sometimes very good.
In this case, it might not have done anything.
That happens sometimes too.
Yeah, it was upset about something, so it rejected my request.
Again, this is research-level stuff, right?
This is not a product yet.
That's one of the reasons why we haven't released it open source.
We're still working out some of the kinks.
And one of those kinks is that it doesn't have much in the way of
error handling.
Christopher.
You mentioned that LLM is generating some language which you then
personally just speak slightly more about that if we...
Sure.
So when we first started this project, it was the LLM at the time
that we were using was the open AI codex model, which no longer exists.
And when we were designing the language, we had a couple constraints
on it that we wanted.
We wanted it to be a one-to-one correspondent with App Inventor.
And basically, for those of you who maybe don't have a computer
sensor, the idea here is that for every single block or function
or whatever in App Inventor, we wanted it to be an operation in the
language and vice versa.
So we didn't want, for example, to take JavaScript and then have to
worry about Lambda functions because App Inventor has no concept
of a Lambda function.
So we...
But we needed something where we wouldn't have to do a ton of training.
So we started from Python.
We said, we'll make it look like Python.
Obviously, it will have had training on a lot of different...
Oh.
Yeah.
Oh, yeah.
So...
Oops.
Yeah, so if we...
Oh, come on.
Sorry, it's being a little bit finicky.
The computer's being a little finicky.
So it's got to construct something.
There's no new keyword.
You just use the class name.
It's got named arguments.
It has...
This is some of the differences, though.
There's no def keyword like there is in Python.
We say, for event handlers, they begin when?
Like they do, at least in the English version.
And then you have the event name and the...
Sorry, the component name and the event name.
This one doesn't have procedures,
but procedures are defined with the two keyword,
and globals are created with initialize.
And then there are relationships like in App Inventor,
every component other than the screen has a parent.
So translator belongs to screen one.
If you had an arrangement,
you would have the thing belongs inside of its arrangement.
And so...
And there's, as I said, there were two reasons...
There were a couple reasons for this.
One was we figured it seemed a lot of Python.
Therefore, it should recognize a lot of these types of things.
And two, we wanted something that didn't have a lot of extra syntax,
because one, we thought it would be difficult to get it right.
Our original proposal was,
could we generate the JSON that underpins the design,
and the XML that underpins the blocks.
But XML is very verbose,
and you have to keep it balanced,
otherwise it doesn't work.
JSON, in terms of verbosity,
obviously, if you got parentheses wrong, we'd be in trouble.
But that was that.
And then the last thing was, because we're paying for it,
we wanted to have as few tokens as possible,
because you pay per token.
So something like Java or C,
if you've got a bunch of curly braces and semicolons all over the place,
you're going to be paying for that,
and you're going to have something that wasn't going to be.
We could talk about things like color,
and how we have to pay for all the extra use that are involved.
But yeah, so that would be,
we had a bunch of reasons for why we wanted to design the language this way.
Obviously, these tools continue to get better.
You can do things like fine-tuning, which we're still exploring,
and maybe there might be even better representations of the language
when you start fine-tuning.
But yeah, obviously, this is a very rich area for continued work,
and we're really hoping to try to get people really excited about LLMs
and kind of open source.
We're really hoping that more of these,
people are releasing weights and things like that for various language models,
so we're hoping that eventually we'll get to the point
where more people can do these types of things.
Do you have to train it with a language,
or could you just kind of prompt it with a manual?
Yeah, so what we do is we just do some prompt engineering.
So we have about 20 to 30 examples on the server,
and then what we do is we compute the embeddings of those,
and I think the embedding space is like 1,024 dimensions.
And then we take your prompt,
and we compare its embedding to the embeddings of all the example programs,
rank them based on distance,
and then we take the top 10 and give those as examples.
So it'll see something like,
oh, put five things in a list, and then here's a list,
and it'll see, here's create a button and do the button.
And then what will happen is it kind of stitches together some of these things.
I don't have a slide in this talk, but actually this picture will do.
So this was the translation app, and in our initial test of this,
we only gave it three example programs.
We gave it a program where if you clicked a button,
it would translate Hello World into French, I think.
So there's no user input other than clicking the button.
We gave it an example of a little app where you could add ingredients to a pizza,
and so that gave it an example of how list views work,
and then we gave it, actually it's not in this one,
we also had one where if you click a button, it speaks.
And so what we wanted to do then is we gave it those three examples and a prompt like this,
and it stitched together the whole program from start to finish
because it saw examples of each of these different functions,
and again for reasons I do not understand,
and I don't think any of us really understand,
it was able to eventually figure out,
okay, well the first thing I have to do is do the speech recognition,
and then the second thing I have to do is call the translator,
and then the last thing, which again is not in this example,
there was a text-to-speech component,
and it said, okay, well now speak the result of the translation,
and it never saw a program, at least in this language,
that did all of those things together,
it saw bits and pieces of those and what they looked like,
but it was able to predict the tokens in such a way that it could make the final app,
and so that's really one of the interesting things,
it's somehow in some cases, not in all cases,
but in some cases it's able to make these combinations that do useful things,
and then sometimes it fails, but that's life.
Can the source run without Google App Engine?
I believe it can now,
Google did the latest versions of App Engine,
the runtime is open source as of App Engine 2.0,
that's the library we currently use,
I don't know exactly how to run it outside of App Engine,
but supposedly yes, in the new App Engine 2.0,
that's all open source, and I believe it's available on GitHub,
or if not, it's on Google's system,
but there is also a pull request you could look at
where the backend has been replaced with Postgres,
and so you can, all the data objects and things are stored in Postgres,
so if you do want to run it without the App Engine piece,
you should be able to do that,
so I would definitely encourage you to take a look at that.
Any other questions?
Oh, yes.
Maybe a bit of attention,
but this is a great tool for learning official design programming,
you don't do the affecting, but how about the rest of the lifecycle?
Are there any instructions or ways of teaching those,
like version control, Q&A, distribution?
We do have some of that,
so we don't have a formal...
Yeah, there we are.
We don't have a formal version control system right now,
but one thing you can do is we have a checkpoint feature,
which essentially you can think of as making a commit,
so when I click on this, it'll say,
okay, I'm going to create a checkpoint called hello per checkpoint one,
and if I keep doing it, it'll do checkpoint two, checkpoint three, checkpoint four,
and so you can use that as sort of a way of effectively creating a commit,
you can continue making changes,
and then when we go back into the project list,
you'll have all of your checkpoints for your application.
Now, granted, that's probably not the ideal way of doing version control
for something like this,
but in the end, each of these projects is essentially a zip file
with that structure we showed in the slides previously,
and so you could, of course, pull it out and work with it that way.
Now, the other thing we've been doing,
and I didn't really talk much about the details of Apply,
because that's a separate hour-long talk,
but it uses a version of App Inventor that we've developed
that supports real-time collaboration,
so you get a multiple people editing,
and there we have the entire event stream,
so you can look at each individual change
and kind of be able to say,
well, we're going to take these edits out or do something like that,
and I don't think this particular release has it.
It's coming in the next one where we have Undo Reader Support,
so if the AI does something you don't want it to do, you can undo it,
so that way you don't lose valuable work because the AI went a little bonkers.
In terms of distribution, we do have some documentation
on things like managing your key store and other signing-related things,
and for the iOS build server,
we do have a bunch of documentation with screenshots
of how you go through the process in App Store Connect
and in the Apple Developer Portal
to set up all of the signing and everything you need to do the iOS release.
Now, obviously, this is, again, an area where more help
in terms of if you think there's documentation missing on this front,
the documentation is all in GitHub in the repository,
so you're welcome to contribute,
and it's written in markdowns, so it should be fairly straightforward.
Quality Assurance.
Yeah, we don't have a lot of that.
I'll be perfectly honest.
One thing that people have asked us before,
and actually it might be even better for teachers in some ways,
but having sort of a unit testing type framework for App Inventor,
so that you could say,
I expect that if I click this button, some output will happen,
because right now for a teacher, when you get a class of 20 students,
you get a grade at their homework, you get a load of each project,
try it out, make sure it works correctly, grade it, and so on.
And so automated ways of kind of doing something that would be helpful,
but that's not something that's currently on our roadmap.
Ah, okay.
Well, I think the answer is still the same there,
which is essentially we don't have stuff for that for students,
but of course there's no reason people couldn't build something around that,
like maybe, I don't know if the code or dojo folks would like to have material
around doing quality assurance of applications,
but that's certainly something they could develop.
I mean, we obviously give the platform, and lots of people have built material around it.
There's no restrictions on other people doing that,
so, and all of the stuff on our website too is CC4 by SA,
so anyone can take our material and reuse it.
Yes.
I have three questions, which I realize are probably all the same question.
Okay.
So one of the reasons why you chose when you wanted to support iOS
to use Swift a lot of them,
is to think of something like Kotlin that might be cross-platform.
Okay.
The second question was, you said that iOS support was coming soon to build servers,
but how does it work without that at the moment,
because it seems like you have some support for iOS.
Yeah.
And then the third question was, like, do you need the companion app
and you not just build, like, APK files?
Yes.
Okay.
So maybe I'll address the last one first, and then I'll go to the other two.
So yes, you could build the APKs and not use the companion app at all.
The nice thing about the companion app is that you see the changes in real time.
So building an APK because it has to package it up,
actually go to a build server, run through the Android build process,
and then return the binary back to you, often can take a minute or two.
Right?
It's the difference between, like, people, like the old sort of compiled test release cycle
versus reloading JavaScript in your browser type of thing.
So the nice thing about using the companion is it saves you a lot of time.
Yes.
So internally, App Inventor uses Scheme as its programming language.
So all the blocks and everything eventually output Scheme code,
and then there's a Scheme interpreter running either on the Android or the iOS device
that then runs the code and, you know, does whatever the behavior is.
Which kind of, I guess, goes to the second question you had about why not rewrite
everything in a completely brand new language.
So what we did is we essentially made a Scheme interpreter for iOS,
and then the question was, well, how do we get the components there?
And for that, we just decided, well, we'll use the native language
because obviously Apple is going to continue to support Swift,
it seems, for the definite future.
So let's just invest in the native platform, just like we use, well,
we'd have to rewrite all of our stuff from Java into Kotlin.
I believe there are tools that maybe help with that,
but I haven't done a lot of investigation there.
And so, yeah, we just said, well, let's just implement the Swift version of everything
and keep it as close to the Apple examples as possible.
So I guess the, oh, and then the build server.
Yeah, so you can today go to the App Store and install the iOS companion app,
and you can try your apps right now, right?
So I could open that HelloPer app and load it on my phone,
actually, let's go ahead and do that.
Are we at time?
Okay, yeah.
Oh, we can do it offline, but essentially the idea is,
can I take 30 seconds, Peter?
So I scan the QR code and I go to my app here.
Yes, I know my app is out of date.
It's actually out of date because it's a newer version,
and so it doesn't recognize the version number.
And so it uses WebRTC to establish a connection and send the assets over,
and then I just get the app running on my phone,
and if I turn my volume up, of course, it doesn't know how to create audio assets.
It only knows how to create visual assets.
So it's still meows, but yeah, that's the idea.
And so as we make changes, as Vishwas was showing,
I would see it on my iOS app.
Of course, if I close the app, then I lose the connection and I lose my progress,
but with the build server, now I've got my regular hello per app,
and then I can do it, and this one is looping infinitely,
so I'm just going to stop it.
That was for a test somebody asked about running audio in the background,
and so I needed to make sure it would run for a very long time.
But yeah, that's the idea.
So packaging the app allows it to be persistent on your device,
and of course you can put it into the app store and all that stuff.
So with that, thank you very much.
I'm happy to stay longer and answer other questions, but I will...