That was helpful. Thank you. Thanks for joining. This is going to be a talk about a fun project
that I started, I think it's almost four years now, so I feel like I'm sort of milking the
idea, but it's pretty cool. It's back in 2019, I guess. I ended up building an artificial
nose using some cool tech, and I'm going to talk a bit about the tech behind it and how I ended
up moving the project from a really, really dirty weekend hack into something that's hopefully
more future-proof and using cool things like Zephyr. So, a few words about myself. I'm a
Benjamin. I'm based in France for the past year, almost to the day, actually. I've been working as
a developer advocate for the Zephyr project at the Linux Foundation, and I do many things, including
as a good French person, I guess, baking bread. And I don't know about you guys, but I've been
trying to perfect my bread recipe for probably over 30 years. Like, I'm still not really happy about
the way it turns out. Like, it's a bit random, right? And so, back, I think, in the really first few
weeks of COVID, with like being stuck at home, lots of times on my hands, I was like, maybe technology
can help me improve my bread recipe. What if I could figure out a device with maybe some AI
in the mix that I could like train to figure out when my sourdough starter would be perfectly
fermented? In my head, at least, the idea would be that I would buy AI, figure out when the sourdough
kind of looks all right, bake the bread, figure out if the bread is good or not, give it a, like,
oh, it's a nine out of 10. Like, it's really crispy, really nice. And then do the training that way,
right? And so, the idea would be to smell the sourdough starter to capture some information,
which in my head, at least, I'm not a chemist, I'm not a food chemist, but measuring things like
the amount of volatilagony compounds and CO, CO2, whatever, there has to be a correlation and like
the perfectly ripe sourdough starter, there has to be a way to identify it, right? And so,
back in 2019, there was also this sort of cool kid on the block, new cool kid on the block,
which was, and which is, tiny ML and things like TensorFlow Lite, finally available on micro
controllers, things like that, right? And the thing is, I know really little about neural networks
myself, like, for some reason, the math, like, whenever I would open a book about neural networks,
and like, oh, yeah, it's easy, you're going to recognize handwritten digits, like, this is a bitmap,
you go through some layers, blah, blah, blah, oh, you recognize the digits, that was going way
over my head. The thing is, playing with physical things, more tangible things, I actually was a
role in just a few hours, really, and with the help of some tools, some of you might have heard
about something like, called edge impulse, it's not strictly speaking open source, although it's
based on TensorFlow Lite for micros, but it helped me train a model, basically, taking some Arduino,
like, an Arduino compatible device, this is a WIO terminal, a Cortex-M4, taking a gas sensor,
feeding the data and like, capturing the data quite often, taking this data into some kind of
training algorithms, and I would be able to figure out the difference, not necessarily between good
bread and bad bread, because remember, COVID, like, flour wasn't even available in the supermarkets,
but booze that I had in my house, so I actually figured that it was able to make the difference
between not only, like, rum and whiskey, but it was actually accurate enough that two, like,
one really pitted whiskey and one slightly less, so it would make up the difference, right? And I
started to talk about the project, because I found it really cool, like, not do the silly bread thingy,
but something slightly more useful, which is figuring out in the human breath, when there are,
when you can spot the markers for fungal pneumonia, Kaleb, the kid almost died, basically, when he was
really young and the doctors couldn't diagnose the disease, turns out that since then, there's now
literature available out there that says that, yeah, there are some markers, and he sort of
built a proof of concept for that, so that felt really good, but what didn't feel really good is
that the code that was from day one available on GitHub of that project that I have to put together
is horrible. It's like 2000 lines of boilerplate, copy paste, typical Arduino code, right? Like,
I mean, I've been gathering bits here and there, of course it works, but it's really, really bad.
Small, just like really quickly, because I think it's worth mentioning,
how does a machine smell anyways, because we're all, I think, familiar, or we all think of
things like temperature sensors and humidity and illuminance, like that certainly comes to mind,
because we actually also use them every day, but there's also sensors that can smell,
they measure the concentration of particular chemicals in the air. The way it works is basically
just a chemical reaction between a tiny slice of metal oxide semiconductor, and based on how many
of the offset compounds can be found in the air, you can measure a variation in the change
in resistance, right? The more VOCs, voltallogonic compounds would be in the air, the higher the
resistance, for example, which means that I could measure, like, start acquiring data,
putting my sensor on top of bottles of alcohol and tea and coffee and whatnot, and capture
basically what I would call the fingerprint or the olfactory fingerprint of a particular smell,
and then with a bunch of AI and ML, basically figuring out what in this raw data identifies a
smell, and so my intuition would be not knowing, again, a thing about signal extraction and all
that kind of thing, would be, oh, well, but if this is whiskey, then if I were to write down what
makes whiskey so special, it would be probably something like, oh, yeah, when you smell whiskey,
nitrogen dioxide goes up, carbon monoxide, not so much, VOC goes up as well, maybe in a slightly
more steady way, and so basically what happens then, the way the model works, is just that, except
that it's a machine doing it, looking at the raw data, doing some basic statistics to extract the
mean, the mean, the max, the standard deviation, like, all those things that could potentially
characterize the smell, and then this pre-processing, this DSP, if you will, then goes through a typical
neural network, so this is fun, you get to the point where you have this funny looking thing,
like you can even go the extra mile and, like, sort of, 3D print, the enclosure, and there's,
yeah, you have a lot of fun. I ended up building and packing, again, like, in those 2,000 lines of
code, plus all the libraries, of course, that I'm pulling, I would have a GUI, I would have
Wi-Fi integration, actually, that's something that I added eventually, and, like, whenever I smell
something, I can push it using MQTT to a server, there's, of course, tons of hardware interactions,
and all that needs to work at the same time, except that if you do it the Arduino way,
and the lazy way, I guess, then you end up just doing this, which is, again, not necessarily,
like, if you're lazy and just, like, eager to get your POC and your thing working, you end up putting
a lot of code in, essentially, a superloop, and so, as often as possible, I need to do all this,
which is acquiring sensor data, which, by the way, you don't need to do that often for getting
good accuracy, like, the way the device works is that I just sample the gas sensor readings
10 times a second, it's not all that much, so every 100 milliseconds, I would read sensor data,
and then I need a bit of time to actually run data through the AI model, which, again,
doesn't really take a lot. The model, at the end of the day, is really simple, so you really only
need a couple milliseconds there, fair enough, and then there's the world GUI aspect, which, again,
if you're lazy, I'm not even, like, whenever a button is pressed, it's not even interrupt driven,
so you need to figure out, like, if a button is being pressed right in the loop, not ideal, but
you do that, and then, if you want, you then post results to an IoT server, and then you don't even
know how long it's going to take, right? Like, if this is synchronous, it might be a problem.
Enter an autos, right? That's basically, for the first few years of the project, it was sitting
there on GitHub, this really crappy thing where people would open issues to be like, really,
I mean, yes, I would put the ready to flash, like, firmware for people to use, but anyone who wanted
to basically tweak the code, they were just scared, and so the thing is, I ended up, yeah,
using Zephyr to try and rewrite, and also to myself, frankly, to learn some of the best practices
there, I ended up trying to leverage some of the features of Zephyr, which is beyond being an
autos, which hopefully would help me move away from the super loop, also get a better solution for
targeting multiple architectures. Like, originally, I would be targeting the weird terminal, which is
some D51 Cortex-M4, but I actually don't mind ESP32, and having the same code,
same portable code, and portable build infrastructure, test infrastructure, I don't mind
getting that, plus all the libraries that also come pre-packaged, and yeah, that's basically what I did.
So, from this point, I guess, the presentation is more about telling you, like, how I replaced some
of the concepts or some of the things that I had in my Arduino code, and point you to some
interesting areas in Zephyr of, like, features and subsystems that are available that you
maybe didn't know existed, and, but frankly, I didn't know existed either.
Sensor acquisition, that might be the sort of the easy part, but I really like the fact that now
my V2 version, if you will, of the NOS, I have essentially, and literally, a dedicated thread
that acquires the data exactly at the sampling rate that I require for my model to perform
accurately, right? That's like, that could be an issue. If I do the super loop thing,
and for some reason, the UI takes longer to refresh or communicating with the
the cloud takes longer, then it will basically shift the sampling rate for the gas sensor data,
which basically means that I will start feeding crap into my AI model at all. So, you may want to
sometimes put the sensor to sleep and make sure that it doesn't draw energy unnecessarily, so it's
actually also integrated in the Zephyr APIs. Then comes the TensorFlow Lite aspect. So, I'm
basically pulling TensorFlow Lite as a library in my application and leveraging something that's
called ZBUS that makes it, especially for someone like me who's not necessarily a hardcore embedded
developer, I basically have this high-level framework where, okay, I have my sensor acquisition
thread that does its stuff, basically puts the sensor readings in a ring buffer, and whenever
there is data that's available for the rest of the world and the rest of my app
to do something out of, then it's effectively like there's an eventing system where,
effectively, my inference thread really gets data, like subscribes to sensor readings so that it does
the stuff and figure out what is it smelling like, and also uses ZBUS to put the result on the same,
like using the same topic mechanism, if you will, so that, guess what, the GUI, for example, can
in turn subscribe to this piece of information to do something useful out of. No need for
fifo's and cues and semaphores, like it's actually really nice, and the overhead is minimal. So,
there's that, and then for the GUI, that's one thing that's really nice with Zephyr is that you
have LVGL, it just works, like there's obviously in Zephyr tons of drivers already available for
a wide variety of display controllers, but then on top of that you even have, like, the high-level
framework that is LVGL for creating a GUI with, like, chart, like, this gauge,
this gauge, and I never know how to pronounce it, like, this gauge, and the charts, like, those are
effectively widgets that subscribe to the data that comes and is being sent on ZBUS and just
displays it, and the code is really, really straightforward, it integrates also with things
like the Zephyr input system, like, if you have buttons, keypads, touch screens, that basically
send events, you can have the LVGL app automatically react to that, right, so that's nice, and as you
may notice, this is not a photo of LVGL running on the actual device, it is a screenshot of LVGL
running in a desktop environment, because you can actually run the full artificial nose
code in a fully emulated environment, if you will, on a POSIX OS, including the GUI aspect,
so that's pretty nice, and like I said, it really feels like you're writing, like, really high-level
applications, I have, I'm defining, and, like, I have a listener that wants to be notified whenever
there is an inference result that's being made available by, probably, by the TensorFlow light
for micro task and thread, and when that's happening, then it's pretty straightforward, you get
the data, you really get it actually as an actual, like, typed message, like, so it's something like
you can actually really make a good sense out of, in my case, the inference result would contain both
a label telling me it's smelling coffee, whiskey, whatever, and a confidence level,
based on how confident the model is that it is effectively whiskey or coffee, and so I can
actually display that on my UI, and the code is really, like, literally moved from, yeah,
2,000 lines of code, I didn't count, but it's a couple hundred max, so there's that, and then
this is sort of nice to have, if you were to do more than just a kind of prototype toy project,
you could think about having the device, probably with something less stupid as the enclosure, but
in the ceiling of the restrooms here in the building, so that whenever it smells pretty bad,
you know that it's time to send someone to clean the place, but you don't want to send someone to
clean the place, like, twice a day if, like, nothing happened, like, if it's, you're on the weekend,
or it's like a day where there's strikes or whatever, or there's COVID and everyone is at home,
so the device would need to be communicating somehow in a way, like, remotely, and for adding that
to my project, it was also pretty straightforward, because there was a, like, full blown networking
stack in Zephyr for, like, TCP, IP, and, like, co-op and MQTT, and, like, all the variants,
all the flavors, and all the kind of connectivity options you may want to use, they're all there,
and so effectively, and I can maybe quickly switch to a really quick demo, which is, I have, so, well,
this is the version with the enclosure, this is the version, which is actually the WIO terminal,
this one is M5 stack core 2, so this is effectively an ESP32, this is the sensor, it's already
configured and already connected to Wi-Fi, so if I were to, I think I need to stop sharing maybe,
if I were to connect to my MQTT, yeah, connected to an MQTT broker,
and in real time, so this is really, like, reaching the internet and then my laptop connecting to the
very same broker that this guy is connected to, and, yeah, apparently it's smelling ambient air,
I guess it's more, like, nerdy or geeky air, and if I put, so this is, yeah, well, that was fast,
actually, this is lemon, and for the anecdote, I, I mean, not that you care, but I actually forgot
to bring the lemon from home, so I bought this one just this morning, so it's different lemon,
I guess that the one I use for training the model, but it apparently works just the same,
so that's, there's that, and what else, yeah, and many, many other things that are pretty cool in
Zephyr, the fact that it leverages K-configured and device tree, just like Linux does, makes for
pretty neat code when it comes to, oh, I want my GUI to be slightly different if my screen is large,
I want to put, to cramp more into the UI, well, that's an information that you can get really
easily from device tree, right, if my screen is wider than 300 pixels, blah, testing framework,
CI integration, every time I commit something and push something and make a modification to the
artificial nose, it gets built immediately, A1, basically, by the way, I wasn't working on
Microsoft back then, and they are absolutely no problem with me putting everything on GitHub,
so kudos to them for that, so now the new URL, if you wanted to check out the Zephyr version would
be the same, with Zephyr in the name, you can find all the parts online, I don't get any royalties
or whatever for that, but seed has actually sort of been like nice, ready to use bundle where you
can order all the parts, and that's it, questions!
Hello, thank you very much, so there is some abstraction where you can use different sensors,
but surely the sensors don't give the same values for...
Great question, I had a slide, I've removed the slide, removed the notes, I forgot,
one thing that I would love to see happen to kind of answer your question is some kind of open
data set, open ontology to actually describe smells in a consistent way, because you're right,
like you would have sensors that are giving you readings in terms of like unitless
concentration, like it's going between zero and 100% of VOC concentration, some would be
talking PPM, some would be whatever, some would have like weird calibration things,
there's, yeah, it's, you're right, so you would probably need to retrain the model,
it's not like you can, at least with this code, it's not like you can easily be like,
okay I'm going to switch from Bosch to Aliexpress, and it's going to work just the same, like you
need to, yeah, I hope this answers the question. One more, yeah. We would like to know how it
did it work with the sourdough and your baguettes? That's super, everyone asks the question, I never,
like I never done the whole thing, like because back COVID, there was no flour, it would have
been painful to bake dozens and dozens of baguettes and eat them anyways, and this is more fun to
play with just random things like spices or booze, and the sourdough thing probably works, frankly,
probably could be done more in a more simple way too, like maybe you just need a alcohol sensor
and just measure the peak, and maybe that's it, I don't know. Thanks everyone. Okay, thank you.