Hi everyone.
So I'm Abhishek Dasgupta.
I'm a senior research software engineer in the University of Oxford and I'm presenting
carpenters offline today with my colleagues Janata and Colin.
And so carpenters offline is about teaching data science and coding skills in low resource
settings in places where you don't have internet.
It's a way to do courses without access to the internet.
So who are we?
So Janata is at Newcastle.
Colin is at the National Oceanography Center.
I'll let them introduce themselves later on.
And there are other people as well.
We have collaborators from the University of Florida and also Stella and Barsha and
Durham.
What is the carpentry?
So for those of you who do not know, the carpentry is a non-profit organization which
was built to teach foundational coding and data science skills to researchers.
And their vision is to be inclusive and to make the software teaching skills as accessible
to as many people as possible.
And they have various kinds of carpentries.
So they share some courses like software carpentry, data carpentry which is focused on data science
and also library carpentry for library and information sciences.
And we have various roles like we have a carpentry instructor so anyone can go through
instructor training and be a certified carpentry instructor.
There are carpentries workshops which goes through the approved carpentries curriculum
which includes things like introduction to Git, introduction to Shell and introductory
courses in R and Python.
And we use technologies like WebBest.
Our course notes are all open source and online.
And we use Etherpad and Google Docs for shared notes and everything is on GitHub.
So how did carpentries offline start?
So again it was at the Software System in the Institute Collaborations Workshop, HACTA.
So we started upon this idea that what if you do not have internet and of course a lot
of the instructions in the current carpentries curriculum require you to access to internet.
So they will say download this from PyPI or download Python or RStudio and install it.
What happens when you do not have access to internet or maybe internet is very expensive
or maybe you want to work on the EuroStar which we found out that internet does not
really work and the Wi-Fi is there but it is a sham.
So we came up with the idea that why not use Raspberry Pi's because they are cheap and
available.
I think this was before they suddenly became unavailable but basically using some sort
of low cost single board computer to host the carpentries infrastructure and allow
people to access it offline.
So it won the HACTA and it also, you know, had the SSI Fellowship on that in 2022.
So the first part of it is actually getting the data from the internet onto the Raspberry
Pi.
So that is a package developed by a team at the University of Florida in collaboration
with us and that is called offline data site.
So offline data site is a component of carpentries offline but you can use it outside.
So if you for some reason want to get R and Python and like the common packages, you know,
pandas and ampi all that, on cash onto your computer so they can work offline, you can
do that.
By default it will install a certain set of packages which are customized for carpentries
but you can add your own packages to that and it's available on PiPI so you can install
it.
What we mirror is the latest installers in Python and R and also we use partial mirrors
of PiPI and CRAN.
You can customize the packages so you can specify your own packages to download and
once you download it you can set your local PiP or CRAN to get data from that and then
we also mirror carpentries online material of course and installers like RStudio.
So I'll hand over to Colin.
Thank you.
There's been three threads to our project in terms of building hardware that people
can actually go and use.
The first is to put the carpentries offline onto a Raspberry Pi that can be booted up
and run at a workshop.
The other is to actually build a bootable flash drive that can be used with an old laptop
perhaps and then the third and latest one is to actually build a miniature HPC high
performance computer that can be used for teaching HPC lessons.
So for the first option we have the Raspberry Pi.
I have here a Raspberry Pi Zero, one of the cheaper ones.
I think these when they're in stock cost about $10 or $15 or euros and they can run as a
Wi-Fi access point mirroring everything.
Some of our lessons require us to have Git Hub so instead of Git Hub we have a program
called Git T which kind of has all the functionality of Git Hub but is self-hosted on the Raspberry
Pi.
We can also run an EtherPAD server, mirrors of all the lessons and CRAN and PiPi mirrors.
So what it looks like we can use this, the Pi Zero or we can use the bigger Pi 1, 2,
3, 4, 5 now.
And when this boots up and I will try and do a little demo.
So when it boots up you should see an access point called Carpentries Offline that will
be accessible.
If you were to then join that access point and go to your web browser and type in Carpentries
Offline.org or 192.1681.1 you will see the web page that is listed there and that will
then enable you to get onto the Carpentries Offline.
Do we dare to do this?
Where's the Wi-Fi chooser?
It should be at the bottom there.
Oh, it's on there.
Good.
Is that joining?
Connecting.
You can see this on the up screen.
It says we're established so let's see if the demo gods are in our favour.
And there we go.
The web page being served from this little Raspberry Pi Zero.
If I was to click, for instance, on the data Carpentry and go to the R Ecology lesson,
there is a complete mirror of the R Ecology lesson that we can then teach from.
And we also have Git T so you can log into Git T and have a very similar experience to
GitHub or if you want to download some software to say you need to install R there is an R
Windows and an R Mac package available for you to download.
We also haven't got it listed on this page but you can then point your CRAN or your
PIPI installation at this server and install things from those mirrors instead of having
to get them off the internet.
And I will put this back on.
Edgy Rome to keep going with the slides.
The slides are idiotic.
Oh, okay.
So I don't need the internet.
It's beautiful.
It's not working.
I'm just going to keep going from there.
So one of the problems we're trying to build the image for the Raspberry Pi is we initially
started with a set of instructions of boot up your Raspberry Pi with an image you've just
downloaded, type this and type this and type this and type this and eventually you'll have
a Carpentries image.
We then moved to having a shell script that could run all of that automatically so that
it was a bit easier to reproduce.
Then at a hackathon we went to, someone suggested what seemed like brilliant idea of we could
run this in GitHub actions and do it all in the cloud and have that spit out a Raspberry
Pi image for us.
Many hard months of work later realized this wasn't quite so easy because the emulator
for the Raspberry Pi is really slow and it turns out that GitHub actions actually has
a six hour time limit which wasn't enough to do all of our installation.
We had a few hacks to speed things up so one of the things we found is that not just is
the computation slow but the network access out of the emulator is really really slow.
So downloading anything inside the emulator was much slower than downloading outside so
we actually download all of the offline data size stuff outside of the emulator, mount
it in a virtual drive and then copy it into the emulated image and build the emulated
image and now we've got it down to about two hours and it pretty much works.
The one snag I'm currently having is that GitHub is not allowing us to upload the final
image and I think we're hitting the maximum file size limit from GitHub and need to find
somewhere else to host our images but there's a link later on to our GitHub if you do want
to go and download the last image we managed to get on there.
As a kind of side effect of doing that I started testing builds in the cloud just natively
on a AMD64 system and realized that we could also then build a Docker container containing
all of this and that was actually kind of useful during workshops because sometimes
some of the carpentries infrastructure would go down midway through a workshop and the
last thing you want halfway through a workshop is telling everyone to go and download something
and finding that the website they want has gone down so we found that we could also replicate
all the infrastructure in the cloud or on our own server if we have a server and it
meant we had a backup version of everything needed to run a workshop which has saved me
on multiple occasions now from workshops where we lost access to something and I've now got
a almost one press solution with a Docker container that I can deploy out to a cloud which also
works very nicely for testing stuff.
It's sort of the tail end of the COVID pandemic.
Raspberry Pi's got really, really hard to get hold of and really expensive.
It was always the joke that Janetta had all of them in her house but I don't think she
was the sole cause of that but the chip shortage certainly didn't help us and so we started
looking for alternatives to the Pi.
Do you want to take over at this point?
Do I want to do this bit?
I don't want to.
Where are we now?
On to option two.
Oh, okay.
So this option two is also because some people say but I already have a laptop so why do
I want to go and buy a Pi?
Especially if we look at countries where you don't have access, that's already a big problem
apart from when there are two scares to find.
So we've come up with the idea of doing exactly what we're doing with the SD card for the
Pi and creating an image for that but creating an image on a flash drive which can also be
downloaded, written to a flash drive and it has exactly basically the same software on
as the SD card and you can just boot any laptop that you can boot from a flash drive.
You can boot from the flash drive and it turns it into the same server as we have here.
Oh, okay, so now let's put this.
So more or less last year, April, I was running a workshop, an intro to HPC and everything
that could possibly go wrong did.
So I was starting to think, well, I also had a shed load of my fellowship money left and
I had to spend that on something and I thought, but hang on, if we extend this project to
cover the intro to HPC, that would be a really cool thing to do and there were a few things
that we could do with the money HPC.
The hardware is more visible so people don't know really what an HPC is, this massive thing
in the cloud somewhere hidden and they don't really ever get to see the HPC that they work
on it and it's also quite easy to mimic the original limitations which you're not going
to do during a lesson on a big HPC but on the many HPC you could actually do that, you
could hit limitations and see what will happen and teach people how to cope with that.
You also, and this was the big thing, people get this email, say register for your account
on the HPC, they don't do that, they show up on the day and you've got to jump through
all those hoop of keeping people registered which is not a quick thing to do because sometimes
there are loads of things to be done before you get your actual HPC account.
So also the nice thing is it doesn't interfere with the real HPC so users can get quite afraid,
they're going to break something on the real machine so in this case you can assure them
it's here, we're not going to break anything of national importance or international security
or something and of course if you don't have access to a real HPC then that's also covered.
Also the day when we ran the workshop one of the things was somebody was doing something
on the node, they were running on the login node and you know what's that, you can't even
log in, let alone run your scripts so you won't have that and you won't also have problems
with network access because it's all local. So like Colin said the reason there was a
scarcity of Raspberry Pi's was because I probably had them all in my house. So although I got
the money for building a many HPC I decided to go for these rock pies which are sold in
the UK by a company called OkDoo and RS Electronic and it was an absolute disaster trying to
get these things ordered and the time I had to work on this thing was passing because
these guys couldn't get my order, it's all too dark. So I collected all the Raspberry
Pi's in my house and built this one that you see there in the picture and that consisted
of three Raspberry Pi 4.4 nodes and then one for the login node, in the meantime I've added
I think two more nodes or something and it's running the Pi OS Lite, the 64 bit one and
then the head node also acts as a Wi-Fi access point so everybody can just log into that.
So in the meantime I did manage to buy the rock pies but we are still in the process
of setting that up because the idea in the end is especially the Raspberry Pi's would
probably make more sense but the rock pies were slightly more at a higher spec but the
idea is in the end to produce two images that you can again just download from the website
and we want to be able to build up this operating system with scripts so we still need to do
the scripting, at the moment I'm still doing everything manually but so we want to work
to do the scripting and then download an SD card image which you can just write to it
so that the people who will end up with this mini-HPC don't need the knowledge to set all
of this up and this is basically the software that we want installed on it because this
is what the lesson covers and what most people will be using and what we need to actually
do what we want to do for the networking, for the scheduling, etc.
And then I've got to get to the point where I credit everybody so all our gets here, if
you want to know where I got the STL's for the printing so a lot of people I do have
to give them credit because everything is their work. I forgot to put this picture on
last night so this is pseudo for executing bash commands so I forgot that one to add
it then, it's the important one.
And then there's the small one that was called RM.
And so here's some more links and credits. The Raspberry Pi image that we have at the
moment that Colin's been talking about can be downloaded from that link there. You can
find us at CarpentriesOrFlying.org and what's not on there would hopefully soon be on there
because it's a work in progress all the time. We also have a Slack channel on Carpentries
workspace and so we can be found there so if you have any questions or anything you can
get us there and that QR code will lead you to CarpentriesOrFlying.org and I think that's
the end.
Thank you. We have time for questions.
Can you share with more how people are using CarpentriesOrFlying outside the team?
Okay so, I forget to repeat the question.
Oh, how can we use CarpentriesOrFlying being used outside the team? So one of the things
so far was because we're still working on this image is that we have not, we've got a lot
of people that keep saying they're interested but nobody has really taken it on yet. So
especially with the Mini HPC also I'm still working on it but I hope to now soon be able
to run my first workshop on it and then hopefully it'll take off from there because then I can
say to people okay we've done it. Also I've kind of used it, I went to UNO's first year
of Strathclyte to give a, to run a workshop and the power went down and the internet went
down and I was able to use it there just from my laptop actually that day. No, I didn't
use the Pi, I didn't use the Pi too because I had it all there but I've known we've not
been able to get people because I feel it's not in a state where I can actually set somebody
off with no experience because we also need to develop an off-boarding lesson. So we hope
to have more hackathons and more work sessions where we can work on these things and get
it ready for other people to adopt. So if anybody here wants to go off and adopt it please let
me know and you know where to find me.
Actually there was another one that I ran in South Africa where we wanted to test this
and we ran into a limit of eight people connecting at that point but I think Colin has sorted
the problem out.
He sorted the newer Raspberry Pi's but there's zero, I think so there's a limit and some
of the older ones do but there was a firmware fix for that. A typical Carpiti's workshop
would probably have between 10 and 30 people so we'd be aiming for that sort of size again.
A typical Carpiti's workshop would be between 10 and 30 people but we had limits with the
older hardware but we think the newer hardware doesn't have that problem and we should be
able to get 30 people on it.
Another question.
Yeah I spoke faster than I rattled over because I didn't know how much time it was left.
And this hat makes me warm.
At least we didn't get the hot hats.
Five minutes.
If there are questions please ask away.
If not, it's cool.
You all have been informed that we have a social event tonight.
We are going as an organizer team of this room to the tavernier bar so you can check
the queue on the left.
That's the way to the tavernier.
We are also organizing an event next week that's going to take place online with some
other talks not only that couldn't make it in the tight schedule today but people who
couldn't travel to Brussels.
We wanted to be inclusive of those people too so that's the QR code in the middle and
the website is the last QR code that's on the back of the room and the right there.
A very important question.
Where can I get a hat?
I've got some more at home.
Next time I go to touring I'll ring one.
These tools are all representative of the carpentries.
This is the Alipay software.
This is the library carpentries.
We also have a protractor but we are figuring out how to get that on the bed.
Thank you very much.
Thank you.