Okay, we're ready for the next talk by Courtney who's going to talk about open source radio
astronomy.
So hi everyone.
I work for Ostron which is the Dutch Institute for Radio Astronomy, we're a government institute
so we're publicly funded.
Apart from being publicly funded we have a lot of research grants and that's basically
what pays my salary and I'm going to talk about our biggest instrument that we have
which is called LOFAR and we actually utilize a lot of open source software there, almost
exclusively there's some caveats.
My name is Courtney Leuker by the way, I'm also into amateur radio, my call sign is
Papa Delta 3 Sierra Uniform, this is going to be quite a relaxed talk, we're going to
give a high level overview of all the different components that are there, there's quite
a lot of them actually so it's not possible to go into detail with the time we have.
This is my first forced them talk ever and also my first talk ever in a lecture hall
so that's quite interesting.
Now some background, I mentioned that we're a government institute, we firmly believe
public money means public code and we stand by that in almost everything we do already,
we have an open source committee that also ensures that we do that and we have basically
two very big telescopes, one's called LOFAR which stands for Low Frequency Array and the
other is the Westerbork Synthesis Telescope also called WSRT.
There's some sister institutes that you work closely with or are related to us, one is
called Kamras which maintains a telescope that we've stopped using and the others are
Jive and EVN, I'm not going to talk too much about those today.
What I want to tell you is that there is this principle that our radio telescopes work on
that is called very long baseline interferometry and this enables us to do radio astronomy
in a way that wasn't possible with traditional radio telescopes.
This is the whole map of LOFAR, there's 54 stations in total, roughly 25 of those are
located in the Netherlands.
I say it's around 2,000 kilometers in diameter but that's no longer true because the one
in Rosen is new and we're now about 2,500 kilometers in diameter.
This diameter is also called the baseline which is where the very long baseline interferometry
comes from.
If we then zoom into a station you see all these tiles and you see these little squares
and those are the different types of antennae and that is what makes this type of radio
astronomy so interesting is that we don't have a single antenna to catch radio waves,
we have lots of them, about 20,000 in total actually which is quite substantial.
This center is called the super-turb and it's located in Exlo, the Netherlands.
How can we actually combine this data?
I told you that traditional radio astronomy relies on a parabolic dish or a single antenna
and we try to scale those up, make them bigger and bigger but of course physics are at the
limits at some point, you can't make a structure from steel that's like 500 meters in diameter.
What we do instead is we combine smaller antennas to act as if they are a parabolic antenna
and the trick about a parabolic antenna is that all radio waves, no matter where they
are incoming from, they all have an equal distance to the receiver so we need to emulate
that with our antennas and we do that in two ways.
That is an artificial delay, an analog artificial delay by just making the line that it needs
to travel across on the PCB or the coax cable longer but we can also do it digitally after
the data is being sampled and then we can aim into the sky and create a very narrow
beam that observes a very small portion of the sky and that allows us to zoom really
deep into space and make very detailed images.
But what is this radio waves actually?
What are those?
What are we observing?
There are two types of radio waves that are being emitted by objects in space and the
galaxy.
We're only going to describe one phenomena today that's called synchrotron radiation.
Basically if you have an ion, a charged particle, you accelerate that, then it starts creating
radio wave emissions.
The frequency and the intensity at the frequency that is actually very dependent, that's all
details that are not very interesting for this talk, but one of these entities that
emit these types of charged particles are sometimes black holes and we'll see an example
of that at the end.
So I mentioned black holes, there's other types of radio astronomy that are very interesting.
We can also model our own ionosphere or enlightening.
Pulsars are pretty interesting, these are stars that are rotating at a periodic interval
and they have very strong radio waves coming from the poles of those stars.
So what does Lover actually look like?
A very small antenna as I told you, we can see on the left that there's like wires attached
to those poles, those are actually dipole antennas and if you configure them like this
where they are like a V-shape, they are called inverted Vs.
These are the low-band antennas and on the right side you see the high-band antennas,
they're like a clover shape, like a tie shape.
Then we combine all these antennas, low-band antennas and 69 high-band antennas in a station
and we send the data at around 3 gigabits per second to our HPC clusters.
There's a two-phase cluster here, the first is GPU processing where we do correlation
and beamforming and the second is a central processing which is more like CPU based.
In the early days our computing cluster looked something like this, we had IBM BlueJeans
machines which were based on PowerPC and they had a 3D torus interconnect which is actually
a quite interesting interconnect.
This was problematic because utilizing the floating point vector extensions required
manually rewriting assembly which wasn't that nice and it was pretty hard to find developers
who were willing or capable to do that.
So we moved to commodity hardware, GPUs, two CPUs per socket, two GPUs per server, two
GPUs per server, lots of networking.
That's really what you see here, we had 32 gigabytes of 10 gigabit Ethernet and then
in 2018 when we upgraded we had 24 times or 23 times of 100 gigabits in FiniBand but you
also see that there's a lot of 10 gigabit Ethernet per device and I'm gonna go into
that wider this in a minute.
If you look at the data flow or more like a software site then you see that the antennas
have ADC so these conferred the analog waves that are incoming to digital signals and then
we do beamforming on the stations and we send data to the correlator and this correlator
also does the correlation afterwards and you can see that once the correlator is done
with it we store this to disk and once it's stored on disk then it's made available to
the central processing.
So the correlator and our GPU cluster cobalt are doing like streaming and the central processing
is more like your traditional HPC.
When we look at the data flow in cobalt there's all this incoming 10 gigabit Ethernet and
this is why we have four or three 10 gigabit Ethernet links per cobalt server.
They are streaming the data and we configure per station where it needs to send its data
to.
Then once it's there it's being transposed at roughly 240 gigabits and once it's transposed
we do have two pipelines that essentially run in parallel, one is correlation and one
is additional beamforming so we actually beamform twice in a sense.
It's little bit more complicated than I'm sketching here but I'm keeping things simple
because stations also have the capability to not beamform and send unbeamformed data.
We have a special buffer that's called the transient buffer where we dump raw samples
and can send those two clusters but the general pipeline is what I'm sketching here.
If I assume into these two pipelines the correlator pipeline and the beamformer pipeline I don't
want you to look at the details too much here because that's not interesting and I really
don't have time to explain it but the trick is almost everything you see here is based
on signal processing, digital signal processing.
That's what we're doing.
We're using the fast Fourier transform, finite input response filters and transforming the
data in like the frequency domain if you will.
Then it's put back into CPU memory at cobalt, some final transformations are being placed
and then it's put into the disk so that's how it can work on it.
At Ostrone we do a lot with software and I've showed you now how the data flows but I haven't
told you what software components are making that data flow happen.
For cobalt, it's actually one solid product that lives in the low-fiber repository.
Please don't all now visit or get lab instance at once because it will die if you do that.
Try to spread that out a little bit over the day.
I'm sure I will upload the slides soon after so you don't have to remember all this.
Then this, all these tools that are listed here basically except for cobalt with lifts
in the low-fiber repo, those are more like what you would find on the SAP side of things.
I'm going to explain, I want to address that this is just the tip of the iceberg.
On our GitHub repo you can find a lot of more stuff that you can play with and I would encourage
you to do so because it's quite interesting.
Then there's also CASA core which is heavily being developed at Ostrone as well but it's
not actually just a product just by us.
A competitor or like a friend of CASA core would be AstroPie, very widely known packages
in the industry.
If you look at the radio astronomy tool kit, so the necessary things to go from antenna
to image if you will, then these are your friends.
There's DP cubed, WaySW clean, and IDG, and Rattler, AO flagger and I'm not going to talk
too much about every beam.
What do these things do?
What are we looking at here?
DP cubed is where you define a complete pipeline, so you have the incoming data, you need to
do some transformations on the data, maybe you want to identify some noise sources that
might be in your data, and eventually you want to create an image, and for this imaging
you need deconvolution as well, and you also need gridding and de-gridding.
So, AO flagger is where you identify noise sources, this can be anything, like radio
instruments are very sensitive, so one noise source in particular would be electric fences,
windmills, solar farms, bed quality LED lighting.
Then we move to the imaging part with WaySW clean because when you have a radio telescope
consisting of many small antennas, in between your antennas there are holes, and that means
that you're not receiving the data as if you would have a very large parabolic dish, there
are some differences.
This creates some kind of fringing in the image that you need to filter out, and that's
what WaySW clean together with Rattler and IDG are doing.
In IDG is your translation from the data domain to like a domain that is useful for radio
astronomical imaging.
So I talked a little bit about CASACOR and how it was industry widely used, it's based
on all the packages that have been around for a very long time, but we've actually
switched it around now, so now CASACOR is built on these older packages, on top of these
older packages, rather than CASACOR depending on these older packages.
There's several unique features here, the UV domain is pretty interesting, so that's
the domain about having your, about the plane that is filled, so those holes in your surface
area if you will.
And there's some phytonda bindings here, so these are all very nice tools that you can
just play with.
We also use a lot of open source tools, and we're doing quite well, there's still some
close source software, I'll get into that in a minute, so we use OpenMPI, OpenMP, Slurm,
GitLab, Grafana, and actually the part that I work on is PyTango, which is a SCADA system.
So with supervisory control and data acquisition, that's basically your interface that we have
on the individual stations, and those stations then configure underlying hardware with the
antennas and the ADCs, and they report how they are configured to a higher level system.
We also use Prometheus, Docker, and a variety of interesting open source tools, this is
just the tip of the iceberg as well, there's much more.
Next to our SEP cluster is also pretty interesting, which is actually where we use Slurm, we also
have a DustSys cluster, which is a cluster shared with many universities within the country.
Things where we can improve, well, we use CUDA, so that's not really open source compared
to OpenCL or Falcon, we're using WinCC for monitoring, maybe you've heard of that package,
it's Windows-based, that's why it's called WinCC, we're trying to face it out for Grafana
and Prometheus, that's going quite well, I'd say.
We have a lot of closed source SEPDA vendor blocks, so if you have your silings or what
have you or your Altera, then they for instance offer IP blocks to implement 100 gigabit ethernet
interfaces, and they're not too keen on you sharing those with the whole wide world.
Then InfinityBand firmware is pretty closed source, I believe there's open source versions
of that, but I don't think I know if they work quite well, and then the main area that
we're actually struggling is with office management tools.
This is definitely the area that we can improve the most at Astrone, we use Office 365 Slack
and Zoom, and as you can see, Copano, Metamode, Jitsi, there's definitely open source alternatives
to this, so there's no real reason why we should be using this.
Of course, you need to host the infrastructure, and that also costs money, so there's some
little way there, I'm not saying that it's definitely cheaper, but there's open source
alternatives to this.
Now I want to show you, I told you about IDG, that does the gridding and de-gridding, I
told you about WSWClean, and the Dravartler part that does the deconvolution now, and
I want to show you how those tools work in practice.
So we have an IDO point source, this is our most IDO radio source that can possibly exist,
it creates a very sharp point in the sky, we put it toward the gridding, and we get
a point function.
What this basically is, is this is the error of our telescope, so we now know, okay, this
is the error it's going to generate in our images, because we don't have complete filling
of the UV plane, there are holes in between our antennae, and then we can use the WSWClean
image together with Dravartler for deconvolution to create an iterative process in which we
iteratively remove the noise from the image.
So actually I'm going to see, oh yeah, that's nice, so here you see these lines, these lines
are the fringes that I've told, and if you then perform these iterative cleaning process
on what are called calibrated visibilities, then we iteratively see that this image is
drastically improved.
So now some example of this, what is the science that we do with Lofar, how does this look
like?
Well this is the, this is a paper by Erk Timmermann, so you can look at it when you spare time,
and what we're basically seeing here is we're seeing huge jets of this synchrotron radiation
emissions that are talked about, and you can see that they actually over millions of years
they vary in intensity, and at the center of this image is actually a black hole, but
you can't see that because it's a black hole, and then on the background of this image there
is an overlay of what's the optical domain, so not the radio domain from the Hubble Space
Telescope, and this is what we have been able to capture with Lofar.
So we're doing groundbreaking science, and we're going to do a lot more, we're in the
middle of a big upgrade that's scheduled for the end of 2014, Vincisi is going to be replaced
with Kavana, we're thinking about Allerta, but I've heard that Kavana has persisted
alarms during, actually forced them today, so we might not need Allerta, and we're using
Prometheus.
We had this low band antennas, and the high band antennas, I briefly skimmed over that
because the details, yeah, you have to cut some corners somewhere, but basically with
Lofar 2 we'll be able to use both of them in a single observation.
We'll also be able to use multiple beams, so we talked about the beam forming, currently
Lofar is only able to have a single beam per observation, and we will also be able to point
at different points in the sky, change that during an observation, and we call this mega
mode, don't ask me why, yeah, we're completely refamping the skater system, we're now using
Pytango, we have sufficiently upgraded hardware, Unibor 2s, we actually sell those to external
institutes as well, so they're available, and we're drastically improving the timing
distribution, so we're currently GPS based, everything is synchronized using GPS, all
the stations across Europe, and we're going to use the white rabbit protocol that's made
by CERN, that's based on a precision time protocol.
Now very briefly this mega mode, what would this schematically look like, so this is basically
what's running on cobalt or GPU cluster, and we do imaging and beam forming, and now we
have one beam and several pointings, and they stay the same during observations, now we
can have multiple beams, and we can repoint during the observation.
That's going to create a lot of flexibility for the astronomers, and I'm going to be very
excited with the science that is going to come from this.
I want to leave you with some links, as mentioned our Astron repo, the Astron website, there's
a very interesting page about 10 years of LOFAR, because we've actually existed, LOFAR
has been in production since 2008, so that's been since quite some time, there's this very
nice map on which you can actually see all the physical locations of all the stations,
how many antennas are active or working or broken, so this is all open data, you can
just look at this, and there's some history about all these presentations that I've done
in the past, so any questions?
Maybe first a short comment, the Chinese built a 500 meter dish, but what I really wanted
to ask is whether you have collaboration with other astrophysical observations like square
kilometer array or something like that?
Well actually we collaborate on the square kilometer array, so there's definitely, can
you repeat part of your question, because people were just leaving?
Well, whether there is shared development in software and stuff?
Yeah, yeah, for sure, for instance on CASACOR as I mentioned, but also WS Clean, we see
contributions from external collaborators, and especially the jive, the part of jive
that I showed at the very beginning, let me see, shouldn't be too long, so here I mentioned
jive and EVN, this is a huge collaboration of parabolic dishes that are all connected
and all the data is sent centrally to Dringelo at the headquarters of Astong, and that's
actually where the EVN network processes all this data, but all these dishes that we use,
those are not ours, right, those are from other parties.
Someone's asking, since everything is processed digitally, can these telescopes focus on multiple
targets at once by processing the data multiple times?
That's an interesting question, and that depends, as I said you have the transient buffers which
dump raw samples, but typically what we do is we already do beamforming on the station,
and if you do the beamforming on the station, you're already looking at some point in the
sky, you're only sending the result data from that beamforming to this cobalt cluster,
you can't beamform again then, the data is lost, it's reductive in nature, but if you
would send the raw sample data to cobalt, and it could somehow process all the data,
which I don't think it has the bandwidth to do so, then you could, at a later point in
time, point at any point in the sky again, which is, that's the job of the transient
buffers.
Thanks.
Maybe I have a question here, would there be any ways or interests for amateur astronomers,
or radio astronomers, to help or work with astronomy?
Well there's definitely job positions on our page all the time, I think, I don't know
if most are in the field of radio astronomy, but what typically happens, and I can briefly
explain, is we have a system called Nordstar, in which astronomers submit their proposals
and describe what they want to do with their instrument, and then we have a community that
looks at that, and that actually accepts these proposals, and then they are scheduled.
This is actually a very good question, because I completely skipped this in the talk, but
I wanted to talk about this, and then things are scheduled in a system called TMS, and
that basically looks at, okay, what part of these stations are required, and to do these
observations and collect these data, then these data are collected and processed on cobalt
and sap, and the data products are made available to these individual astronomers who've done
that, and they get exclusive access for a period of time to do their research.
Okay, thanks.
I was more thinking about, just if someone is in Africa with a homemade dish, is there
any way to capture something with an SDR, and add a little bit with data, or the scale
of things is so different that...
What's actually very important, or it's rather we need to model a lot of things and calibrate
a lot of things, so that's why all the stations are roughly similar in shape, similar in shape,
have similar hardware, so it would be definitely possible to buy your own station, build your
own station, and have the same hardware, and then hook it up, that happens all the time.
Different countries do that, buy stations, and then we add them, but having vastly different
hardware and then hooking this up to the system would be very difficult, it's not designed
to do that.
Okay, so if you would make a very cheap station that could be built by amateur astronomers,
you could deploy that everywhere in the world, and then make your public radio astronomy
like that.
Interesting, thanks.