So, hello everyone. So, welcome to our talk and really thank you so much for staying for
this long. This is like the second last of the session of the day. So, really appreciate
you being here. So, today we're going to be talking about modernizing legacy messaging
system with Apache Pulsar. And here, you know, we have Enrico and then myself too. We're
from Datastax. Okay. So, but before we start, if you like a copy of our, you know, slide
deck, here's the QR code and also the short link if you want. I'll let you take a moment.
Good. Okay. Okay. Well, even if you missed, don't worry, we'll be sharing with you our
connection info. Then you can connect with us. We can always be there to answer your
questions too. So, with that, let me start. First, just a quick introduction. Who's Mary?
So, I'm a streaming developer advocate at Datastax. And Datastax is a company based in California.
Starting in Apache Cassandra, Managed Cloud. And then now we also have the Managed Cloud
for streaming, which is Apache Pulsar. And I was also a developer advocate before joining
Datastax last year. And I'm based in Chicago. I'm also the president of the Chicago Java
users group. And I'm also a Java champion. And before this, I was spending over 20 years
or so being a developer myself too. So, that's me. And then this is Enrico. Enrico. Oh, yes.
Sure. Sure. I'm Enrico. I work with Mary. I really enjoy working with open source communities.
So I'm involved in a few Apache projects like Pulsar, but all the big Datastax or ZooKeeper.
And also I collaborate with Maven and Curator. I'm participating also in some CNCF project
like Pravega that is still about massaging and distributed streaming. And also contributed
to RDB that is a Distributed Embeddable Java Database.
Okay. Great. Thanks, Enrico. I'm really happy today to be here with Enrico because we were
just working remotely, finally get to meet here in Belgium when he lives in Italy. And
I'm in Chicago. So, okay. So without further ado, this is the agenda like within 20 minutes.
So it's going to be a little bit quick, but we'll end up having Enrico also doing some
quick demo as well. So first, let's kind of give an introduction to what is JMS, assuming
you know, not everybody is familiar with that. So some introduction. And then we'll talk
about Apache Pulsar and why Pulsar. And also just quickly describe the Pulsar architecture
and how do you do the mapping between JMS and Pulsar. And then how do you use JMS API
with Pulsar. And Enrico will show that. And then that's how we're going to be doing.
So first of all, just some core concepts too, right, of JMS. And as such, right, JMS is
all about also messaging, but it's very much a Java centric technology. And it's here,
as you can see, right, it's also published, subscribed kind of model, making use of destinations
that it supports queues and topics. So messages, producers, consumers, these are typical like
pop-up producer, consumer type of pattern. As such, it's a pattern, but this has its
own implementation. And basically too, it makes use of the JMS context and that will
help you with the connections and sessions. Okay, so about destinations, right. So essentially
too, it supports both queuing and the topic too. And so it acts as a broker in the topic
case, but for queues. So each message is basically, as such, right, message queue is you drop
the message there and then it gets picked up and then it's kind of done, right, by the
consumer like that. It's browsable, this queue, first in, first out kind of approach. And
then with topic, it allows for multiple subscriptions too. And message dispatch according to the
subscription type as well. And consumer, as far as consumers styles go, you can have
blocking, which is in the blocking received methods and that's all application driven.
And also, yeah, okay. And then there's also making use of the message listener method,
which is a JMS to driver driven in that case. And as far as producer styles go, the blocking
will be send method or there's also a async send too. And that will be like with completion
listener. So that's real quickly. And then as far as administrative operations go, as
we know, JMS does not cover administrative operations. And how do you manage the destinations
and doing, you know, connection properties, all of these things, the defining security
models or resource limits, all of these things and configure all of these at JMS itself doesn't
have to do it. So how do you manage it? It usually relies on your vendor. How do you,
you know, we kind of do all of the management too is through some vendor way of allowing
you to do that. And so basically too, there's also API also to let you work with administrative
objects too. And so basically, they're, you know, supposed to be kind of also provided
by the system as well. And as far as destinations go, there are queue and topic references.
And connection factory basically is the, is essentially too, using connection factory
is the client that allows you to connect to the system in that case. And then there's
also JMS, right? The API is essentially allows you to interact with Java EE or now is Jakarta
EE, but back then there's Jakarta Java EE. And in that case, you can basically make use
of EJB components. There's stateful, stateless EJB. That's used in web surflets or, you
know, the Jax RS, Jax WS endpoints, right? And it allows you to also do background like
doing scheduling kind of way of doing things. And then there's also message driven beans.
So these essentially too is basically their JMS specific kind of beans to handle messages
in there. And it's basically managed by the container, the, you know, J2, JEE container.
When you receive a messages from a container, then it will be essentially be, you know,
activated in that case. So J, the Java EE container provides support for like all of
the, you know, life cycle management pulling of these context dependency injection of these
things and transaction supports of security standard API. All of these tools basically
relying on the container to do that for you. And then there's also to what about external
resources. So a lot of times, and that's how it relies on resource adapters. It allows
you to essentially extend the Java EE container in that case. So in some key points, it basically
to use it is you need to have the resource archive file. So dot RAL file that will contain
the code and you have to then configure the resource adapter and everything. And it allows
you to essentially create administer objects, right? That conforms to these objects will
conform to the standard API and is implemented by the, by the core inside the resource adapter
too. So these are the different packages like basically Java X dot JMS. In this case, it's
I think in the new, new version would be Jakarta, but we're still talking about Java, the older
JMS in this case, and will be connection factory queue and topic. So usually each objects to
a bound to a JNDI naming and directory interface registry, right, provided by the container.
And so it's specific to the container as to how you do deployment too. And that's how
it usually works. Now then let's get introduced, right? So now we talk, talk about JMS stuff
is a bit more legacy stuff. So what are some of the options, right? To, to kind of leverage
on today's like more modern world that allows you to work in a cloud native environment.
But also we want to introduce to you Apache Pulsar is an open source platform and it's
cloud native and it supports distributed messaging and streaming too. And as such too, this is
the link where you can kind of find out more information or this is actually more the,
the GitHub repo. So wanting to highlight it because we don't have too much time, but
basically it's very cloud native in nature. It's born with the cloud native DNA and various,
you know, it's basically the key point of it is that why do you want pulsars? I think
what, I think at least one of the key point, it separates out the compute and the storage.
So basically Pulsar can focus more on working with the messages delivery, right, dealing
with all the messages coming in, delivering all of these things. And then, you know,
you have a whole laundry baskets of all the log messages, then what do you do with it?
Rather than dealing with it, Pulsar said, let me get bookkeeper to handle it for me.
So, so that way Pulsar can focus on that, you know, just the messaging part and coordinate
with the bookkeepers. So that's what it does. And it also supports multi-tenancy and that's
a very nice way of helping you to organize all of your messages, as well as some features
that are more kind of ready for enterprise level, like, you know, geo replication is
also a major thing in that. And also it has what is called like tiered offset. It's basically
if your messages get code, right, and bookkeeper, you don't want it to take up too much room.
Then you want to move it to, or actually, I should say, it gets kind of in the one storage
and you want to move it off to cold storage. So all these, as Pulsar has built in and it
knows it. So native Kubernetes support all of these things, schema, it has a Pulsar schema
connectors, and you can use the basically Pulsar IO framework to build different connectors.
And currently we're supporting like almost a hundred different kind of connectors, too,
in there. Message processing, you can use the Pulsar functions framework, so you don't
need to use anything outside to do message transformation as you are building your data
pipeline. And also the nice thing, too, is that it doesn't restrict you to only using
Java as your client. You can use other things like C++, Python Go, and other community contributions
to such a cloud. There's also Node.js, also.NET C-Shop client, too. So that's really
flexible and really functioning real well in Pulsar. So let's kind of really quickly
kind of take a look. I already mentioned some of it. Essentially, too, it's a blazing performance.
That's what we all want. Provides you with true like real-time type of processing. That's
why we want it, right? It's basically millions of JMS messages can be handled if you have
JMS leveraging on such a platform. So it's all good. Horizontal scalability. If you expand
your infrastructure, adding more servers and nodes and all of these to it, Pulsar will
handle that for you. You don't need to rebalance all of your topics, and you don't need to
deal with offsets, right, such as in maybe like Kafka, things like that. It has its own
way, so then you don't have to worry as a developer. Worrying about all of these infrastructural
things. So all of these things are just listed here. I know there's a lot of, you know, works
in here, but it allows you to kind of get a bit more into detail, and we can share with
you this thing. So let me pass this on to, let me see. Oh, let me kind of quickly, I
thought this was on. Okay, so just a really quick basic architecture. This kind of pictorially
described to you what I just talked about. We only have so little time. So this is just
describing to you, right? Producers, consumers can be written in, you know, many different
languages, not just with Java, and it gets managing, you know, by bookkeeper that deals
with all of the storage side of things, and very dynamic. As you can see, this kind of
quickly summarized in picture what Pulsar can do for you. Okay, and then here, just quick
summary Apache Pulsar. Again, take mixtures of a pop-up type of architecture, right, and
that's what it is, and supports like multi-tenants, namespaces. Different subscription modes
do that. You can also leverage on that, essentially turn Pulsar into a queuing kind of capability
if you use an exclusive type of mode to do, you know, subscription. And what other thing?
Yeah, so there are different modes. It's just highly flexible is what we're trying to tell
you about the story. So here, we have a little bit of story about that. We can talk more
about it later. Yeah, so I just want to map Pulsar concept to JMS. JMS is pretty straightforward.
So the model is quite flexible because it is with a queuing, but also a pop-sub. And
in Pulsar, the mapping is really natural because you can map a JMS topic to a Pulsar topic,
whatever it is, Pulsar standard topic, partitioned topic, virtual topics. A JMS queue is like
a Pulsar shared subscription, and the JMS is like a Pulsar message with an envelope and
with the body. So in JMS, we have several consumer types. So I'm not going to enter
the details, but there is a subscription type that matches the JMS requirements. One important
thing is that if you want to use JMS with Pulsar, you don't need to install any additional
plugin because the JMS API is built over the standard native Java client because the Pulsar
features are a super set of JMS. So it's only about implementing an API. You know, in JDBC,
you have an API that allows you to connect to every database. In JMS, you just have to implement
the API and follow the specs. If you want, you can deploy a server-side component just
to push some of the computations. So for instance, in JMS, you have filters. You can filter the
messages. So if you want, you can filter them on the broker. Otherwise, you can simply
filter them on the client side. I'm just showing some examples of how to use Pulsar with JMS.
Maybe if you are already familiar with JMS, that's pretty simple. So in JMS, you start with a
connection factory. So we have Pulsar connection factory. And this is JMS 2.0. And you can get
a JMS context. You get a reference to a destination. This is create queue. Create queue is not creating
a queue. It's creating a reference to a queue because JMS doesn't deal with administrative
operations, as Mary said. You create a producer. You can send as many messages as you want. And
if you want to consume, you create a consumer. And you can use receive or set the message listener.
This is from standard Java. If you're using Jakarta or Java Enterprise, actually, yes,
I've been helping a few companies to migrate from Java Enterprise to Pulsar. So I know much
more cases about Java Enterprise more than Jakarta. But that's it. So for instance, if you
want to write and you have an Enterprise Java bin, then you can ask to the container to inject the
connection to Pulsar. And this is a standard Java Enterprise code. So this code runs with
ActiveMQ, with TIBCO, with whatever you want, whatever you are running. And the container
injects the connection factory and the destination. And you can, as in the standard Java code, you
can get a reference to the JMS context and then you send. We will see later how the administrator,
for instance, with Apache Tomy, connects all the parts. The consumer, usually in Java Enterprise,
you use message driven bins to consume from destinations. So yes, this is a simple message
driven bin. You configure all the relevant things that you want. For instance, usually you
configure the destination that is still a logical name and a subscription type or the
parallelism of the kind of things. In many containers, you can configure the things on
other descriptors or descriptors on user links and files. You implement a callback on message.
Every time a message is dispatched to the application, the code runs and if everything
goes well, the message is acknowledged to the Pulsar broker and it won't be delivered anymore.
If there is any exception that is thrown, Pulsar will deliver again the message.
In Tomy, there is a very simple way to deploy the resource adapter. I'm deploying the resource
adapter for Pulsar. So Pulsar RA, you configure the connection to Pulsar. Now in the demo,
I'm using localhost and this is the most interesting part. I create a logical queue,
so full queue. This is a queue and I bind it to a physical destination. So the container
will create a Pulsar connection factory and also the Pulsar queue.
The demo is on my GitHub space. So yes, you can run it by yourself. I'm going to use Apache Tomy
8, Starlight for JMS. I'll talk about that later. It is basically the JMS implementation.
I create the object with the same file that we saw and Apache Pulsar to.11.
So we have one application that consumes, one that produces and Pulsar will run locally.
So let me switch to the console. Oh no, yes, the code. The code is really simple. This is on
GitHub, so you can check it out later. So this is the producer. I'm not writing the code that
instantiates or assigns some value to the factory or to the queue. I'm scheduling the execution
of this method every two seconds and that's it. Very easy. On the JMS listener, these are two
separate applications. Usually in a real world application, you have some application that
produce the data. Then you have a pipeline that transforms your data and something else that
consumes the data. This is pretty common. So here, on message, depending on the type of message,
I'm printing the content and message. Here, I'm just declaring the reference to the logical queue
that I want. In this case, OpenAJB that is still Tommy will resolve the binding with the physical
queue via JNDI. We are running out of time. So I have a script to run all the demo. The script
simply installs two instances of Tommy, Pulsar, copies the configuration file, deploys the
resource archives, changes some ports because I'm running multiple services on my machine.
So there will be conflicts. Copy the consumer application to Tommy one, copy the producer
application to Tommy two, then start the Pulsar standalone. That is a quick way to start Pulsar
locally with all the services, but only in one JVM process. Tommy one, Tommy two, and then we will
see the logs. So there is some noise initially because it is installing everything. This is
Pulsar. This is starting. These are the two Tommy. Actually, we don't see. Oh yes, this is good. So
Tommy two is sending the messages. Tommy one is receiving the messages. So it works. It's a very
straightforward setup and very common way to develop with Java Enterprise. Let's drop up.
Two minutes probably. Yes, okay, good. So JMS is very useful and it allows you to switch very
easily to another vendor. Usually with JMS you don't use very specific features. Usually in my
experience with JMS, maybe you're using TIBCOR, you're using ActiveMQ. You configure on the container
some special flags, but the code usually is pretty standard. Yes, so switching to Pulsar is usually
easy. Pulsar is cloud native. It's scalable horizontally. So like Mary said, really, if you
it looks like a promise, but this is real, you can add machines, add or remove machines, and
the service automatically adapts. Actually, at Datastax we are running it as a service on the
cloud. And so this is very powerful because you can automatically adapt the resource consumption.
And also you can move the data that is not actually consumed to tier storage. And this
allows you to really lower the cost. It's open source. It's a vibrant community. If you want,
you can reach out to me on the community. And there are many people that are very enthusiastic.
Pulsar is young. It is only five years old, something like that. But in the past two years,
it grew very fast because it is really the next generation. Maybe someone working with ActiveMQ,
then I did it in my previous jobs, ActiveMQ and then Kafka and then Pulsar. Now it's time for Pulsar.
If you want to use Pulsar, you can use Starlight for GMS. I'm the initial author and main maintainer
for Starlight for GMS. So yes, feel free to ask me any questions. It's open source. It's on GitHub.
Pulsar Connection Factory, if you're using standard Java, there is a resource adapter
that works well with many containers. And it's already tested and it is running on production.
Okay. And these are just real quick. If you like, get this copy of the slide deck. But otherwise,
there are resources in here, community info, references to all the Pulsar information on
GitHub and also in our Pulsar site. And also then just additional information too with data stacks.
If you're interested, we offer like the $25 credit per month for personal projects. So
wanting to share with you, I know it's not true, open source in that sense, but we do have astra.datastacks.com
and all of the astra streaming is our company's supporting this in our cloud. So oops, where did
it go? Sorry. You tried to subscribe to us. Okay. So how do you contact us? This is the slide
just containing information about Twitter handles and LinkedIn, all of these things. So please do
consider staying in touch with us. We'll be very happy to answer more questions that you may have
and all you want to share with us, your project idea, we'll be happy to answer. And those sound
to Jay's luck. Yes. That's right. So thank you. Thank you so much. And I think that's any questions.
Sure. What the Pulsar functions or Pulsar function is a lightweight processing framework that
usually it's very easy to enrich the data that you have on your topics. So it's
for very lightweight processing. So if you have to do more complicated processing, you usually
move to something like Flink or other things. But Pulsar function is very useful when you have to
really process your data. And also it is the base for Pulsar.io that is the connector framework.
So basically in Pulsar, you can deploy on the Pulsar cluster your code that transforms your
data on your topics. Yes. It starts from a message on Pulsar and usually it ends with another
message on Pulsar. So it's really useful for transforming the data that is on Pulsar or
to push your data outside of Pulsar. I don't know if this answers. We need to continue. Oh, yes.
There is a question over here. If you want to have a discussion and also on Fuji Slack,
you can have discussions with people, but usually at the top they are married.