[00:00.000 --> 00:10.000]  This is the largest room I've ever given a presentation to, so thank you for showing
[00:10.000 --> 00:11.000]  up.
[00:11.000 --> 00:14.200]  Also, let's see.
[00:14.200 --> 00:17.720]  How many of you are using logging right now in your job?
[00:17.720 --> 00:18.720]  Wow.
[00:18.720 --> 00:20.960]  Okay, that's a lot of you.
[00:20.960 --> 00:21.960]  This is the wrong talk.
[00:21.960 --> 00:23.800]  No, I'm just kidding.
[00:23.800 --> 00:27.200]  How many of you would say you're doing Python logging well?
[00:27.200 --> 00:29.880]  Yeah, that's what I thought.
[00:29.880 --> 00:30.880]  Well, it's okay.
[00:30.880 --> 00:34.400]  You're in the right conversation because we're going to tell you all about how you get started
[00:34.400 --> 00:41.320]  with Python logging and all the wonderful things that, hold on, that's my boss, hold
[00:41.320 --> 00:42.320]  on.
[00:42.320 --> 00:45.440]  Hey, yeah, I'm in the middle of a presentation right now.
[00:45.440 --> 00:47.480]  What's up?
[00:47.480 --> 00:49.320]  He deployed on a Saturday?
[00:49.320 --> 00:50.320]  Are you kidding me?
[00:50.320 --> 00:52.720]  What do you mean, production's broken?
[00:52.720 --> 00:56.400]  Yeah, tell me you check the logs.
[00:56.400 --> 00:57.400]  He's not logging.
[00:57.400 --> 00:58.400]  Oh my goodness.
[00:58.400 --> 01:01.000]  Well, please don't fire him.
[01:01.000 --> 01:02.920]  Tell him to tune into my talk.
[01:02.920 --> 01:06.240]  We're going to go over the basics of logging and then we're going to get all the way down
[01:06.240 --> 01:08.640]  into some more advanced logging configurations.
[01:08.640 --> 01:09.640]  Okay?
[01:09.640 --> 01:10.640]  Yeah.
[01:10.640 --> 01:11.640]  Ciao.
[01:11.640 --> 01:15.280]  So, nobody wants to be in that situation.
[01:15.280 --> 01:22.600]  Honestly, that went over way better than I thought it would.
[01:22.600 --> 01:24.000]  Nobody wants to be in that situation.
[01:24.000 --> 01:26.360]  So before we get too deep though, I'm going to just say hi.
[01:26.360 --> 01:27.360]  I'm David.
[01:27.360 --> 01:29.960]  I'm a developer advocate for OpenSearch.
[01:29.960 --> 01:32.760]  Thanks to the OpenSearch group for letting me come here.
[01:32.760 --> 01:39.440]  OpenSearch is commonly used as a log store, so it fits in nice with Python logging.
[01:39.440 --> 01:44.200]  Previous to this, I'd worked as a data engineer, network automation engineer, DevOps engineer.
[01:44.200 --> 01:48.400]  I've done a lot of engineering and all of them had one thing in common and it was I
[01:48.400 --> 01:53.360]  needed a lot of logs to really understand what was going on at any point in my application.
[01:53.360 --> 01:55.720]  So you might want to get your phones out at this point.
[01:55.720 --> 01:57.280]  I have lots of QR codes.
[01:57.280 --> 01:58.280]  Just kidding.
[01:58.280 --> 01:59.280]  There are only three.
[01:59.280 --> 02:01.000]  And we're going to go to code right after this one.
[02:01.000 --> 02:05.240]  So that's my link tree if you want to connect with me and hear more about OpenSearch because
[02:05.240 --> 02:07.720]  I'll talk your ear off, I promise.
[02:07.720 --> 02:11.000]  So with that, you can follow along.
[02:11.000 --> 02:12.920]  There's a gist for this whole presentation.
[02:12.920 --> 02:14.720]  It can all run for the most part.
[02:14.720 --> 02:18.640]  I was working on it last night till very late.
[02:18.640 --> 02:20.280]  And we're going to go ahead and get started.
[02:20.280 --> 02:26.120]  So as with anything, we're going to import logging off to the races.
[02:26.120 --> 02:30.760]  And now you can check off your JIRA card and put it on the backlog.
[02:30.760 --> 02:31.760]  Just kidding.
[02:31.760 --> 02:34.200]  We all know that logging is a lot more than just importing logging.
[02:34.200 --> 02:36.440]  So we're going to start with the logger.
[02:36.440 --> 02:41.480]  Logger is kind of the core concept of Python logging and all loggers start with a name.
[02:41.480 --> 02:44.520]  So we're going to do underscore underscore name.
[02:44.520 --> 02:50.080]  Does anyone know what underscore underscore name actually will resolve to in this instance?
[02:50.080 --> 02:52.440]  Any guesses?
[02:52.440 --> 02:53.640]  It's underscore underscore main.
[02:53.640 --> 02:55.760]  Come on, guys.
[02:55.760 --> 02:58.040]  So we create a logger.
[02:58.040 --> 02:59.040]  We give it a name.
[02:59.040 --> 03:01.640]  And fun fact of the day, that logger is global.
[03:01.640 --> 03:10.680]  I can import Python logging and do get logger, of course, from anywhere and use this logger.
[03:10.680 --> 03:11.920]  As it's too small.
[03:11.920 --> 03:12.920]  Yes.
[03:12.920 --> 03:14.240]  There we go.
[03:14.240 --> 03:17.880]  Try not to delete code while I'm at it.
[03:17.880 --> 03:18.880]  There we go.
[03:18.880 --> 03:20.640]  We're going to make this smaller.
[03:20.640 --> 03:22.960]  We're going to make this up here.
[03:22.960 --> 03:26.680]  So as we know with logging, there are several different levels you can set.
[03:26.680 --> 03:30.480]  These levels are a filtering mechanism for Python logging.
[03:30.480 --> 03:35.360]  Fun fact of the day, though, most people don't know this, is these logger or these levels
[03:35.360 --> 03:37.360]  actually resolve two numbers.
[03:37.360 --> 03:40.200]  And those numbers are used to do filtering.
[03:40.200 --> 03:46.880]  And fun fact of the day, you can actually make your own log levels as if five, six,
[03:46.880 --> 03:49.160]  six log levels, seven log levels isn't enough.
[03:49.160 --> 03:51.600]  You could add more potentially if you needed that.
[03:51.600 --> 03:55.160]  So we're going to go ahead and send this to terminal.
[03:55.160 --> 03:57.720]  And we're going to take a look at all those log levels.
[03:57.720 --> 04:00.120]  Here we are.
[04:00.120 --> 04:05.920]  As we can see, they resolve to 10, 20, 30, 30, 30, 30, 40, and 50.
[04:05.920 --> 04:09.120]  So warn and warning both resolve to the same log level.
[04:09.120 --> 04:15.200]  So they're both there in case you need two log levels that have the same level.
[04:15.200 --> 04:17.080]  And these levels, again, are used for filtering.
[04:17.080 --> 04:23.040]  So as you are sending out logs, they're used to filter out logs that maybe are less important.
[04:23.040 --> 04:27.880]  So the number, the lower the importance of the log.
[04:27.880 --> 04:29.560]  So let's talk about emitting a log.
[04:29.560 --> 04:34.200]  Here we're going to go ahead and send a log with logger.info.
[04:34.200 --> 04:40.440]  So that's going to emit a log from the logger with the info level attached to it.
[04:40.440 --> 04:45.840]  Upper case info is for sending logs, upper case info is the level.
[04:45.840 --> 04:47.560]  And nothing actually happens.
[04:47.560 --> 04:50.680]  It just gets sent to the terminal and nothing happens.
[04:50.680 --> 04:55.320]  And the reason why nothing happens is because we've not told it where it needs to go.
[04:55.320 --> 04:57.480]  We need log handlers.
[04:57.480 --> 05:00.960]  So that is what we do.
[05:00.960 --> 05:06.880]  We're going to build a simple syslog handler for, sorry, streaming handler for sending to
[05:06.880 --> 05:08.560]  standard out now.
[05:08.560 --> 05:11.240]  So handlers receive logs from the logger.
[05:11.240 --> 05:12.760]  It's pretty straightforward.
[05:12.760 --> 05:17.840]  I'm going to go ahead and create this real quick.
[05:17.840 --> 05:19.920]  We're going to set the level of our handler.
[05:19.920 --> 05:23.840]  Remember our levels are our wonderful filtering mechanism.
[05:23.840 --> 05:25.920]  Set the level to warning.
[05:25.920 --> 05:31.760]  And the top one will not show up, but the bottom one should because it is greater than
[05:31.760 --> 05:33.240]  warning.
[05:33.240 --> 05:34.240]  And there it is.
[05:34.240 --> 05:35.960]  This will.
[05:35.960 --> 05:37.680]  That's pretty dull.
[05:37.680 --> 05:39.920]  Nobody really wants a log that says this will.
[05:39.920 --> 05:41.680]  That tells us nothing.
[05:41.680 --> 05:44.040]  You have no clue what's going on in your application.
[05:44.040 --> 05:46.120]  What time did this will?
[05:46.120 --> 05:48.880]  What time will it won't?
[05:48.880 --> 05:50.360]  So we need to add some context.
[05:50.360 --> 05:54.600]  And the way we get context is, of course, with log formatters.
[05:54.600 --> 05:58.840]  And I've jumped ahead too far in my presentation.
[05:58.840 --> 06:01.720]  We're just going to take a quick back step real quick.
[06:01.720 --> 06:04.640]  We're going to talk about some of the other handlers that are built in.
[06:04.640 --> 06:06.520]  So there's the rotating file handle.
[06:06.520 --> 06:11.400]  This one's particularly useful if you have too many logs and your file gets too large.
[06:11.400 --> 06:18.240]  It can actually automatically rotate that file every X amount of size into a new file.
[06:18.240 --> 06:22.800]  And then you can specify for it to delete logs after a certain amount of time or a certain
[06:22.800 --> 06:23.800]  amount of files.
[06:23.800 --> 06:26.160]  We have the syslog handler.
[06:26.160 --> 06:28.240]  Syslog is a standard for logging.
[06:28.240 --> 06:33.040]  HTTP lets you send logs to arbitrary HTTP endpoints.
[06:33.040 --> 06:38.760]  Time rotating file handler, believe it or not, is a timed rotation of your log files.
[06:38.760 --> 06:42.720]  Whether it be every day, every minute, or please God forbid every second, do not do
[06:42.720 --> 06:43.720]  that.
[06:43.720 --> 06:45.000]  You will end up with thousands of files.
[06:45.000 --> 06:48.280]  I did actually every hour once, and that was a huge mistake.
[06:48.280 --> 06:52.560]  I ended up with like 15,000 files after a little bit.
[06:52.560 --> 06:58.160]  And the SMTP handler, if you are masochistic.
[06:58.160 --> 07:00.400]  Now we'll talk about formatters.
[07:00.400 --> 07:01.560]  OK.
[07:01.560 --> 07:02.960]  So let's go ahead.
[07:02.960 --> 07:07.640]  We're going to create, set our console handle to the info level.
[07:07.640 --> 07:09.040]  We're going to create a formatter.
[07:09.040 --> 07:13.000]  The formatter is going to include the date and time, the name of the logger.
[07:13.000 --> 07:16.240]  Which again, if you're using underscore underscore name, that's going to be the name of your
[07:16.240 --> 07:20.520]  module, whether that's like search dot util dot whatever.
[07:20.520 --> 07:25.680]  And then the level name that it was admitted at, and a message.
[07:25.680 --> 07:26.800]  Pretty straightforward.
[07:26.800 --> 07:28.800]  Then we set the formatter.
[07:28.800 --> 07:31.400]  Formatter gets attached to handlers again.
[07:31.400 --> 07:33.600]  Logger dot info.
[07:33.600 --> 07:35.920]  Look at my pretty log.
[07:35.920 --> 07:37.800]  Well, just kidding.
[07:37.800 --> 07:39.640]  There's no log there.
[07:39.640 --> 07:41.760]  Is it because I have a bug?
[07:41.760 --> 07:44.040]  Because I wrote this way too late at night?
[07:44.040 --> 07:45.040]  Probably.
[07:45.040 --> 07:49.320]  Or more likely because there was something I actually tricked you on earlier.
[07:49.320 --> 07:53.640]  The truth is, the reason our first log wasn't admitted wasn't because there wasn't a handler
[07:53.640 --> 07:59.480]  attached to it, but it's because Python's logging library by default sets loggers and
[07:59.480 --> 08:04.680]  handlers to the warning level right out of the gate, and we admitted at the warning level.
[08:04.680 --> 08:11.680]  So fun fact, you actually need to set the level for both the logger and your handlers.
[08:11.680 --> 08:12.680]  So we'll go ahead and do that.
[08:12.680 --> 08:16.520]  We'll set the logger level, and we're off to the races.
[08:16.520 --> 08:18.200]  Let's look at this pretty log.
[08:18.200 --> 08:21.120]  Man, that is so beautiful.
[08:21.120 --> 08:24.320]  Now I know exactly when I had a pretty log.
[08:24.320 --> 08:26.720]  I know exactly where it happened in the main.
[08:26.720 --> 08:28.280]  I know what level it was at.
[08:28.280 --> 08:31.720]  It was informational log.
[08:31.720 --> 08:34.040]  So we've talked about a lot so far.
[08:34.040 --> 08:37.120]  We've talked about loggers, handlers.
[08:37.120 --> 08:41.720]  So let's do a top to bottom just real quick to make sure we're all talking about the
[08:41.720 --> 08:43.320]  same thing.
[08:43.320 --> 08:45.760]  So we have loggers.
[08:45.760 --> 08:48.760]  Loggers emit a log at some level.
[08:48.760 --> 08:51.560]  They also filter out logs at some level.
[08:51.560 --> 08:56.680]  So if your logger is set to filter out warning logs and you send it an info, it's never
[08:56.680 --> 08:59.440]  going to get to the handler.
[08:59.440 --> 09:03.640]  Handlers receive logs and then send them to some specified output, whatever that may
[09:03.640 --> 09:04.640]  be.
[09:04.640 --> 09:08.320]  And then formatters attach to handlers, and they enrich the output.
[09:08.320 --> 09:09.920]  So they add context.
[09:09.920 --> 09:11.920]  And there's a bunch of other contexts.
[09:11.920 --> 09:12.920]  That's not mentioned here.
[09:12.920 --> 09:17.040]  It's in a Python logging library.
[09:17.040 --> 09:20.440]  I guess it's like at a certain time in the presentation, it just decides it wants to
[09:20.440 --> 09:23.080]  do that.
[09:23.080 --> 09:29.400]  So these are all wonderful.
[09:29.400 --> 09:30.400]  Wait a minute.
[09:30.400 --> 09:31.400]  I'm getting there.
[09:31.400 --> 09:32.400]  Logging.
[09:32.400 --> 09:33.400]  Logging.
[09:33.400 --> 09:34.400]  Yes.
[09:34.400 --> 09:35.400]  All right.
[09:35.400 --> 09:36.400]  Here we go.
[09:36.400 --> 09:37.400]  We're just going to scroll on.
[09:37.400 --> 09:38.400]  Oh, yes.
[09:38.400 --> 09:39.400]  Here we go.
[09:39.400 --> 09:42.440]  So setting up logs from, you know, in each individual module is a pain.
[09:42.440 --> 09:47.960]  So you can actually pre-create loggers ahead of time with a dictionary config or a YAML
[09:47.960 --> 09:52.880]  config file because I know all of us just love creating YAML configs and having them
[09:52.880 --> 09:54.280]  everywhere.
[09:54.280 --> 10:02.240]  So with this, you can create as many logs as you want and specify them with dictionary
[10:02.240 --> 10:03.500]  config.
[10:03.500 --> 10:06.960]  One other really important thing to mention, and I probably should have hit it when I was
[10:06.960 --> 10:10.960]  talking about loggers at the get-go, there's a specific reason why you can set it at the
[10:10.960 --> 10:13.960]  logger level and at the handler level.
[10:13.960 --> 10:18.320]  So handlers give you very fine-tune access over what you're looking at.
[10:18.320 --> 10:21.280]  So where it's going, which output, you know.
[10:21.280 --> 10:25.640]  So a lot of times people will specify certain levels for handlers and then they will use
[10:25.640 --> 10:29.200]  their global debug level to set their loggers.
[10:29.200 --> 10:33.960]  So say, for example, you're debugging an application locally, you're going to set that to debug,
[10:33.960 --> 10:38.320]  of course, but when you push it to production, you don't want that.
[10:38.320 --> 10:44.200]  So a lot of people will have, you know, a production logging config and a development
[10:44.200 --> 10:46.440]  environment logging config.
[10:46.440 --> 10:55.480]  So with that, there is actually one other slight challenge with loggers, and that is
[10:55.480 --> 10:57.640]  they're blocking operations.
[10:57.640 --> 11:04.000]  So like I said earlier, if you're a masochist and you like the SMTP log handler, you could
[11:04.000 --> 11:05.200]  be in a real pinch.
[11:05.200 --> 11:10.600]  So say, for example, I'm on your application, you have this nice web server, and all of
[11:10.600 --> 11:17.120]  a sudden it hits a critical error, it sends a message to your SMTP server, and your SMTP
[11:17.120 --> 11:22.880]  server is slow, it's chugging, so you're taking five to ten seconds for it to register
[11:22.880 --> 11:24.720]  that and send a response.
[11:24.720 --> 11:28.680]  Do you think I'm going to stay on your web page for five to ten seconds while it sends
[11:28.680 --> 11:29.680]  an error log?
[11:29.680 --> 11:33.720]  Heck, no, I'm closing out and I'm going to somewhere else, I don't know, amazon.com
[11:33.720 --> 11:35.720]  to buy whatever I needed.
[11:35.720 --> 11:38.320]  So we have to handle this.
[11:38.320 --> 11:42.320]  We have to understand that, hey, this could potentially block, so how do we unblock our
[11:42.320 --> 11:43.320]  applications?
[11:43.320 --> 11:48.880]  Well, it's by making our applications simpler, obviously, and using multi-threading.
[11:48.880 --> 11:52.840]  That was a joke, you can laugh.
[11:52.840 --> 11:58.480]  So with this, we can actually import queues, and what happens is there's a queue handler
[11:58.480 --> 12:00.400]  and a queue listener.
[12:00.400 --> 12:06.360]  So the queue is the shared memory space that can be accessed both the handler and the listener.
[12:06.360 --> 12:11.880]  You create the handler, the handler receives all the logs, and then distributes them to
[12:11.880 --> 12:12.880]  the queue.
[12:12.880 --> 12:17.960]  The queue listener starts up on its own independent thread, and it's going to listen to the log
[12:17.960 --> 12:23.600]  queue, and then distribute it to any of the handlers that you specified it should.
[12:23.600 --> 12:27.000]  So let's go through that end to end again.
[12:27.000 --> 12:30.840]  We've got queue handlers, receive the logs, place them on a queue, the queue hands it
[12:30.840 --> 12:35.520]  over to the queue listener, which then hands it on to your other loggers.
[12:35.520 --> 12:37.320]  Now your application is unblocked.
[12:37.320 --> 12:41.320]  It drops that queue, or that log on the queue, and then it's off to the races.
[12:41.320 --> 12:47.760]  You can use SMTPlib if you really wanted to.
[12:47.760 --> 12:51.520]  So let's talk about pulling this all together now, right?
[12:51.520 --> 12:57.480]  So we have these logs, they sit on our local machines, and that's fine, but if you are
[12:57.480 --> 13:03.160]  a large organization, you might have hundreds of servers.
[13:03.160 --> 13:05.560]  Take a second to breathe.
[13:05.560 --> 13:10.240]  You might have hundreds of servers, hundreds of network devices or whatever, and I'll give
[13:10.240 --> 13:13.960]  a real example of when I could have used this.
[13:13.960 --> 13:19.080]  So when I was a network automation engineer, we had a particular log that my boss found
[13:19.080 --> 13:25.200]  on some of the servers, or router switches, et cetera, and I spent the next three hours
[13:25.200 --> 13:27.440]  logging into each individual one.
[13:27.440 --> 13:31.240]  We had thousands of network devices.
[13:31.240 --> 13:36.440]  So I logged into enough of them, wrote down the logs, and then correlated, and I said,
[13:36.440 --> 13:44.040]  oh, look, every Thursday and Saturday at the exact same time, this log happens.
[13:44.040 --> 13:49.080]  One email later finds out security team is pen testing against us, and we don't need
[13:49.080 --> 13:50.720]  to worry about it.
[13:50.720 --> 13:56.520]  Again, three hours, just trying to correlate what log was happening where, when.
[13:56.520 --> 14:02.600]  So this is exactly what you can avoid by using something like OpenSearch, ElasticSearch,
[14:02.600 --> 14:06.080]  Loki to aggregate your logs.
[14:06.080 --> 14:10.240]  And again, if you do want to follow along with this later, this just is a Docker compose
[14:10.240 --> 14:14.600]  file that will let you spin up some sample containers with OpenSearch.
[14:14.600 --> 14:25.400]  So we'll import logging in our OpenSearch library, create an OpenSearch client, and this is where
[14:25.400 --> 14:30.280]  I'm going to break for just a moment and talk about custom handlers.
[14:30.280 --> 14:33.400]  So we mentioned handlers are where you send your logs.
[14:33.400 --> 14:36.840]  You can implement custom handlers, believe it or not.
[14:36.840 --> 14:42.280]  All you need to do is, I was going to say, inherit from, there we go, that's probably
[14:42.280 --> 14:45.240]  the right word, logging.handler.
[14:45.240 --> 14:52.800]  Then you create and emit definition, and that needs to have self and record, and that will
[14:52.800 --> 14:55.640]  send the record wherever you specify.
[14:55.640 --> 15:00.840]  So in our case, we have, it's going to take, and it's going to format created time.
[15:00.840 --> 15:06.000]  Also I did not implement the formatting library, because I wanted to send it as a dictionary
[15:06.000 --> 15:08.720]  to OpenSearch just because that's what it works with.
[15:08.720 --> 15:14.840]  You can also use something like FluentDit, FluentBit, or FluentDlogStash to parse out
[15:14.840 --> 15:18.480]  your logs later, and we'll talk about that just briefly.
[15:18.480 --> 15:20.280]  So we've got our created time.
[15:20.280 --> 15:30.280]  We've created this wonderful record, OpenSearchClient.index, we'll send that log to OpenSearch.
[15:30.280 --> 15:32.280]  There we go.
[15:32.280 --> 15:33.280]  And then we'll set it up.
[15:33.280 --> 15:36.360]  So we're going to create our logger named log.
[15:36.360 --> 15:40.880]  We're going to set the logger's level to info so that we get the info records.
[15:40.880 --> 15:47.040]  We're going to create the OpenSearchHandler and add logging.info, set that level, and
[15:47.040 --> 15:50.400]  add our handler.
[15:50.400 --> 15:52.760]  And we're off to the races.
[15:52.760 --> 15:53.760]  Boom.
[15:53.760 --> 15:56.400]  Well, that was kind of anticlimactic.
[15:56.400 --> 15:58.520]  You can't actually see it going into OpenSearch.
[15:58.520 --> 16:03.120]  But I promise you, it chugged along and it went into OpenSearch and, ha-za, let's go
[16:03.120 --> 16:05.320]  into OpenSearch.
[16:05.320 --> 16:07.640]  And this is actually OpenSearch dashboards.
[16:07.640 --> 16:14.560]  So again, if you do this on your local machine, passwords admin, usernames admin.
[16:14.560 --> 16:15.560]  Very secure.
[16:15.560 --> 16:19.520]  We're actually looking to change that, but that is coming soon.
[16:19.520 --> 16:26.760]  So we go here and we're going to go into stack management, hang with me for just a second.
[16:26.760 --> 16:32.640]  So we created a custom index with this, and that index is named based off of the logger
[16:32.640 --> 16:33.800]  that sent it.
[16:33.800 --> 16:42.200]  So we've got logger name, which was log, and then with the date time so that we can roll
[16:42.200 --> 16:45.520]  those off after a certain amount of days.
[16:45.520 --> 16:49.200]  We're going to create an index pattern.
[16:49.200 --> 16:55.560]  And we are going to say, we're going to say absolutely nothing because this is the problem
[16:55.560 --> 16:58.880]  with doing things like, oh, just kidding, I just don't know how to use OpenSearch.
[16:58.880 --> 16:59.880]  Here we go.
[16:59.880 --> 17:00.880]  It's only my job.
[17:00.880 --> 17:01.880]  Don't worry.
[17:01.880 --> 17:04.600]  Boss, I swear, if you're watching this, no, I'm just kidding.
[17:04.600 --> 17:05.600]  So there we go.
[17:05.600 --> 17:09.520]  So we have these two indexes that have been created because I was testing yesterday and
[17:09.520 --> 17:11.280]  today.
[17:11.280 --> 17:16.040]  So logstar just says, hey, any index that looks like log with anything after it, group
[17:16.040 --> 17:19.200]  them together.
[17:19.200 --> 17:23.680]  Ask for a time field, which is our created field, and create index pattern.
[17:23.680 --> 17:24.840]  There we go.
[17:24.840 --> 17:30.720]  OpenSearch, auto-detects, all of these different files, types, et cetera.
[17:30.720 --> 17:35.000]  You can actually specify mappings, and I actually really recommend that because different visualizations
[17:35.000 --> 17:38.800]  require different types of mappings, but that is not what we're talking about today.
[17:38.800 --> 17:41.360]  We're talking about Python logs, darn it.
[17:41.360 --> 17:48.760]  So we're going to go over to Discover and go into our logs, and we are going to make
[17:48.760 --> 17:50.760]  sure we're looking at today's logs.
[17:50.760 --> 17:53.280]  Actually, well, let's look at this week.
[17:53.280 --> 17:54.760]  Here we go.
[17:54.760 --> 17:56.480]  And we have three hits.
[17:56.480 --> 17:57.480]  There we are.
[17:57.480 --> 17:58.480]  These are our logs.
[17:58.480 --> 17:59.480]  Look ma, I'm logging.
[17:59.480 --> 18:02.880]  And you could actually see the exact point in time when I switch from this is my log
[18:02.880 --> 18:04.720]  to look ma, I'm logging.
[18:04.720 --> 18:12.400]  So that was, yeah, I had 2.37 p.m., so anyhow, with this, you can actually go ahead and
[18:12.400 --> 18:16.280]  then visualize spikes, peaks, valleys, when this is happening.
[18:16.280 --> 18:18.280]  You can enrich it with device information.
[18:18.280 --> 18:21.460]  So you can say, hey, when was this log sent?
[18:21.460 --> 18:23.960]  Is there a particular device that's having a lot of issues?
[18:23.960 --> 18:27.840]  So now instead of logging into all of your servers, you can go and bounce the correct
[18:27.840 --> 18:29.880]  ones.
[18:29.880 --> 18:35.360]  And if you saw Doton's presentation yesterday, you'll know you can use this for monitoring
[18:35.360 --> 18:42.840]  anything, whether it's CI CD pipelines, Python logs, network logs, et cetera.
[18:42.840 --> 18:44.880]  So let's see.
[18:44.880 --> 18:45.880]  Look at that.
[18:45.880 --> 18:48.160]  You're doing more logging now than 99% of the population.
[18:48.160 --> 18:49.160]  So congratulations.
[18:49.160 --> 18:57.160]  Clap your hands for yourselves.
[18:57.160 --> 19:01.280]  So I'm going to talk real quick about just a very simple, common logging architecture
[19:01.280 --> 19:07.320]  for capturing distributed logs and why you would want to do that.
[19:07.320 --> 19:14.960]  So more often than not, you're probably actually going to want to log your, you don't want
[19:14.960 --> 19:17.760]  to put your logs locally on the file system.
[19:17.760 --> 19:22.760]  And the reason why is because your file system is not going to go down, and God forbid if
[19:22.760 --> 19:26.280]  it goes down, there's no hope for that log getting out anyways.
[19:26.280 --> 19:30.720]  Your service remotely could disconnect because whether you're doing an upgrade or something
[19:30.720 --> 19:34.880]  along those lines, so it acts as a little bit of a caching mechanism.
[19:34.880 --> 19:38.560]  And then you'll normally have it logged to a file, and then you could use something like
[19:38.560 --> 19:43.000]  Fluent Bit or Beats, and those will ship your logs out.
[19:43.000 --> 19:47.360]  And the wonderful thing about this architecture is, again, if OpenSearch was to go down because
[19:47.360 --> 19:53.240]  you're doing an update, or if something was to happen critical with your Python app, it
[19:53.240 --> 19:58.200]  can quickly write that log out, and you can also do enrichment.
[19:58.200 --> 20:03.680]  So I talked about getting which server sent that log.
[20:03.680 --> 20:11.320]  So Fluent D, Data Prepper, and Log Stash all can take and enrich your logs with the context
[20:11.320 --> 20:13.200]  information that came with them.
[20:13.200 --> 20:20.520]  So say, for example, it says, I received this log from X, Y, and Z server, 10, 20, 90,
[20:20.520 --> 20:23.480]  32, 83, or something like that.
[20:23.480 --> 20:28.240]  Then it can go and do a reverse DNS lookup and say, hey, who has that?
[20:28.240 --> 20:30.440]  Which service is assigned to that?
[20:30.440 --> 20:33.800]  And then it can add that information in and then push it into OpenSearch.
[20:33.800 --> 20:40.160]  So now you've all of a sudden gone from having a log that says, hello world, to or got here,
[20:40.160 --> 20:43.160]  please, please, do not.
[20:43.160 --> 20:47.720]  And you can have it pushed into OpenSearch, know what service is causing the issues, and
[20:47.720 --> 20:50.880]  visualize on dashboards.
[20:50.880 --> 20:52.600]  With that, I'm finished.
[20:52.600 --> 20:55.080]  Please scan, look at OpenSearch if you're curious.
[20:55.080 --> 20:58.200]  It is open source, Apache 2 licensed.
[20:58.200 --> 21:00.960]  All of our features are being developed in the open.
[21:00.960 --> 21:05.040]  And with that, I want to ask it, does anyone have any questions?
[21:05.040 --> 21:06.040]  Yeah.
[21:06.040 --> 21:18.040]  Thank you.
[21:18.040 --> 21:20.640]  Hello?
[21:20.640 --> 21:21.760]  Okay.
[21:21.760 --> 21:23.160]  How would you...
[21:23.160 --> 21:27.800]  I'm mostly familiar with Sentry, and I'm very curious how would you compare this to that,
[21:27.800 --> 21:31.680]  because as far as my friends told me, they're pretty different products, but they do similar
[21:31.680 --> 21:32.680]  stuff.
[21:32.680 --> 21:33.680]  Yeah.
[21:33.680 --> 21:36.040]  So Sentry is more of an APM, isn't it?
[21:36.040 --> 21:37.040]  Is that word familiar?
[21:37.040 --> 21:38.040]  Okay.
[21:38.040 --> 21:43.600]  So OpenSearch has some APM capabilities, which is, I don't remember the word for it, but
[21:43.600 --> 21:46.640]  it's for app monitoring and specific to the application.
[21:46.640 --> 21:50.880]  So this has some APM capabilities.
[21:50.880 --> 21:55.840]  It might not go as deep as some of your auto configurations for other APM tools though,
[21:55.840 --> 21:57.600]  but it can ingest APM logs.
[21:57.600 --> 21:59.680]  So that's a good question.
[21:59.680 --> 22:00.840]  Thank you.
[22:00.840 --> 22:03.680]  You had two async questions from the chats.
[22:03.680 --> 22:08.840]  The first one is, what about f-strings and logging?
[22:08.840 --> 22:13.760]  Because I wrote this presentation in like a couple hours, and yeah.
[22:13.760 --> 22:19.280]  So I'm trying to modernize, but again, it's old habits die hard, so I'm still using f-strings.
[22:19.280 --> 22:21.040]  Please don't get on me.
[22:21.040 --> 22:24.440]  It was the next one, sorry.
[22:24.440 --> 22:25.440]  Yeah.
[22:25.440 --> 22:31.200]  The question is, what about structured logging, and in particular, strike log?
[22:31.200 --> 22:32.880]  I'm not actually familiar with strut log.
[22:32.880 --> 22:38.600]  So I'm actually moving myself more towards using OpenTelemetry instead of logging, or
[22:38.600 --> 22:40.200]  alongside logging, we'll say.
[22:40.200 --> 22:46.440]  So OpenTelemetry gives you a trace, actually, which can tell you the full stack of what
[22:46.440 --> 22:48.480]  happened during your application.
[22:48.480 --> 22:55.400]  So everything down from which function was called to which load balancer sent the information
[22:55.400 --> 22:56.400]  over.
[22:56.400 --> 22:59.920]  So you get an end-to-end trace of what happened, which in my opinion, I think, is a little
[22:59.920 --> 23:02.280]  bit more handy than just logs.
[23:02.280 --> 23:10.640]  I think we have one more question in the middle.
[23:10.640 --> 23:14.040]  Just a quick question from the discussion I'm having with the code worker.
[23:14.040 --> 23:19.520]  So we're thinking about moving our logs to JSON format, because it's easy to understand
[23:19.520 --> 23:22.960]  for non-Python people and searchable.
[23:22.960 --> 23:27.560]  If we were to switch to OpenSearch, and I really liked the presentation, do you think
[23:27.560 --> 23:34.600]  it's still feasible to make logs searchable in and of itself, or is OpenSearch that's
[23:34.600 --> 23:40.440]  stable and usable and from your experience, there's no need to search on your logs?
[23:40.440 --> 23:41.440]  Yeah.
[23:41.440 --> 23:42.440]  No, that's a great question.
[23:42.440 --> 23:44.120]  You do need to search your logs.
[23:44.120 --> 23:48.560]  OpenSearch is a search engine at its core, and that is why it is as good as it is with
[23:48.560 --> 23:50.200]  logs.
[23:50.200 --> 23:55.880]  As for JSON versus other formats, I think there's no particular preference, but OpenSearch
[23:55.880 --> 23:56.880]  is certainly stable.
[23:56.880 --> 24:01.560]  We have 150 million downloads, so we are here to stay.
[24:01.560 --> 24:07.840]  It's been adopted by a lot of companies such as Oracle, Ivan, Instacluster, Opster, Amazon
[24:07.840 --> 24:12.400]  Web Services, of course, because I work there, and many others.
[24:12.400 --> 24:17.960]  So I would say it is very stable, production-ready to use, and yeah, it's a really great way
[24:17.960 --> 24:18.960]  to search your logs.
[24:18.960 --> 24:24.480]  In fact, I have a lightning talk at 1655 in another room, so I think it's a Kubernetes
[24:24.480 --> 24:25.480]  room.
[24:25.480 --> 24:28.160]  So if you want to talk more about searching your logs, I'm going to be talking about search
[24:28.160 --> 24:29.160]  relevance for logs.
[24:29.160 --> 24:30.160]  So thank you.
[24:30.160 --> 24:31.160]  Thank you, online people.
[24:31.160 --> 24:48.880]  Thanks a lot.