Logging Concepts and Architecture

0:00It's time to talk about logs. That's right. We're not going to go out there with our lumberjack axe and chop down trees and make logs. We're talking about system logs. That's right. So we got Windows logs, Linux logs, all your network devices create logs. There's logs everywhere. So let's jump in and talk about logs. Let's get ourselves ready to understand why we need them, where they come from, what formats they're in, all these wonderful things. So let's get started. So the first question is, why do we need logs? That is a great question. So our systems all create logs, but why do we need them? There is definitely a need. And that's because logs tell the story about what happened. So without logs, we don't really know what happened. Okay.

0:50And to think about this, if there's an incident and you begin investigating this incident, take your magnifying glass and you go in there, what are you going to be looking at? Well, I'll tell you the logs because the logs, they tell a story about what happened. So here's a scenario. Let's say a user over here, uh, report suspicious logging activity at about 2 15 PM. Okay. And the windows authentication logs, we've got our windows logs here. They show this happening. All right. But there's a Linux application. We'll put an application down here running on a server. And it shows that this suspicious logging activity here also tried to access the application, but this shows up at 8 16 PM. So what's the discrepancy with these times here? Because if an attacker logs in with this suspicious activity, and then we see the login and windows, but the application is, well, six hours later, do you think that evil bad guy or, uh, just hung around for six hours, waiting to try to access the system's application?

2:03Of course not. And this can make it very difficult to do an investigation. So we're going to talk about this here, time discrepancy and how to solve that so that you have your logs all using the same time source. So they match up. Okay. So that is something very important. So let's go ahead and, and talk more about some laws. Where do these logs come from? Well, they come from all your operating systems. That's for sure. So you got your windows, you got your Linux. So, uh, here we go. I'm going to try to draw a little penguin here. And there we go. Okay. Give them a flappy arm here. There you go. There's our penguin. And then of course, uh, well, that's my apple. So there you go. Of course, it comes from your Mac and your Linux and windows. They all create logs, but also all of your network devices can create logs too. So you're talking about firewall, IPS, routers, switches, access points, all these things create logs. And the thing is even applications create logs. So all your apps as well. And this is wonderful because logs tell the story of what happened. All right. Now time is very important. And we kind of talked about a

3:25scenario here a minute ago where our windows authentication logged at 2 15 PM. And it was kind of questionable. There were some weird activities, some strangeness, but our application server over here said it happened at, I think, would we say 8 16 PM? So we had this time discrepancy there. So the thing is we need to be using something called network time protocol. Okay.

3:56And network time protocol known as NTP uses UDP port one 23. All right. Just so you know, that's kind of important because you've got to make sure that your firewalls aren't blocking it. If you're using it now, why do we use NTP? Well, on all of our systems here, no matter if it's windows, Mac, Linux, doesn't matter. You configure an NTP source and probably two of them, actually, maybe three. And what this is, that source is you're pointing to an NTP server and you want all of your devices, this includes your routers and your firewalls and your switches and access points and IPS, all of your things, all of them need to point to the same NTP source. That will ensure that they all have the same time they're synchronized. So like the, the spy movie synchronize your watches.

4:47Well, that's what we're doing. We're synchronizing our watches or clocks on all of our systems so that all the logs are going to have the same timestamp. So they're going to be able to be analyzed as one set of logs, because as you have these logs come in, if we had something at 2.15 and then 8.16 and you're trying to investigate them, you know, they're connected somehow, but this time really is a pain because now I have to go look at these logs and realize that they're six hours behind. So I have to continually to remember minus six hours. Well, when you're looking at five or six or 10 system logs, that's just a real pain. Let me tell you, it makes life hard. We don't want that. So NTP is how we fix that and make sure all of our logs are synchronized time-wise and all of our systems. All right. Super. And that is very important.

5:40Now log retention, something else. So how long is the question? Should you retain your logs? That's the big question. That's right. So what is the answer then? Well, here's the answer. Ready? This is like industry secret knowledge. Okay. I'm going to let you in on this. The secret is it depends. Was that a big letdown? I hope not. It depends. Okay. And that's because this is going to be based on your organizational policies. All organizations are going to have policies saying this is how long we retain logs. Okay. And it might be that some logs are retained longer than other logs. It just depends. But here's the thing. You need to understand that log retention is a thing. So why is this necessary? Why are we talking about it? Other than I need to know that, yeah, log retention is a thing. All right. Let me clear a little space here and let's talk about this. So we know that the policies are going to guide what we're doing with our log retention. Now, here's why it's a thing. On your systems, logs are only kept there for so long. So it may be basically, it could be based on time, how long you want to keep them,

6:59like 30 days or something. Or it could be space. So maybe you're going to allow for 500 megabytes of log space. Well, when you fill this space up or you reach that time limit, what's going to happen is, by default, generally, they start overwriting themselves. All right. Now, what's the problem with that? Well, if today is, we'll say, April 10th, and we realize, hey, something's not right. Something happened. There was an incident or something. So you start tracing back.

7:35Well, if you only keep so much log, let's say 30 days, if you keep 30 days worth of logs, what happens if that incident occurred 35 days ago? Well, guess what? You're not going to get the full story. You're going to get some after effects, but you don't really know what happened at the beginning of that incident. This is why log retention is important. Well, it's also important for compliance and some things like that. But for now, let's just focus on this and security and incidents. Okay. So it's very important that we understand log retention and that we need to keep it for a certain amount of time so that we can go back and look at what happened X number of days or weeks ago. So when it comes to log retention, though, let me make a little more room over here, right over here. There's something that you need to understand, because when you're retaining logs, well, space isn't free. That's what I'm getting at. And your logs, oftentimes what we'll do is you might actually send those logs off of the system to a log repository. And maybe this over here keeps the logs for 90 days. That way you have

8:55a separate copy of your logs, which is always good. And you can go back 90 days. And oftentimes this is known as like warm storage. And sometimes they even get shipped off from there to what's called cold storage. And it might be kept there for, you know, seven years, who knows. But the thing is, this is expensive here or well, somewhat relatively, but down here, cold storage is usually pretty cheap. However, it takes time to get access to this data. This data, you can just jump in and analyze as all, you know, till the sun goes down, till the cobs come home, all that. But this, you usually have to request access to it, and it takes a couple days. So there you go. That's your log retention.

0:00Okay, let's talk about a SIEM now, or SIEM. Some call it SIEM, some call it SIEM. Security Information and Event Management. So what this is, we talked here about a log repository. Well, this is a type of log repository. It is a SIEM. You send your firewall logs, your IPS logs, all your systems logs, application logs, and this has intelligence built into the platform. So you could have a general log repo server and all it does is, it's just a storage space for your logs. That's all it does. You can go in and search the logs, but that's it. Okay. Well, what a SIEM does, or SIEM, it has intelligence built into it. So it analyzes. So yeah, it does the analyzing of all of your log data and it's looking for abnormalities and malicious activity.

0:56And then it can sound the alarm over here, letting you know something's going on, so you can investigate. And that's what a SIEM is. Now I will say SIEMs or SIEMs, they are not meant for long term storage of your logs. Okay. If you want to keep long-term storage, you're not going to do it in your SIEM. And that's because of how much data gets into this database here and it can get bloated and get really slow. So it's not meant for long-term storage. We use a log repository for that. So continuing with our SIEM, it uses something called threat intelligence. And it's a feed that you sign up for and it can feed into your SIEM and it provides information like IP reputation, domain reputation, attack signatures, email reputation. And reputation is, is this a malicious IP or not? That kind of information. Is this email address or domain name used recently in attacks? Okay. So it's basically, it's making it smarter, keeping it up to date with the current activities that are going on in the security realm.

2:04Now, something else that SIEMs do is called normalization and correlation. So let's talk about normalization first. So here we go. So normalization, what this is, is the process of organizing, structuring, and standardizing of data. So really what it's doing is trying to reduce duplicate information and make searching easier. Let me give you an example of what this means. So if you've got logs coming in from various systems and devices, well, let's say the field names, because in logs, you have field names. Think of it like a spreadsheet. And one of the field names is going to be, let's say SRC underscore IP, and that equals a source IP address. But another system might call this source underscore IP and another one, origin underscore IP or origin client or source client. So they can have all these different field names, but guess what?

3:04It's the same thing. And what normalization does, it does away with all those names. Now they're still there under the hood, but it replaces it with one field name. So you can look through all your various system logs as if they were one big log. And that normalization really helps to make searching logs much easier because the next thing is correlation. So what are we doing in these logs? Well, we're looking at linking separate isolated security events from the logs and various alerts across different systems and time windows to form a single meaningful attack narrative. So really, what are we doing? We're telling the story of what happened. That's what correlation is. It says, oh, the firewall saw this source IP over here. Okay. We've got the IP address. We've got the server. Okay. We saw it come in. Oh, and I saw it then it shows up in this windows server where it went to, uh, it tried to log in and then maybe it went over to this web server over here and it tried to access something. So what we're doing is putting together the storyboard of what happened.

4:08That's what data correlation is. Okay. Super. Moving on. Okay. So why do we need SIM or syslog other than for log storage and analysis? Okay. I'll tell you why, because let's say we've got this bad guy down here and he breaks into one of our systems over here. Guess what is most likely going to do as part of his attack. I'll tell you, delete the logs. That's right. So your logs gone, poof vanished. So, uh, there's a problem. You realize there's an incident. You log in. There's no logs to look at. You don't know what happened. Here's the thing. If you are using syslog or maybe you install some type of agent to ship logs, it's a shipping agent. What happens is you're sending those logs to a log repo like syslog is or to a SIM. That means you have a separate copy of your logs. So during an attack, if these logs get erased, you still have a copy over here. That is another reason to use some type of log repo or SIM. All right. And as part of log ingestion and shipping, okay, which we just kind of looked at, there's various ways to do this, but you're going

5:29to be shipping logs from like your firewalls, from your various servers and operating systems into a SIM. I'm going to put up here or syslog or some type of log repository. Okay. Let's take a look at those options. Here we go. Here's some various options. So you could put an endpoint agent. So oftentimes you'll install an agent on an OS and it will do the shipping for you. It'll ship your logs. You just tell it which logs and of course the destination right there. Okay.

5:59And then it'll do that. Now syslog, which uses TCP and UDP 514, by default it uses UDP, but you can configure to use TCP. But what that does, and generally we do this with our firewalls and our network devices, we use syslog a lot. And what it does is you basically just point it to a destination and of course tell it a, what we call a logging level. And we're going to get into that. And then it will ship those logs to that destination. You can also use Windows Event Forwarding. You can use APIs or you could use file-based collectors. These are all options to get your logs to a log repository. Okay. Now let's talk about data log levels. Now each system out there is going to have some type of logging level, and then we're going to see this because we're going to go through and look at this. Don't worry. But it means, you know, how much to log.

6:53And I'm not talking about space necessarily. I'm saying how detailed do you want those logs? Do you want every single little thing that happens on the system log? Or do you want just normal things like your authentication, maybe a file access, maybe permission changes? Let's see. What else? New user accounts created, new permissions assigned, those kinds of things. You might want these, but you don't really care about the other little things. Okay. So you have to figure that out, and you'll need to configure it as such. Okay. Now when it comes to syslog, and a lot of systems will use this, they use 0 through 7, a numbering system. Now Windows doesn't do this for its Windows logs, but a lot of other systems do. Now the thing is, you'll need to decide that. So let's look at these logging levels here. Okay. Here we go. 0 is known as emergency, meaning the system is unusable really bad. But then all the way at the bottom, we have debug, which records every little thing.

8:06And I do mean every little thing. So here, your log storage is going to be huge. You're going to have a lot of data. Here, you're going to have a tiny amount of data. And then, of course, everywhere in between. So then you've got alert, and critical, error, warning, notice, and informational. And here are the definitions. So I would spend a couple minutes just looking at this. Okay. It's pretty important you understand how these numbers and words correlate to each other. Now the thing is, when you're looking at this, if you choose the option, let me change colors here, down here, you're going to have so much data, you're just going to be overwhelmed. So we don't normally do that. Now during troubleshooting or something, you'll enable that debugging. Okay. But not normally.

8:50Okay. We don't run like that every day. So it might be that we turn this level on, our warning. Because if you have a bunch of excess data, it can cause what we call false positives. And a false positive simply means it's not true. Okay. That's what it means. Not true. So it's not really a problem. It'll raise the flag, say, hey, we've got a problem over here. But you know what? There's really not a problem. And this can cause what we call alert fatigue. Get that curly around there.

9:21Okay. Alert fatigue. That's right. Okay. Okay. We're going to wrap this up now with a logging lifecycle. So basically we start by generating logs. Then we use agents and services to collect those logs. And then we're going to transport and ship them. They're going to be ingested at our log repo. They're going to be stored and retained based on our retention settings. If we're using a SIEM or SIEM, analyzing correlation along with alert and investigation. And then we're going to be archiving for our retention policies or disposal of the logs.

9:55So that's it, folks. That's logging in a nutshell. In the next nugget, we're going to jump into Windows and start taking a look at the Windows logging options. See you there shortly.

0:00All right. Welcome back folks. Now it's time to work on windows logging. So I've got a lab down here. So what you'll do is just go ahead and launch the lab if you would and give it probably about five, six minutes to fire up as it starts the virtual machine and such. Then you can follow along with me if you'd like to. Let's jump over here. So in the lab, we have a password. Let me move this out of the way. We have a password file over here. It's got your passwords in it. And basically this is the password. Now these should be automatically logged in, but if they time out and log out, there is your password right there. Okay. We've got two hosts in here. What? VMs really? Server Nug.

0:33This is a domain controller in windows. And we have an Ubuntu host as well. So we're starting off here in windows. So let me go ahead and maximize that. Yeah, that didn't work too well. Uh, so there we go. We'll just leave with this. Okay. So here we go. What are you doing? We're looking at logging. So where does windows send logs? Well, they're stored on here, but how do you access them? Well, use the event viewer. So event, give this a second. It's a little slow getting started. And there's our event viewer. So we'll just click on that and let this open up and we're gonna take a look at the logs that are available in here. Now I will tell you right now out of the box, windows does not log enough information. Okay. Uh, we need more info and we're going to see how to turn that on here momentarily.

1:20So it's adding the snap in. We're going to get our event viewer up so we can take a look at the logs in windows. There we go. Let's go ahead and maximize. This is slide this over a little bit. Okay. So here we have our logs, windows logs, and you've got application, security setup system and forwarded events, but that's not all. If you come under applications and service logs and give that a second, it's thinking there because there's a lot in there. Let me tell you folks.

1:50Okay. Uh, what I had the problem here. I don't want to let you know where I clicked on this and it just gave me an hour glass and it just sat there. So if it does that, go ahead and close it out and reopen event viewer. It just didn't start right or something. Uh, just to let you know. Okay. So under here you see we have active directory, web services, DFS replication, directory services, DNS server logs there under Microsoft. You've got all these and look, I mean, if we go under windows, look at this thing, it's just a huge list of all of these logs.

2:22So there's a ton in there. So what I want to show you though, is these up here, these are our primary logs, like your security and stuff. So let's take a look at one of these. I'm going to open up an event over here and let's talk about what we see in here. So you've got a security ID saying it's a system, a, an account name that's involved in this event. You have account domain, you have a log on ID here and a log on type. Okay. And you don't need to memorize these or anything, but over here, event ID, this is 4634.

2:53That's how windows identifies various types of events and event ID. And so you can see this log information in here. So there's a lot. You can come over here. You can clear the log, which attackers like to do. You can filter the log. You can create a custom searches like custom views over here. And this is the logs. Now, if you right click on this and go down to properties right here, you'll see right here, this is the maximum log size.

3:25Now what's it going to do when that maximum log size is reached. We talked about this and this is set to overwrite events as needed. All right. So that's an overwrite or do you want it to archive the log? Okay. Or do not overwrite events, clear logs manually. You have those options there and there's a clear log button. All right. So that's how you can set your log retention. You can also do it through policies like group policy and such. Okay. So that's where you go to view your logs. I'm going to minimize this because now what I'm do is I'm going to open up group policy. So go down here. I'm going to type group policy management, because I want to show you where you can turn on additional logging.

4:08So let's just go into the default domain policy here. Say, okay, we're going to edit this, right? Click on it. Once it loads up here, there we go and go to edit and it's going to bring up the settings. Okay, super. So let that populate. I'm going to move that down just a little bit. Okay. So let's take a look at this. If I go under policies and then I go down to windows settings and then down to security settings, and I need to slide this over so you can see what's happening here.

4:42A little more. There we go. What we have here. Then we go down to local policies right here. Let me expand that. You have audit policy. Now in our group policy here, Microsoft doesn't call it logging. They refer to it as auditing. So if you go to audit policy, it says auditing. So if you turn auditing on, it means it's going to log it. So right here we have a bunch of not defined. So you'll want to come through and define these. Now it does log when somebody logs on and logs off, it logs the authentication process, but there's some things it doesn't log.

5:19All right. So let me do this. Let's see here. These real quick. So you have audit account log on events. So if you double click on it, you can define it and say success and failure or success or failure. Just depends on what you want there. I'm going to cancel that. But there's audit log on events, object access, audit policy changes, all these things. So what I'm going to do real quick, I'm going to minimize both of these and I'm going to right click here and create a new folder. There we go. And I'm going to call it test. Excellent.

5:53Now I'm going to go in here and give it just a minute here. Okay. I'm going to create a new text file and there we go. Text document. And I'm going to say doc one. Why not? And I'm going to put some information in it. Just like that. Okay. File save and close. Now let's go back to our event viewer. I'm in security. I'm going to right click and I'm going to refresh. Okay.

6:26So what we have here is log on, log off, special log on, log off. It doesn't show anything where I was working with creating a file or accessing the file. Well, that's because you have to turn that type of logging on and there's a couple different places that you have to do it. So let's go back here to our settings. So what we're going to do is we're going to select object access, audit object access, and I want both success and failure. And I'm going to say, okay, now we see it has changed over here. Now you can do this for all of these different things to make sure that they meet your organization's needs. Now, if we scroll down a little further, there's also advanced audit policy configurations.

7:07Let's expand this. There we go. These are audit policies. So here's one for account log on. So if I select this, audit credential validation, Kerberos authentication service. So you can get deeper in there. There's directory service access, log on, log off activity, object access, policy changes, privileged use all these things. So this is where you come in and you really configure what you want windows to log. Okay. That's where this is done. Then you push it out via group policy.

7:39Now, if it's just a single host and it's on a domain, you can go to your system policies and you'll see these same thing in there or something very similar. So now that we've turned our auditing on for object access, which if I go back here to audit policy, you'll see we turn that on for our object access. Well, since this is using group policy, I'm going to close this. I need to open a command prompt here. There we go. Cause I need to refresh our policy. So what I'm gonna do is GP update force and that's just going to force the system to update the policy that I just changed. All right.

8:16And that way we'll be able to show you something here in just a minute with our object access. Okay. That took about 45 seconds or so. And there it goes. It's done. All right, super. So I'm gonna close that. Now I'm going to go back to my security logs. Let's see. Is that it? There we go. I'm going to open up the event viewer again. I'm going to do a quick refresh on security. Okay. Refresh. Okay. Ooh, here we go. Here's some object access stuff happening now. Okay.

8:49And if you look down here, this is the object name. So it tells you what's accessing it, who's accessing it. So what I'm gonna do is just minimize this for now. And here's what we're going to do. I'm going to, cause I actually have to go back to the test folder. So let me close that out. Uh, let's see. This is Active Directory User Computers. I don't need that. I'm going to right click on this and go to properties because you also have to turn object access on, uh, on your files individually, or well, your directories. Okay. And it'll flow down from there.

9:23So what you do is go to security, go to advanced. Then you go to auditing right there. And you're going to say, add select a principal. We're going to say everyone. There we go. Okay. And I'm going to say full control. So anytime they do any of this stuff with that file and all sub files, because by default, let's see. Yeah. Right here. It says only apply these audit settings to objects and or containers within this container. All right. And that's fine. There we go.

9:53Say, okay. And okay. Again. And okay. Again. Now auditing is turned on here. So if I go into test and I open doc and I close it and I open doc and I close it and I open doc and I add some data to it, just like that file save and close. We'll close that. Okay. Now let's go back to our logs. There we go. Let's do a quick refresh on security. There we go. And file system. Okay. Well, this is different. Let me do another refresh.

10:24It does take it just a couple seconds. Okay. There we go. Okay. So we see that. And what I'm looking at is right down here, looking to see what was accessed. So there we go. Uh, systems, more stuff like that. Oh, that's log off file system. Uh, there, Oh, here we go. Now that's windows explorer. It's me. Explore, explore. Uh, let's see. Let me do another refresh and now it's showing up right here.

10:56There we go. So this is the administrator. That's me, uh, accessing the desktop test directory. There it goes. And there's a test again, test again, test there's my doc one where I accessed it. Okay. So you'll see you can turn on lots of auditing features or logging features in windows. And if you want to enable object access and record when people log files, you not only have to turn it on in the background. So it logs it. You have to go to individual directories and it is inherited.

11:28So it does go down the tree, but you need to turn it on there as well. So that is windows logging in a nutshell. Next up, we're going to jump over to our Linux or Ubuntu host over here and take a look at logging in Linux. See you there shortly.

0:00All right, here we are on our Ubuntu host and I want to start off with a little disclaimer saying different Linux distributions work differently. Okay, and we're going to be focusing on Ubuntu here, which comes from Debian. It's a distribution from Debian. All right, so just as a little disclaimer. Okay, so within Ubuntu we have two different logging systems. The first one is systemd-journald. Okay, and it collects and stores journal events. We use the journalctl command. So let's bring that up here. Let's open our terminal here. So we access that journald logs from systemd via the journalctl command. Okay, and so if I hit enter, it's going to print out a bunch of information in this journalctl command. Let me actually maximize this. There we go. It's going to put out things related to kernel messages, startup processes, system services, apps, all those things. Okay, and you can kind of just scroll through here and see all this information. There's a lot of information here. Hit q to stop. Okay, and then we're going to clear the screen. So let's take a look at different commands. So journalctl is the base command. You can do dash r to list it in reverse. So you're going to see the most recent

1:21activity. All right, so things like this. Okay, I hit q again and clear. Get us back up to the top. There we go. Okay, I can also use, instead of minus r here, we can use dash n 20, and this is going to show you the last 20 log entries. All right, so this will give us the last 20. There we go. Okay, and we'll clear again. We could also use, let's go here, dash f, and what this does is it creates a stream. So I'll hit enter, and it's going to, as things happen on the system, it's just going to show the events in real time. So I'm going to go over here and click on these Firefox and watch what happens. If I click it, there you go. It's starting to do things now, right? We've got new logs. So this allows you to kind of view a live stream of the logs. All right, I'm going to hit control c. Go ahead and stop that. Clear it. We'll try to spell clear properly. Turn my caps lock off.

2:29There we go. Oh goodness. Okay, now you can also use grep, and if you're not familiar with grep, it's a Linux search term. So you could do journalctl pipe. You're passing that whole journal log over to search. So I could then search for something like Firefox. Okay, and this is only going to give me the log entries that include Firefox, and you'll see there's a lot of them. So let's stop that. All right, control c. There we go, and clear. So that is another way you can search through those. Okay, so that was, let me type this out, so you can see systemd-journald.

3:12Okay, so that's one logging system on Linux. Now another logging system on Linux is rsyslog, and rsyslog is used to send data into files, and most of them are located in the var log directory. So if I was to do an ls var log, you're going to see a bunch of log files in here. There's alternative support, auth log, boot log, bootstrap, btm, dmessage, cloud, dpick, kg. You know, there's a ton of them in here. syslog, ubuntu. There's a bunch of log files.

3:47Well, that's because it is actually shipping those logs via syslog into these files. All right, so let's say I wanted to search my auth log for failed logins. What I would do is start grep, and I would do dash i, and then put failed in quotes, and do var log auth dot log, and I can search that log for the word failed, and there we go. We see there are some entries there. So that's one way that you can look through those. Now, yes, this is why something like a sim is so helpful, right? It brings all our logs into one place, lets us easily view them.

4:28So let's take a look at the seven most common Linux log files. I'm going to bring this in so we can take a look at this. So here we have var log, and all these are var log. We see we've got your syslog. That's just general system and services activity. Auth log dealing with your authentication. Kern log dealing with kernels and hardware and firewall messages. Boot, it's events generated during the boot and service startup failures. Your audit log is detailed security events collected by the audit d process. History dot log, packages that are installed, that's your software that you install, upgrade, or remove through the apt package manager. And then ufw dealing with allowed or blocked events in the Ubuntu firewall.

5:14So these are just seven of the most common. I'll include this below the nugget here. Okay, so now that we know that, we've got log files in here. Okay, we use journalctl to search journal logs. We can search through these logs. Okay, we know where they're at. But let's now look at log settings within journald. We're going to go back to journald. So here's what we're going to do. I'm going to take a look at something real quick. I'm going to do lsetc systemd, just like that. And let's see what's in here. I want to see there is a journal conf right here. See this file. So let's take a look at that file. So here's what we're going to do. This is for our journal settings.

5:56So let's do this. Let's do a nano, which is a text editor. And let's do etsy systemd journal dot conf and enter. So here are the settings for our journal d. So if we scroll down in here, and this is really kind of a master file of the settings. You actually have to go through and create a folder and put settings in there. But I just want to show you the settings. That's really what we're getting into. I'm not teaching you how to configure all your Linux logs. But what we have here are various settings. They're all commented out, which means they're not being used. But let's just talk about a couple of these real quick. Like storage here, it's just set to auto. So it should then try to save the logs and be persistent through storage.

6:40You can use compression. There's lots of rate limiting information in here. Let's see what else is in here. Let's scroll on down here. Here we go. Max file retention. Max file second one month. So right here, it's going to save those logs for one month. Also looking for max level store. Max level store right here. So this is set to this is your zero through seven. This is debug. So that's seven. We definitely wouldn't want to debug that log every little thing. Then there's also a system max use in here. And let's see, where is that? That should be up here somewhere. System max use right here. System max use. And what this does, this is going to define how much space can be used for your journal delogging. Okay, so these are just settings. So you'll see in here, this is kind of what it looks like. So let's go ahead and close out of here. Yes. Yes. Close terminal. Okay. Let's open a new terminal and start fresh as we start looking at our syslog. Now this is different. So what we're gonna do is do ls etsy rsyslog.d

7:56Enter. And we have here 20 ufw.conf, 21 cloudinit.conf, and 50 default.conf. Now what these are is it goes in order and it looks at these. These are logging information. So here, this is the configuration for firewall logging. This is the cloud initiator logging. And this is your default settings in here. Okay, so let's take a look at those default settings. So we're going to do is etsy rsyslog.d slash 50. Tab that out. Default.conf. Enter. And here you go.

8:29So, oh, I counted that. I should have nano'd it. Let me hit up arrow and press A. And let's get rid of that. And let's go to nano. So we can see this a little better. Okay, here we go. So here we can see the auth and priv. Those are going to the var log auth. And then here we have additional going to var log system. Your kernel right here going to var log kern. Any mail going to var log mail. And you see these others like cron. If you just edit this, see I don't have, it's not writable. It's a read only at this point. But if you go and edit, you can then enable these things. And these are the settings here. And so these are where we can define where to log to. Now, if you wanted to create retention policies and stuff like that, that requires actually creating a retention policy file for each of the various log files.

9:26Okay. So you can do that as well. So this is kind of the basics of Linux logging. So we got to see some Windows, some Linux. And next up, we're going to take a look at syslog. So I'll see you there shortly.

0:00All right, welcome back folks. Now that we've got our Windows and our Linux logging out of the way, let's take a few minutes to look at syslogs. And that's because generally your network devices are going to be using syslog to ship the data most of the time. But Linux also uses syslog. Okay, so what syslog is, it is basically it's a standardized shipping method. Okay, there we go. Method or really it's a protocol. I should put that in. Protocol with a specific format. So it has a format. Okay, and that's what we're looking at here. So let's start here and go through one of these. So again, remember with syslog you have zero through seven. Zero meaning emergency. I'll change colors. Emergency, right, that's our zero. And let's go back to blue over here or let's go green or debug. That was our seven. Okay, so those are our numbers there. So let's look here at one of these log entries. These are coming from a Cisco ASA and 165. Let's see here. This 165, it's a priority field and it's calculated using the facility and severity.

1:13Okay, Bob, you're throwing new words at me. What are we talking about here? Well, severity is this number here. Okay, that's the severity. The facility, I'll put that out here. Facility, there we go. What that is, that's where the log came from. So where, get my E in there. It came from, and generally this is a type of service on the system. So it might be from the kernel. So it'd be like Kern or it might be from user or from mail or daemon of some sort. That's what we're talking about, the type of service it's coming from. All right, so it's the combination of those two that you get your priority over there. All right, so let me get rid of this here. All right, let's continue here. So then we have a timestamp. Hey, you got to have a timestamp, right? So we know exactly what time this happened. All right, so that's our timestamp. Then we have here edge ASA. That's the host name that it came from. So that's your host that it came from. All right, and then we see ASA-5-111010. Well, this means it was generated by a Cisco ASA. We've got our ASA there. The five, that's the severity of our logging, which is, if we were to look back at our

2:30chart, it's notice. There we go. That's that level. And then we have 111010. That is a Cisco message ID. Just like in Windows, we have an event ID, and in Linux, it has an event ID or code. Basically, that's what that is. All right, and then after that, well, that's the message. This over here in the brackets all the way down to here, that's just your message. That is what was sent. Okay, so we have a severity here. So we have a priority. We have a timestamp. Let me change colors here. Timestamp, a host, and then right here, this is key. This is going to tell us our logging level and the event ID there. All right, and then, of course, you have your message. Oh, and then down here, just a little notice about the facility and also our severity.

3:18Okay, so now that we see what this looks like, let's take a look at how to configure it on an ASA. Now, you don't need to know this, okay? This is not a Cisco course. I'm just giving you what it looks like, just so you have an idea. Don't try to memorize this. Don't worry about that. I just want to show you what it looks like, okay? Because we went through the Windows settings and the Linux settings. I want to show you a network device setting. So here you go. So you enable logging. Logging enabled. That makes sense. Logging timestamp. You're telling it to add a timestamp to your log entries. Then we're saying to save this to our little buffer, okay? And it's going to save at the five level, which was our notice. But not just that. It's everything below. So we have zero, one, two, three, four, five, six, and seven. So if you're saying log at this level, it includes all of the lower levels as well. So there you go. Logging buffer size, we're just saying how much space to give to the log. And then here we're telling it to wrap. So once it reaches a full buffer, to go ahead and save it to a different place in memory before overwriting, so that we keep a copy, okay? And then here you're just telling it how much to

4:34save. And the thing about Flash is, oftentimes in network devices, Flash is persistent. So it persists through a reboot and it's not lost. Okay. Then, well, what about shipping my logs off? Well, here you go. Logging trap notice. So we're saying basically a trap is when we send data. That is known as a trap. So basically once a certain event has occurred, go ahead and do this. So here what we're saying is send five and below to our logging host. And then down here, we're defining our logging host. This is the interface. This is the IP. And then we're saying use UDP 514, which is syslog. Okay. And we're just saying where to send it. So this is a glimpse as to what you would do to configure a device. Now there's GUIs and stuff as well. I just want to kind of give you a little insight, a little behind the scenes as to what it looks like. Now, the thing is you really do need to understand the syslog formatting. You want to know that so that you're able to identify maybe the host name or the event ID or the syslog value there. Pretty important. Okay. All right. So guess what, folks? That wraps up this skill. Well, yeah, it sure does. Except for, that's right, our

5:53validation challenge and it's coming up. I hope this has been informative for you and I'd like to thank you for viewing.