Design an Azure Logging Solution

0:00[AUDIO LOGO] Welcome to the AZ-305 certification exam, designing Azure solutions. This is a major part to a cloud architects suite of tools. What are we really talking about here before we actually dig into the stuff that this particular content is about? When we talk about design solutions and design certification exams, what we're really trying to figure out here is how well do you know the individual products and their features within the Azure ecosystem and how do they all fit together as puzzle pieces to build a full fledged end to end solution.

0:44That's what the certification exam is all about. It's not about implementation. It's not about where do you go to click this button, to make this thing happen. And it's not about what PowerShell commandlets do you actually use to implement this given solution. No, instead. It's in your particular situation, you should use this particular feature in conjunction with that particular feature to arrive at this full fledged solution. This keeps in mind things like high availability, disaster recovery, even scalability, how quickly can we spin up or tear down new resources.

1:23Beyond that, it also keeps in mind cost. Cost is a major factor when it comes to using cloud resources. If you're on Twitter at all, you no doubt see a lot of Cloud Billing memes flying all over the place because it's very easy to rack up a huge bill if you don't understand how billing and the features work within them. Now this particular set of videos is focused in, first, on designing a logging and monitoring solution. Now I know that out of the entire scope of all of the things that we're going to cover in this set of videos, logging, monitoring, even governance and access are not the most exciting things on the exam.

2:04But here's why I would argue that they may be the most important. It's because designing a solution has an end. It has a finite endpoint once it's designed. Implementing a solution has an end. It has a finite endpoint. It's once we've clicked the buttons and now the resource is spun up and working. But monitoring, logging, and governing a solution is something that is ongoing for a very long part, for a very long time. And giving our employees a resource that they can go to to monitor and watch and troubleshoot our resource after it's deployed, that's something that could actually have a huge impact on people's entire career.

2:47That could be their entire job is to keep the resources up and running once they've been deployed. The monitoring and logging solution is a major part of that. Now for the most part we focus in on Azure monitor in this set of videos. But there are other ways that we can get insights into what our applications are doing, depending on the resources that we actually deploy. So without further ado, what we're going to do in this set of videos is focus in on the monitoring and logging solutions, what are the options available to us, and why would we configure them in certain ways.

3:22Let's go.

0:00[AUDIO LOGO] The first thing we're going to do is talk about Azure monitor. And in particular, we're going to talk about where does Azure monitor get its data and what can we do with that data? Really Azure monitor collects two types of data. The first one is actual log records themselves. The second one is metrics, like streaming statistics or telemetry on the resources that we build.

0:30So what I'm going to do is I'm going to jump into the Azure portal and we'll explore Azure monitor to get a better feel about where this data comes from and what can we do about it. Let's go. So when it comes to monitoring your resources that you have deployed in Azure, you are no doubt going to begin your journey right here with Azure monitor. Shows up just as monitor from the Home page. If you don't see it on your Home page, simply search for monitor. And it looks like this.

1:01It's got a little gauge on it. And if you give it a click, it takes you in where you can see there are a whole lot of ways that you can leverage Azure monitor in your entire environment. Now insights are there entirely own separate thing. We've got its own dedicated video coming up to it. In fact, Insights are not something that are deployed by default, there are things that you actually have to go in and turn on and sometimes even configure on their own.

1:32Instead, what happens mostly by default with Azure monitor whenever you deploy your Azure resources, you have these two items right here that I'm circling, metrics and logs. Now here's what I want you to think about these. What are the two different things? What are the two differences between metrics? Metrics, if I were thinking just in a basic Windows viewer here, when I think of metrics, I am thinking of something like the actual.

2:03When you bring up the actual Task Manager, let me go ahead and bring in a task right here, Task Manager. And I go to the Performance tab. Let's bring it over here and I go to the Performance tab. This is what metrics is. Well, of course my computer is going to freeze as it comes on there, it goes right there. This is metrics. Metrics are near real time live streaming data of what's going on with the various components or guts within your environment. Whenever you deploy things, like anything that would consume, any type of compute resource in Azure, that's a virtual machine, that's a function app, that's even a logic app.

2:45Things where you actually monitor app services-- app service plans. That's basically extracting away the virtualization. Even containers have their own metrics that we need to keep up with. Whenever we actually start to think about the types of things that consume-- compute consume resources and we need to make sure they're behaving appropriately and understanding the impacts that they could have on our billing performance of the applications, we go to metrics because metrics are the live streaming items here.

3:19One of the cool things about metrics also is the fact that we can very quickly hook in metrics with alerting systems here. So if we actually need to be proactively monitoring and getting alerted to when our metrics fall outside of a range, this is when we can configure alerts. And Azure monitor serves as a landing zone for all of the metrics in our entire ecosystem. This is not just one resource that we can monitor from within metrics, we can monitor all of it.

3:52Similarly, we have logs. Now what is the difference between logs and metrics? Well, if I were to say if this were metrics where we're actually live streaming my CPU here, then logs would be equivalent to my Event Viewer. So if I actually bring up the Event Viewer right here and I look at things like my application logs in my Windows logs, it's going to launch up. Here you see all of these. Every single time something happened in my environment, it wrote a log to my Event Viewer. This is what logs are all about.

4:25And both of these, by default land in Azure monitor, so long as we configure them or turn them on. And for the most part, they are turned on by default, it's a really important thing to wrap your head around. The easiest example to use is spinning up a virtual machine. When it comes to a virtual machines metrics, the virtual machine deploys with something called a diagnostic extension. This is an extension that gets installed on our virtual machines.

4:58This allows the extension to grab metrics data, basically statistics like we just saw coming out of my Task Manager and stream it into Azure monitor. When it comes to logs, logs are slightly different. There is a monitor logs agent that gets installed on a virtual machine that streams the log data in right here into the actual logging environment. Now again, we did talk about something up above that I just glazed over.

5:29There is something called Insights that is its own separate agent that we could install. And it'll install, it will actually stream Insights into the mix. So if metrics are great for actually looking at live streaming telemetry about how my resources are performing, what are logs good for? Well, with logs, we're actually logging all of the events that happen on our resources. Within Azure, we even have the ability to create our own applications. It's actually the point of the cloud after all, isn't it?

6:01We write our own applications and let the cloud hosting. Well, with that, we can create our own logging solutions too and we can tell our applications log your data to the actual Azure monitor logs, if that's what we want to do. Right here, this is the big win. Once it's logged into the actual logs environment, we can write our own complex queries to filter out or grab log events as they happen or if we ever need to go back historically and find out specific information that happened in the past.

6:38And we can even leverage those specific queries to trigger alerts, that's right. Metrics and logs can both be used to trigger alerts. That way you can be proactively notified whenever issues come up in the system itself. And that's another major, major win that you may not have realized actually existed within the Azure offerings themselves. So if we were to break down what are the all resources that actually log to Azure monitor. You actually want to break them down at the actual Azure hierarchy level.

7:14For instance, we may have an Azure tenant. Tenant level data. Tenant changes themselves can be logged to Azure monitor. Within that we may have different kinds of subscriptions. That's right, subscription level data can be actually monitored and logged into the environment themselves. We also have-- as part of both of these, we actually have Azure AD, Active Directory information and changes can be logged into the actual Azure monitor information itself.

7:48We of course also have operating system information. This is both for IaaS and PaaS information, whenever we deploy things like virtual machines. We've already talked a little bit about that. We know that they're going to install different kinds of agents, whether we want to do metrics, logs, and insights. Of course, PaaS, this could be things like again Azure app services. There's still an operating system under the hood. We've just abstracted our need to manage it away.

8:20We can still get the metrics from that underlying operating system and stream them and log data into the hood as well. That's a big win. So then from there, we can actually look into the individual resources that we deployed underneath the operating system. If we were actually deploying an App Service, a logic app, a function app, something like that, we could log and monitor our metrics from our resources themselves. And then beyond that, we actually have the application itself that we can monitor via metrics and logs.

8:55Again, that's something that I just hinted on. When you have the ability to write your own code, you then have the ability to tell it to log to actual Azure resources themselves. And of course that comes with using just the plain old Azure SDK. So if you're developing an, say .NET, you can use the Nugget packages for Azure monitoring. And just tell it to log straight to the Azure monitor application. And the SDK just does it for you automatically. Of course, if you're using other frameworks or languages like Python or Go, those SDK exists as well too.

9:29All you have to do is just add that particular module or that library into your SDK or into your application. And there you go, you're monitoring and logging. So when it comes to actually using Azure monitor, we've got something called insights which we haven't really talked about, which gives you deeper understanding of what's going on. We have the ability to view our metrics and our logs right here. That's one way that we can visualize our data. Workbooks are another way that we can create interactive reports to actually visualize our data.

10:03And we'll take it even a step further and say that we can actually push this out into Power BI dashboards if that's what we want to do. It's worth pointing that out there. We also have the ability to use alerts to respond to events that happen within our metrics or our logs. And I'm also going to push this out a little bit and say, we can even configure auto scale events in the event that a certain threshold of metrics or logs have been triggered as well.

10:33So we can dig into this. We can actually analyze what's happening. We can visualize what's happening and we can automate even the response of what's happening within our Azure environment. So this is understanding at a very high level what Azure monitor is all about. It does a lot of really cool things for us. In the next few videos, we're going to dig deeper into what else we can do with the Azure monitoring resources. I hope this has been informative for you and I'd like to thank you for viewing.

0:00[AUDIO LOGO] Next up, we're going to focus in on Azure Log Analytics workspaces. This is a very commonly deployed resource. Whenever we want to collect things like logs from any of our resources, lots of times you see this being deployed with virtual machines, alongside virtual machines. Because as virtual machines, servers create logs, we need to put them someplace. That's what a Log Analytics workspace does.

0:31The very important concept to wrap your head around is that the Log Analytics workspace is a resource within Azure. And with that comes governance, like we do with other Azure resources. So let's unpack what that really means in this video and explore the Analytics workspaces a little bit more. Let's go. Now we're going to focus in on the Log Analytics workspace. Whenever we actually use Azure monitor to collect logs, a Log Analytics workspace is a resource that gets deployed with it.

1:05And that is what we're currently using. Now what you need to think about a Log Analytics workspace really is it's basically Azure table storage. We're taking resources that can generate logs and streaming those logs into storage that we can then query after the fact. Now the fact that this is a resource, very important thing means probably the biggest aspect of the Log Analytics workspace is we can govern access to it.

1:38There's really two aspects to the whole thing that make a Log Analytics workspace special. And that is the fact that we have first of all, a boundary of who can access it as well as a geographic location for the log data itself. We're not restricted to just having one Log Analytics workspace. We're going to have multiple Log Analytics workspaces. So that way, if we really needed separation of logs where they're stored for whatever reason, whether it's latency, whether it's internal policy, whether it's access controls, whatever the case may be, we can actually deploy the resources that do exactly that.

2:20But the big one really like I just said, it all comes down to governing access to this. We may have our global admins. They need access to all of the data that's contained within Log Analytics. But then, we may have a group of virtual machine admins. And we have virtual machines that we spin up within Azure and they stream their telemetry, their log data straight into the Log Analytics workspace.

2:52Outside of that, we may have Azure app services admins and they do what you think the name is. They administer the App Service resources that get deployed within Azure. So I'll deploy-- I'm just going to put AS right here for App Services. And they generate their own logs and stream data in to the actual environment itself. The cool thing about this is even though this one Log Analytics workspace contains my virtual machine logs and my App Service logs, I can now say, OK.

3:30App Service admins are only allowed to access the logs that were generated by the App Service resources themselves. And virtual machine admins are only allowed to access the logs by the virtual machines themselves. They call this the resource context. And that's why it's called resource context logging or resource context controls. The cool thing about it is now we can actually govern who has access to which logs within a Log Analytics workspace.

4:08Now when it comes to actual deployment of the Log Analytics workspace, like I said, this is really table storage which really means we're really just looking at a storage account under the hood that we're working with. We can deploy a Log Analytics workspace in conjunction with a premium blob. This is basically Azure data lake Gen 2. If we have very large amounts of data that need to be accessed and processed very quickly, premium blob storage will get the job done.

4:39Otherwise, we could deploy a generic storage account, which gives us access to hot, cold, or archive storage tiers. That way, depending on the types of logs that we're storing, we could determine if these need to be hot logs, cold logs, or even stored in archive storage. Beyond that, we can set immutability settings. Basically meaning if we have legal holds in place, we can put legal holds. I got to fix my pin there, legal holds on our archive storage as well as set time based retention policies for our Log Analytics workspaces too.

5:18So a lot of the features that come with storage accounts are actually applicable here to Log Analytics workspaces. Now like I said, data that gets stored in a Log Analytics workspace is placed into tables. In fact, if you want to see the tables under the Settings section, now in the Preview we can actually see the tables that could generate it. Now as we add more resources into the Log Analytics workspace that are generating their own logs, they're going to create their own tables. You can see here though these are Azure tables that are stored here, aren't they?

5:52So we can see things like VM connections, VM processes, VM compute. And you would think about these just like Windows event logs. Windows event viewers that would be storing compute information, connection information, process information. In the actual Event Viewer, they would be stored here in the actual table storage. You can also see under the workspace data sources. If I click on Virtual Machines here, well, this is where those virtual machines would be listed If we actually ended up having them. But we don't have them right now because this is just an empty landing zone as it stands.

6:25So what are the limitations to these tables or the storage accounts? Well, there's not a lot of limitations to it. By default when we deploy these, the data is stored on a cluster. If we ingest more than, greater than 4 terabytes of data, here's the big kicker right here, per day then we need to deploy our own cluster to actually host our own data. Otherwise, Azure itself is going to be able to physically store this data in the dedicated workspace on its own.

6:59So when it comes to an actual design for deployment of a workspace, what are the big considerations to keep in mind? Well, let's say I deploy my virtual machines in the South Central US data center. Not a big deal, just the standard deployment there. If I deployed my Log Analytics workspace in say the UK, what could be the problem using here? Well, of course, we've got a latency issue that could take place just by default.

7:29Even though latency between Microsoft data centers is very quickly. But even the bigger issue here is we could incur outbound data charges. That's a big issue here. Ideally, we want our Log Analytics workspaces as close to the resources that they're monitoring as humanly possible. That way we have the lowest latency as well as the outbound data charges are suppressed that way. Some of the issues that could come up is if we're actually deploying applications within governments or geographic regions that actually have data sovereignty or compliance issues, that would dictate exactly where log data needed to be stored.

8:14That could be a challenge for you that you may want to keep in mind. But the whole idea is to avoid these outbound data transfer charges. So that way we keep the workspaces in the same region that the Azure resources that are being managed are. That's the big thing that we want to keep in mind. So when it comes to that, when it comes to the actual resource deployment, we want to keep it close to the resource itself. But then it comes down to-- what are the actual permissions of who can access this?

8:44At first we have a workspace context. This means that for each individual user or groups of users, they would get their own workspace. So in the previous example, I talked about my VM admins. Would it make sense for me to actually deploy a workspace that's just dedicated to virtual machines? Maybe it depends on your particular environment and how your internal policies in governance are set up. It's a very clean way to do it.

9:14All virtual machines go to the virtual machine workspace. All Azure App Services go to the App Service workspace. Kubernetes clusters go to the Kubernetes workspace. Makes sense. But then we also talked about the possibility of there being a resource context, meaning we can have groups of different resources all log to the same Log Analytics workspace, again maybe that's the cost, maybe that's to be within our compliance of our own internal policy, our government policy.

9:46And then we set up permissions or access controls, basically our back-access controls, for each of the groups to have permissions only to the data that they actually need to have access to. So this is getting to know the power of the Log Analytics workspace. This is a space where we can actually dig into the individual logs that have been logged by each of our different resources. Virtual machines, storage account, system center, applications themselves, even automation can all log here within the workspace.

10:20We can put this in any region that we really desire. And we can govern access to the individual logs based on the generating resource, that's the resource context. I hope this has been informative for you and I'd like to thank you for viewing.

0:00[AUDIO LOGO] As your Insights is a catch all term, that's not a one resource that you deploy. Really what happens is when you deploy resources or even applications straight to Azure themselves, you have the ability to collect additional information and telemetry from the resources or applications that you just deployed. Insights grab that telemetry and put it somewhere, streams it somewhere so that you could unpack it and digest it a little bit more.

0:37Now like I said, it's not just Azure Insights, it can be things like virtual machine Insights or Application Insights or even something like function app Insights. That's what we're going to take a look at. This video is what are the different Insights that you can deploy within Azure and when might you use them. Let's go. Now honestly we're going to start shifting to the more fun things to talk about when it comes to our monitoring solution. I'm back here on Azure Monitor.

1:08And you can see that I've gone ahead and I've jumped into the Workbooks. What are workbooks? Workbooks are predefined queries that come off the shelf. Now it's worth pointing out, you can custom create your own workbooks if that's what you want to do. But for people who are like, hey, I need a quick solution that can leverage my logs and my metrics and just give me a quick visual as to how my infrastructure is performing, workbooks are a great win for you.

1:41Now behind the scenes, I've gone ahead and I've deployed a very basic virtual machine. Let me jump over here and show you real quick, it's called monvm for monitor virtual machine. And then I jump down here into the monitoring section. And under Insights, I went ahead and I enabled Insights. When you first click on this, it's disabled by default, so I enabled it. Then under logs, I did the exact same thing. I enabled the logs as well. So I just want to click the Enable Click to confirm and let those get started.

2:14It takes 5 to 10 minutes for the agents to be installed in the virtual machine and for data to start streaming in. But with that being done, now I can start to leverage some of my other more important things like actually looking into the Insights as well as the workbooks themselves. So let's take a careful look at what workbooks can bring to the table. Of course, we have the Quickstart which is just an empty workbook where we can actually run queries and visualize the queries. But if I scroll down here under the Virtual Machine section, there are four workbooks that we can very quickly click on by default. We can look at things like Key Metrics.

2:54Then we've also got a performance workbook which tells you the basics, CPU, memory, disk, and network. We've got the performance counters which actually leverages the metrics collection itself. And then we even have an availability. You could almost think about these like dashboard. Let's click on the Performance one here for a second. And if we were collecting a lot of data in here, you would see, let's go ahead and change the subscription to pay as we go. And we'll choose the workspace that I have. This one is my EUS workspace.

3:25And then click off that. And it loads my man VM up right here on the screen. So as you can see, when I enabled Insights and logs, of course, it immediately began logging data to my Log Analytics workspace. Well, with Workbooks, we can now use this pre-built item to query the data that's in my Log Analytics workspace and generate a quick report for me to actually see. We even have a little visual here that increments over time.

3:55You see I've got two little bars here at the very end because this resource is relatively new. So we can see some really interesting information right here within my virtual machine. Now this counter that we're looking at is my available megabytes here in RAM. But if I wanted to look at things like what is my logical disk, my free space, my reads and writes, my network utilization, my CPU heartbeat, I can click this dropdown right here. And you can see it immediately changes this like so. We can also see the aggregators that are used here.

4:28So if I want to look at this basically as a statistical distribution, I can choose that as well as we can do things like what is the average as well or the distribution that we're looking for in the trend. We also see this top 100 machines by default. If I just wanted to pick on the top 10, we would actually be able to drill down into this in chart form. That's the beauty of the top 10 machines. So when we have top 100 machines, it's going to be in list form, very tabular. When we look at the top 10 machines, we would be looking at an individual line chart here but it would have 10 lines on it, each color representing one of the 10 machines that we're looking at.

5:07So Workbooks, great win when it comes to using something like that. Availability, same drill here. This is going to be a really important one that you're going to want to look at. You're going to want to look at what is the uptime and the available hours and the total hours that it would be available for. Now in my case, the data is very skewed at the very beginning. Says this machine has been available for 72 hours, or has been online for 72 hours but only available for one. That's not true. I just spun this machine up about 30 minutes ago. So the data is a little wonky at first and it's going to need some time to find its feet.

5:41But there we go. Why does it say 72 hours? Because the time range here is set to the last 72 days. So since this machine has only been up for one hour of the last three days, that's why it throws off the metrics right there. Nonetheless, you can see, wow, that would be a very useful item. Now that's just for virtual machines. As you can see here, we have some other things that we can monitor within workbooks. Synapse. Well, what is Synapse? Synapse is a SQL server. Effectively, that's a very, very generous generalization for all of the things that Synapse can do.

6:18But for the most part, you, the infrastructure minister, leveraged Synapse apps to serve as a OLAP heavy SQL Data Warehouse. So I'm going to put DW right there. It does, as you can see right there, also allow you to leverage on Apache Spark cluster for processing data and querying data if that's what you want to do. And in this particular case, it comes with a couple of workbooks for actually monitoring your spark cluster if that's what you want to do.

6:48But the primary use case for Synapse, I would argue is to actually leverage the OLAP SQL Data Warehouse. Now we also have the ability to look at container clusters. We can actually dig into our hosted applications themselves. This is where we write our own code and stream it into the Log Analytics workspace. So from there, we can actually monitor our own data. We can even look into Log Analytics workspaces metrics here. We can actually look at our own workspace usage reports as well as our Log Analytics health, the agent health status.

7:24Now of course, like I said, we can create our own workbooks if that's what we want to do or we can even borrow templates from other users who have created their own templates in the entire environment. Really good stuff to look into, you see things like Azure Cache, Key Vault, storage accounts, virtual desktops. There's a lot of other templates that you can build on top of. And I'd really encourage you to dig into that when it comes to working with the Workbooks. But I did say in addition to the workbooks, we have another monitoring tool.

7:58If I jump back to the Overview, we have this thing here called Insights. And Insights are honestly, one of my favorite tools that we have. Insights combine metrics with logs in order to actually dig deeper and ask bigger questions about the architecture of your applications. How is your application, your virtual machines, your infrastructure, your resources performing and how do they communicate with each other? How's the communication between resources interacting and working or having issues here.

8:35With Insights, we can monitor applications, containers, network Insights, and virtual machine Insights. But we can also look into resource groups. We can actually get Insights into resource groups themselves. We can look into Azure Cache. If you've worked with Redis cache before, it's a very similar thing. We can work with Cosmos DB, a globally distributed document database, very similar to MongoDB.

9:05We have a Key Vault Insight where we can actually get reports on what our Key Vault requests have looked like. Who are they coming from? Who are we giving our credentials or our secrets out to in the first place? What are the performance issues that we've had, any latency between an application connecting to the Key Vault and getting the response back. Then we even have Storage Insights. Again, this is a unified report where we look at the performance of our underlying storage account.

9:38What is the capacity of our storage account? How available has our storage been and how has it been interacting with other Azure resources who are trying to access that particular storage account? As your app insights or Azure insights really hit home for me, if you actually click, let me show you how I do that. I did that with-- I'll say, if you go to the Insights, you can click View All Insights here. And you'll actually see basically a list of all of the things that I just listed off here in addition to things like hyperconverged infrastructure, SQL databases, Service Bus, and so on.

10:10Application Insights are really the first time where it really drilled home for me, how powerful this particular resource is. Because with applications insights again, it was just an SDK that you dropped into your application and turned on, and that was effectively it. From there, the SDK or really the Application Insights package itself starts monitoring all of the requests that actually take place on the application. So if I had a website and people start off with my Home page, let's call it home.html.

10:47And then from there they navigate to their Cart page. And then from there they navigate to Checkout page. And then from there they navigate back to a specific product page. Maybe they clicked on it because they got there from an ad that popped up in the sidebar or something like that. As your app Insights keeps track of all of this navigation information within the application itself, it also keeps track of things like what are the geographies that my end users are coming from.

11:21Am I getting most of my data or most of my requests coming from the UK, the United States, Australia, wherever they may be? It keeps track of not only the standard metrics that we keep track of as well as the logs themselves, but it then digs deeper into finding things like geolocation and the paths that people are taking through the application. And it generates this in a live dashboard for you to actually see what's happening in your application. It's a fascinating thing to check out.

11:54Now like I said, I already deployed my virtual machine Insights. So if I go to Virtual Machines, I can then go to Analyze data. And from here, I can choose my subscription, my resource, and then pick the virtual machines that I want to analyze. So here we go. I'm looking at things like the performance of my virtual machines via workbooks. But this is all done through my insights portal. So now, I can look at my CPU utilization for my virtual machine over a given trend of time based on the 95th granularity of the actual utilization.

12:34Same thing with memory utilization on this virtual machine, byte set and so on. And if I had multiple virtual machines, they would all be listed right here, just in different colors of line so that we can see how virtual machines are performing when compared to each other. Now, the map is what I was talking about where we're looking at how processes within the virtual machine connect to each other, as well as geographies where clients are connecting into my virtual machine in the first place. Now, this map feature is only available on specific operating systems.

13:09It's not available on the Windows Server operating system that I specifically picked for this virtual machine. So just know that is a thing. If I click on it, you'll see. It'll fuss at me and say, I don't have any virtual machines that are available for this. So if you wanted to leverage this virtual or this map feature, make sure you're selecting a compatible image to get it going. So this is understanding Workbooks as well as Insights to dig deeper and visualize what's really happening within the resources that you deploy in Azure.

13:40I hope this has been informative for you and I'd like to thank you for viewing.

0:00[AUDIO LOGO] The Azure Data Explorer does exactly what the name indicates it would do. It explores data. This offering has really transformed a lot, even in recent history. I know I started working with Azure five or six years ago at this point. And at the time, Azure Data Explorer was just a really great way to explore the contents of your storage account. Now it really gives us a robust way to unpack all sorts of data that could exist within our entire environment regardless of where that data comes from.

0:37Great use case here to use this for monitoring our resources, whether by metrics or our logs yet again. So in this video, we're going to take a look at how Azure Data Explorer could be part of the design for your monitoring and logging solutions. Let's go. So Azure Data Explorer-- if I actually bring it onto the screen here, anybody with an Azure tenant an Azure Active Directory tenant can get to it. It's dataexplorer.azure.com. It'll ask you to sign in with your credentials, and this is where you'll land.

1:12Now, again, Data Explorer has really evolved quite a lot. I actually remember using something called Storage Explorer and thinking like, oh, well, this is a great starting point, as Storage Explorer does what you think it's going to do. It's going to explore your data. But with Data Explorer, it takes accessing the data and ratchets it up all the way. With Azure Data Explorer, now what we're doing is we are analyzing big data in real-time.

1:45Where does this come into play when it comes to monitoring? Well, first of all, if we have a lot of resources in Azure that just a lot of data very, very quickly, that's a major point that we want to keep up with. If we have a lot of logs, we have a lot of insights. If we have a lot of metrics that we want to keep up with, that's a start. This oftentimes goes hand in hand with monitoring data that comes in from Azure Active Directory or a SIEM solution like Azure Sentinel.

2:21Sentinel, its entire point, its entire purpose is to grab all of this data that comes in from all of our resources, both in the cloud and on-premises, and process it very quickly to look for security anomalies. That's the idea here is to track out those anomalies. That's A-N-O-M-A-L-Y. There we go. Data Explorer really shines when it comes to grabbing all sorts of data, whether structured, semi-structured, or unstructured data crunching it very quickly, and then spitting it through some machine learning or AI to help you find not only anomalies but also prediction forecasts of where it's going to go.

3:15When you want to use Data Explorer, these would be your use cases. The first one is you need interactivity. It needs to be interactive analytics. This is going to be a major part where we're going to be needing to interact with to find aggregations or correlations in our data or detect anomalies. That's what interactive analytics are all about. We also have a lot of Vs. The velocity of our data, the growth rate of our data, or the rate at which the data moves and the rate at which we need to monitor it is very, very critical and sensitive.

3:55If we need to monitor data in real-time, that is a very big reason why we would need to use Data Explorer. But also, the sheer volume of data that we have is tremendous. Then lastly, the variety or the different kinds of data sources, as well as whether or not the data is structured in the first place, is a big deal. But then maybe we haven't even had the time to ingest and model the data yet, basically meaning we haven't had our ETL solutions run to pull the data from their sources.

4:30Instead, we want to analyze raw data as it lives row-level raw data on the fly. That's the big issue here. This is another great use case for Data Explorer. Multiple querying happening at the same time. We call this querying with concurrency. So what we're saying here is maybe we have multiple users analyzing the same underlying data at the same time. It's a tremendous amount of data and just processing one query alone takes a tremendous amount of compute resources.

5:07This is where Data Explorer would really shine. Now it even gets more complicated than that. And that's why I am going to bring some Microsoft documentation onto the screen because they have a great flow diagram as to when you would actually want to use the Azure Data Explorer. It says Start Here at the very top. And you go through a basic flow where you answer a bunch of yes and no questions. And ultimately, what these questions are trying to do are say, is there a better, more targeted fit for what you're trying to accomplish that you should use a different type of tool like a SQL database or a Cosmos Database?

5:46Should you use Spark or Azure Batch? Should you use a Synapse Data Warehouse or a SQL pool? Otherwise, it all kind of takes you down here to Azure Data Explorer is a good operation here. There's is a good option. So where as your Data Explorer really shines at the end of the day, is does provide you better flexibility when it comes to working with tremendous amounts of data. Also enabling near real-time analysis, including dashboards.

6:19It's a big, big deal. With these dashboards, we can do things like pattern detection or time series analysis. And when you think time series analysis, you need to be thinking logs because every time your log runs, it's logging what is the date and time that this exact log was issued, and that's where the time series analysis really comes into play. This does come with role-based access control so we can limit who can query what types of data.

6:49We then have things like integration directly into machine learning, including training machine learning models, which is a major part of using machine learning. You have to train the model to learn what's normal in the first place. And that way, you can then detect anomalies. So that means that this does integrate with Azure's machine learning workspace, the machine learning tools, as well as Databricks, which is a very common data crunching solution.

7:23Very, very useful in the olden days of Azure. But now, recently, I've started using Azure ML, and I've really enjoyed my time working with that as well, for what it's worth. So you can also see where this really comes in handy when you're using the same solution like Sentinel, that's S-I-E-M for security monitoring. So when you're using these security solutions like Sentinel in conjunction with big data streaming and machine learning, it can detect security anomalies even quicker, logging them and then visualizing them in dashboards.

7:58This is, again, for really large data sets, but it is an incredibly powerful tool and one that you should design for if you're ingesting just a tremendous amount of data. So if you're in a very large enterprise and you're collecting logs from all of your network devices, whether they're switches or routers or firewalls, or intrusion detection and prevention devices. Then you've got all of your security servers, then the virtual machines, the databases, then client desktop machines.

8:31Then IoT devices. All of this data is going to be coming in and logged through things like Kafka or Event Hub itself and leveraging applications like Logstash to actually be doing the collection of the logs and the ingestion of the logs itself. They make their way into Azure, where they eventually live in Azure data lake before they're ingested or transformed and moved even further down if they need to.

9:02Azure Data Explorer is a piece of this puzzle-- I'm going to put Azure DE for Azure Data Explorer here-- is a piece of this puzzle for grabbing the data at various points, analyzing it, and detecting eye issues in near real-time. So understand if you have a tremendous amount of data for a large enterprise coming in, Azure Data Explorer should be part of your monitoring solution. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[AUDIO LOGO] So there you have it, designing a monitoring and logging solution coming to you via video format. I really like this content because after you've deployed an application before and then had to troubleshoot it, you may be thinking to yourself, oh, man, I wish I had spent a little bit more time tuning up that monitoring and logging solution. That way I could have maybe gotten to the crux of my actual resources or my problems a little bit sooner.

0:35Beyond that, it does help you get insights into how your application is currently performing. You can get ahead of things that way by saying, you know what, this application probably needs to be tuned a little bit. Maybe we need to scale up my resources a little bit, or maybe we need to scale them down because we overprovision the resources in the first place. This is a really important factor yet again, because like I said at the very beginning of this set of videos, you can spend a finite amount of time designing and implementing a solution, but that solution could be up and running significantly longer than the design and implementation took.

1:12And that's where the monitoring and logging comes into play. That's where we keep the resources going. And really that's where you spend the bulk of your time, that's where our staff will spend the bulk of their time. And that's why I can't stress this enough, even though it's easy for your eyes to gloss over when it comes to logging and monitoring, this is actually a really, really important topic, arguably one of the most important topics and one of the most cost effective topics, one of the those cost effective wins that you can get.

1:42So thank you for joining me in this content on monitoring and logging as we progress through the AC 305. I hope this has been informative for you and I'd like to thank you for viewing.