Intro to Certified Kubernetes Security (CKS)

0:00[AUDIO LOGO] Hi, guys, my name is Trevor Sullivan. And welcome back to another CBT Nugget skill. In this skill, we're going to be exploring the Certified Kubernetes Security certification. And we're going to go through, very quickly, some of the different topics that you're going to need to be familiar with and probably be expert level at with in order to pass the Certified Kubernetes security exam.

0:36If you are brand new to Kubernetes, I would strongly encourage you to start out with the other certifications that are available for Kubernetes. So there is the Certified Kubernetes Application Developer or CKAD. And then, there's also the Certified Kubernetes Administrator as well, the CKA. And both of those are much more easy to pass. There's a lot more fundamental concepts in Kubernetes, just things about setting up clusters, deploying application workloads, setting up role-based access control, and things like that are pretty straightforward.

1:13Once you start to dive into purely the security side of Kubernetes, you're going to be looking at a lot more back-end configuration stuff, that you don't typically see with the Certified Kubernetes Administrator or the CKAD application developer exams. So the Linux foundation here and, more specifically, the Cloud Native Computing Foundation that is working actively on the Kubernetes project and providing oversight for Kubernetes and a variety of the other Kubernetes tools that are out there in the open source ecosystem, they've got this guide that talks about some of the different things that you'll want to be familiar with.

1:52Some of the different topics that are going to help to prepare you to take and pass the Certified Kubernetes Security, or CKS, exam here. Now, there's a few high level domains and competencies here that you're going to have kind of different biases on. So some topics are going to be heavier hitting on the CKS exam. And other topics are going to be a little bit lighter weights, but are still going to be important things to understand. One of the nice things about starting off with the CKA, the administrator or the CKAD, the application developer exams for Kubernetes is that those are going to kind of pave the way for you to kind of build up towards Certified Kubernetes security, rather than just if you're brand new to Kubernetes, biting off more than you can chew all at once.

2:42And then, just going straight for the security topic. It is just going to be a lot more gradual to start off with CKA and CKAD. And then, once you've got that foundational knowledge of how to deploy workloads, and do some basic configuration in Kubernetes, and things like that, then you're really going to get into the CKS. Now, I've tried with the CKA and the CKAD certification courses to go a little above and beyond. So you're not necessarily going to have to watch the entire course just to pass the exam.

3:13But I've actually added a lot of other value added tools that you'll probably come across in the open-source ecosystem surrounding Kubernetes. And I've actually done the same thing with CKS as well. So there are going to be some tools within this CKS training course, that probably won't appear on the certification exam. But by gaining familiarity with those tools and just getting comfortable. The more time that you spend with Kubernetes, the more different types of tools that you're using with Kubernetes, it's going to better prepare you for actually passing these exams once you get to the point of taking it.

3:51So we're going to be talking about these different domains and competencies. And I've actually already recorded a lot of the training for CKS. I'm not quite done with it as of this recording. But I kind of saved this for the end, so that we could kind of build out the training content and see where things kind of evolved to. And then, I came back and recorded this introductory one. So I'm actually going to be building off of some of the experience that I've gained just by building out the training for the CKS training course here at CBT Nuggets thus far.

4:22So at a high level here, the domains and competencies, you're more than welcome to check out this content over on the Linux Foundation site or on the Cloud Native Computing Foundation site. They have a lot of the same information on both of them. But under domains and competencies for CKS, we're going to have some kind of fundamental things that are going to overlap a little bit with the CKA course, where you've got cluster setup. It's going to be really important for you, whether you're an administrator, an application developer, a DevOps engineer, a security engineer, it doesn't really matter what role you have.

4:54If you're going to be actively working with Kubernetes, it's really going to pay off for you to get experience just setting up a Kubernetes cluster from scratch. And so when we talk about cluster setup, we're talking about everything from provisioning a Kubernetes cluster to just high-level familiarity with the different Kubernetes distributions that are available out there. There's lots of different Cloud-managed services from the likes of Digital Ocean, for example. They provide a managed Kubernetes service.

5:25So you can literally just come in here and say, create me a Kubernetes cluster. Or you can go into the LiNode platform and say, create me a Kubernetes cluster. You can plug-in a couple of lightweight details, give it a name, give it a few worker nodes to join to the cluster. And, bam, you are good to go. Also, the major Cloud vendors like Amazon Web Services and Microsoft Azure have their own managed Kubernetes distributions as well. Also, Google Kubernetes Engine. So all the major Cloud vendors have their own distributions.

5:55However, the reason that I tend to like more of the kind of smaller vendors, more of the niche vendors like LiNode, and Digital Ocean, and Vulture even, they are another vendor out there that has a managed Kubernetes solution. The reason that I like these smaller Cloud vendors is because they really specialize a lot more than the major, the big three Cloud vendors, Azure, AWS, and Google Cloud. And so they're able to provide easier to use APIs. They're able to provide nice CLI tools that are just really simple to use.

6:27They also provide just a nicer web user interface. So if you're just going in and using the web interface to manage your account with LiNode or Digital Ocean, it's just a lot simpler of an interface. And there's a lot fewer moving parts. Whereas, if you're using one of the major Cloud vendors to provision and manage your Kubernetes cluster, yes, on the positive side, you do have a lot more customizability over the network configuration and the node configurations. You have a lot more selections to choose from in terms of what virtual machine types you want to join to your cluster.

7:00But I personally, for my particular use-cases of just learning about Kubernetes, getting familiar with it, and being able to rapidly provision and de-provision clusters, I personally get a lot more value out of these smaller Cloud vendors, because they really do focus on the user experience. Their web consoles are just really nice and pleasant to use. And another benefit here is that these smaller niche Cloud vendors actually tend to keep up-to-date with managed Kubernetes versions a lot better than the major Cloud vendors.

7:34When you get into the large Cloud vendors like Microsoft, and Amazon, and Google, they tend to lag behind on Kubernetes versions a bit more because the fact that they're larger companies means that there's a lot more rigorous testing that they have to do. There's a lot more edge cases that they have to plan for during upgrades. They have to just consider a lot more variables than what these smaller Cloud vendors have to worry about. And so because they do focus a little bit more tightly on their niche area within the Cloud industry, they are able to move much more rapidly.

8:08And they also have very competitive pricing plans as well. And they're very straightforward pricing plans. That's actually another benefit of using these smaller Cloud vendors to provision clusters is that their pricing is just more straightforward. A lot of people get confused when you talk about pricing of AWS. Do you have to pay for this? Do you have to pay for this? Do you have to pay for this? And with these smaller Cloud vendors, their pricing is really transparent. It's simple, straightforward. And so if you're trying to do cost planning for your organization or your business unit, then these smaller Cloud vendors do provide a lot of value in that regard.

8:42Now, aside from the Cloud vendors that provide managed services to deploy a Kubernetes cluster, there's also tons and tons of different Kubernetes distributions available. My personal favorite is K3S from Rancher Labs. And there's a bunch of others as well. There's Minikube. So this is kind of just a mechanism that you can use to rapidly spin up a cluster. There's also Kubernetes in Docker or the Kind software. This is yet another distribution that allows you to run Kubernetes in a Docker container.

9:13And then, you also have things like MicroK8S from, I believe, it's canonical. This is just another distribution of Kubernetes that has some pre-installed components inside of it. And there's a whole bunch of other distributions of Kubernetes. There's K3D, K0S. And I can't think of all the others off the top of my head right here. But as you can see, there's just a lot of different distributions of Kubernetes that allow you to either use a Cloud vendor for the managed side of things. Or if you just want to spin it up at home, in your home lab, or just in on a virtual machine that's running in some Cloud vendor, it doesn't really matter which Cloud vendor, if you don't want to use the managed offerings, there's lots of self-managed solutions that you can use to spin up clusters as well.

9:54So knowing how to set up a cluster and do some basic configuration on it is going to be a really important concept, regardless of which certification you're getting. But as a security engineer, it really helps to know that kind of thing. Because then, you can start to learn the different knobs and levers that you have control over in order to customize your Kubernetes deployments. I hope this has been informative for you. And I'd like to thank you for viewing.

0:00[AUDIO LOGO] Hi, guys and welcome back. So in the last video, we talked a little bit about the different certifications that are available from the Cloud Native Computing Foundation. And I wanted to just continue talking about some of the different domains and competencies here that you're going to need to know for the Certified Kubernetes security exam. And, again, keep in mind that I've recorded already a lot of the training content for this course.

0:35And so this is just, I'm just pulling from my experience that I've got while I've been preparing the content for this course and just trying to give you a high level overview over some of the different topics that you're going to want to know about. So let's move on from cluster setup and focus a little bit more on Cluster Hardening and System Hardening. So for Cluster Hardening here this is going to be covering topics like restricting access to the Kubernetes API. As you probably know working with the API server is a really important concept because pretty much everything that you do to manage Kubernetes from a resource management perspective is going to be done through the Kubernetes API server.

1:14Granted there is a lot of back end configuration that you can do as a security engineer on the actual Kubernetes components themselves. So you can customize different command line arguments on things like the scheduler and the controller manager and the API server itself. And we actually do talk about some of those things like how to set up audit logging, how to work with Kubernetes events to monitor the state of the cluster and things like that. The audit log is actually a really important concept because it shows you what is going on behind the scenes.

1:46And it allows you as a security engineer the maximum observability over what different users of your clusters are actually doing inside of those clusters. So you can get access to basically the raw API calls that are being made to your Kubernetes API server so that you can detect any potentially malicious activity and to just keep tabs on what people are doing. And then this is really important for incident response. So if you ever run into a security incident and you need to do some investigative work to figure out how a malicious actor gained access to an application or gained access to a cluster or a node within the cluster, then having access to things like audit logs, in addition, to node level logging, is going to be really important for you to have all the necessary data to investigate those incidents.

2:37So the audit logging feature in the API server is actually a really important concept behind that and also being able to use role based access control to limit the different operations that users are able to perform against the Kubernetes API server as well. There is actually one other thing related to the Kubernetes API server that I wanted to talk about, which is a little known feature called API Priority and Fairness, and I actually have a skill that goes into depth on how this feature actually works.

3:07But it actually allows you to assign what's known as fairness rules or I forget exactly what the resource name is called but it's basically just a way for you to set up a rule in your Kubernetes cluster that allows certain users depending on certain attributes like where the traffic is coming from, or who the user is, or what groups they're a member of, and it allows you to control and prioritize network traffic going to the API server rather than just leaving it to the whims of the default configuration of the cluster to determine who's going to have priority.

3:42So if you have some kind of outage situation where your API server is being hammered by a malicious user, then you can prioritize administrative traffic from trusted users. And so understanding how to use that priority and fairness feature, especially in large enterprise environments, is going to be an important topic around ensuring that you have continued high availability and accessibility to all of your Kubernetes clusters. Now, there's also some stuff here around keeping Kubernetes up to date as well.

4:15Now keeping Kubernetes up to date is actually really, really easy if you're using one of these managed cloud vendor solutions because they actually allow you to go into your clusters, and basically just initiate an upgrade. Whereas if you're using a self-managed distribution of Kubernetes like k3s or any others out there, then they're going to have their own separate upgrade processes. And hopefully they have good documentation as k3s does, on how to actually go through and perform version upgrades on your cluster. Now, one of the reasons that it's really important to understand how to properly upgrade Kubernetes is because Kubernetes has a very rapid development cycle where there's new features that are constantly being added, or changed, or sometimes removed even.

5:00So to give you a recent example, in Kubernetes version 1.25, the Pod Security Policy or PSP feature was actually completely removed from Kubernetes. And so anybody who had been previously using that feature is going to potentially experience problems when they attempt to provision Pod security policies if they upgrade their cluster from, let's say, 1.24 and earlier versions where it had been deprecated but still available as an optional feature.

5:31And they upgrade their cluster to 1.25, that could potentially cause a breakages for the users of that cluster. And so by properly understanding how to upgrade the cluster to do the appropriate planning steps is going to be a really important concept for you to understand from a security perspective. Let's also talk about system hardening here as well. So keep in mind that if you are deploying Kubernetes clusters, you don't just have your master nodes. The master nodes are arguably one of the more important components just because they run your backend API server and the controller manager, which basically manages state on the cluster.

6:09It's also got your etcd key value store, it's also got things like your scheduler that determines which nodes in the cluster to run pods on. And so the backend master servers are, of course, very important. But then on the worker operating system side, you also need to ensure that the worker nodes are secured from a logging standpoint, you need to make sure that they have enough capacity available to run the different application components on your cluster. You also need to ensure that there is things like quotas in place to prevent users from competing for access, or taking up too much access on a cluster for the different capacity of CPU and memory that's available on those worker nodes.

6:53And so understanding things like quotas in Kubernetes just from a resource management perspective is a really important concept. But we also want to make sure that the operating system itself has been secured as well. So you're not going to want to go on to these operating systems and just install tons of software packages onto them. And that's going to really open up your attack surface on those worker nodes. So generally speaking, you're going to want to make sure that the operating systems on those nodes are up to date. But you don't want to be installing a lot of software on them, and opening yourself up to potentially new vulnerabilities that may exist in some of those third party software packages that you install.

7:32If you do find yourself in a situation where you feel like you need to install something on those host operating systems for your Kubernetes worker nodes, then just be carefully considering that decision. And if you need to, then maybe just spin up a separate virtual machine outside of your Kubernetes cluster to run whatever it is that you feel you need to install on the nodes themselves. After all, if you do need to run something like a logging agent, or a metrics agent, or some other kind of security agent on your worker nodes, you can do that by creating something in Kubernetes called the daemon set.

8:06And the daemon set is basically a resource you create in Kubernetes that will actually spin up a container on each node inside of the cluster. It'll create a pod for each node in the cluster. So when you join new clusters or new nodes to the cluster, it's automatically going to provision whatever that logging or metrics agent is onto that new node. And when you delete a node from the cluster, then Kubernetes is basically just going to kill off the pod that belongs to that particular node automatically, so that you don't have to worry about managing those individually.

8:39So understanding just the host operating system management process or the worker nodes is also a really important concept. From an IAM roles perspective, this is-- basically what we're talking about here is limiting the permissions on the different identities in Kubernetes. So if you create service accounts for your applications or if you create a service account for somebody to access the Kubernetes cluster with a token, then you want to make sure that you're not giving those users, those identities, in your cluster too much access to the cluster itself.

9:10Generally speaking, different development teams are going to have access only to namespaces within Kubernetes that they specifically need access to. And so you don't want to give, let's say, a developer on some random dev team, cluster admin access if you have five or 10 different dev teams that are using the same cluster. Because that one lone developer that's part of a single dev team could then have a very significant impact if they accidentally do something that damages other application components that other development teams are responsible for.

9:42So using that least privilege model to ensure that there are not too many permissions being granted to the different users of your cluster is really important. So role-based access control. Also minimizing access to the external network. So this is where you basically want to use things like firewalls and network policies and third party network policy plugins like Cilium and Calico as well to control access from your pods that are running inside of your Kubernetes cluster, to the external networks.

10:14Now, as I mentioned, Cilium is a really nice network policy engine here. I actually just recently created some training that covers network policy with Cilium. It also has a lot more advanced features as well, some higher layer, things that allow you to inspect HTTP requests and DNS requests, and things like that. But it also supports just the built-in network policy in Kubernetes that allows you to lock down the network traffic in your Kubernetes cluster to make sure that your pods aren't able to reach out to IP addresses on the internet that they shouldn't be reaching out to.

10:46And in the opposite direction, from an ingress perspective, ensuring that other malicious users on the internet are not able to attack certain applications that are running on your cluster. So if you've ever been familiar with something like Fail2ban, for example, that's an application that runs on Linux that basically automatically bans IP addresses if it detects malicious activity coming from them. So you can think about applying those types of rules to your cluster, but you're doing it at the pod level rather than at the host level. So, anyways, network policy is going to be a really important concept for you to understand in Kubernetes for Certified Kubernetes security, as well as firewalls at the host operating system, at the VPC, or software defined networking level as well.

11:31So there's also one other utility, which is something like AppArmor set comp, these are just built-in components in Linux that allow you to lock down access to the Linux kernel. These are pretty advanced features, so I wouldn't go too deep on these particular topics. But you are going to at least need to be familiar with things like AppArmor, Set Comp, and what Cisco calls in the Linux Kernel R so that you'll be able to understand those concepts if they do come up on your take of the CKS exam. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[AUDIO LOGO] Hi, guys and welcome back. So in the last video, we talked a little bit about cluster hardening as well as operating system hardening for the worker nodes that are joined to your cluster. Let's move on a little bit and talk about Microservice Vulnerabilities here. This is a 20% of the exam here. So this is going to be a pretty significant topic on the Certified Kubernetes security exam.

0:34One of the first topics that is covered under minimizing Microservice Vulnerabilities is to set up the appropriate operating system level security domains using things like PSP. Keep in mind that Pod Security Policy is actually deprecated and that's been removed in I believe Kubernetes 1.25. So PSP is an agent topic but you are still going to want to be familiar with it because if a question pops up on the CKS exam or and maybe your organization has older cluster versions that are still running in its environment, then you'll want to at least have familiarity with what a Pod Security Policy is and how it applies how it associates two pods within your Kubernetes cluster.

1:18So even though it's a deprecated feature, spend some time with it, spend some time playing around with Pod Security Policy resources. Yes, you're going to get those warnings from CUBE kubectl that will pop up saying that, hey, the PSP feature has been deprecated and it's going to be removed in the future. But that's totally OK, because again you probably just want to know that for the exam. And we have a skill in this course that's going to talk about Pod Security Policies. Also the feature that's replacing Pod Security Policies is called the PSA or Pod Security Admission controller in Kubernetes.

1:50So that's the new version of PSPs so to speak. It's a little bit less customizable than PSPs because it has built in tiers that it allows you to apply to your Kubernetes pods. But PSP and PSA are the new security features that will allow you to lock down access to your Kubernetes pods so that your users of your cluster are not able to spin up privileged containers or containers that are allowed to have privilege escalation or having certain permissions set up on them.

2:25We want to make sure that those types of things are blocked so that developers are not deploying privileged applications that could then be compromised by an attacker much more easily than a pod that has the appropriate security context being applied to it. So make sure you spend some time with that. Also OPA here is short for Open Policy Agent. So an OPA is kind of a generic framework that allows you to create what are known as policies and it doesn't just apply to Kubernetes. OPA is actually a generic utility that allows any application out there to basically send a payload to it, and then can actually write these customized rules inside of OPA.

3:06OPA will basically do a determination whether that action should be allowed or rejected, and then it'll send the response back to the caller. So OPA gatekeeper is a specific open source project within the OPA ecosystem. There's also another utility called Conftest as well. That's an open source project that uses the same language that OPA gatekeeper uses to write these different rules, these policy rules. The language is called Rego and it's based on the Golang syntax.

3:38And so OPA gatekeeper is this generic multipurpose policy engine that does integrate with Kubernetes so that you can create these customized policies. And then you can basically determine if certain API requests whether it's to the Kubernetes API server, or to your own customized application that you've deployed onto a Kubernetes cluster, or even an application that you haven't deployed on Kubernetes, maybe you've just deployed it onto a virtual machine just bare metal so to speak. Or maybe just using Docker just a standalone Docker agent rather than using Kubernetes.

4:12OPA has a really broad application as far as policy application goes. It's very customizable, it's very general purpose. And so OPA is not just specific to Kubernetes, but it is a very powerful policy engine. Now one of the policy engines that is not listed in this particular rubric here is something known as Kyverno. So Kyverno is a third-party open source application that is also a policy engine. And what's really powerful about Kyverno is that it actually plugs into another very important concept in Kubernetes known as admission control.

4:47Admission control is I would say a security related feature. It's primarily security related because it basically allows you to plug into the Kubernetes API server. You can actually intercept API requests that your end users are sending to the cluster. And then it allows you to make a policy determination very similar to what OPA gatekeeper is capable of doing, but it's specific to Kubernetes. And it allows you to accept, mutate, or reject API requests that are coming into your API server in Kubernetes.

5:18And then based on the request parameters you can basically say, OK, this user is allowed to do this or they're not allowed to do this, or they're allowed to do this but I'm going to modify something about their request before I actually pass it on. So admission control and kyverno in particular are really powerful features, really powerful applications that you can use within Kubernetes. Admission control is really more of a feature of Kubernetes. And kyverno is a third party application that you actually install into your cluster.

5:49But what's really nice about kyverno, and we can actually head out to the kyverno website right here, is that they actually have this pre-built list of policies here that allow you to control certain aspects of your cluster. So when it comes to things like enforcing certain registries to be used for your container images, or whether it's enforcing certain security context configurations for your pods, or whether it's adding image poll secrets you can access your private container image registries, or, let's see, adding certain labels, maybe you want to enforce that.

6:27Maybe you want to enforce adding network policies to new namespaces that are created on the cluster. There is a huge list in Kyverno here of different policies that have already been built by other people. And this is basically just a registry of these pre-built policies that you can go through, you can select which of these policies you want to apply. Let's take a look at Add image poll secrets here. And then you can actually deploy this cluster policy as a custom resource inside of your Kubernetes cluster once Kyverno has been installed.

6:58And then this policy, Kyverno will actually enforce the rules that are defined inside of this policy. So Kyverno is a little bit more specific to Kubernetes as compared to OPA Gatekeeper and NETCONF Test, but it-- because it is more specific to Kubernetes, I would say that it's a bit easier to use than OPA Gatekeeper. And so I would really encourage you to take a look at Kyverno. I actually really like Kyverno because it's really well supported, it's got a lot of development effort going on behind it.

7:29It's very easy to install. And the fact that they provide this registry of pre-built policies for Kyverno, just makes it that much easier to implement inside of your environment. Now managing Kubernetes Secrets is going to be another major topic of microservices. So when we talk about secrets in Kubernetes, Kubernetes does have a built-in resource called a secret resource type. Unfortunately, Kubernetes doesn't really encrypt or do any advanced protection of those resources, aside from applying role-based access control rules to those resources, because they are a secure resource, just like any other resource in Kubernetes.

8:07But Kubernetes, all it does is it base 64 encodes your secrets and then it stores them inside of the etcd key value store. And that's about it. So if you really want an enterprise-grade secrets solution, you're probably going to want to look to a third party solution. Of course, the major cloud vendors out there provide managed services for secrets. So if you're using Amazon Web Services, then you've got a AWS Secrets Manager, you've got AWS Parameter Store which allows you to encrypt secrets with AWS Amazon KMS Key Management Service.

8:41There's also Microsoft Azure Key Vault if you're using the Microsoft Azure Cloud platform. And I'm sure Google Cloud probably has something similar to that, although I'd have to look up exactly what they call that particular service. Now another nice thing about secrets is that there's actually an open source solution out there from a company called HashiConf. And they produce this open source tool called Vault. And you can actually run Vault on your Kubernetes cluster. It integrates very nicely. And so I've actually created some training that shows how to set up Vault on your cluster, to unseal your Vault. And then make those secrets that are stored inside of your HashiConf Vault system available to the application pods that are running on your Kubernetes cluster.

9:25So Vault is pretty well documented. HashiConf generally does a pretty good job with their open source tools and their documentation for those open source tools, so I really encourage you to check out Vault. Also pod to pod encryption with mutual TLS. The little m there is short, short for mutual TLS. This is an important concept because by default when pods are communicating with each other on the cluster network internally to Kubernetes, those pods are not going to be encrypting network traffic between them.

9:55And if an attacker was somehow able to intercept network traffic that was going from pod running on host number one, to a pod running on host number two, then that attacker could inspect that network traffic, and use that network traffic to grab an authentication token, and then use that token to perform unauthorized tasks in a system. Or they could modify those requests as they're being sent between different services. And those are just things that we want to avoid. And by implementing mTLS by way of what's known as a service mesh solution in Kubernetes, it makes it very, very easy to automate the certificate management for mTLS.

10:34If you'd try to do manual certificate management for mTLS or you tried to write your own system to do that, there would be a lot of complexities that you would have to plan for. And so service mesh solutions like Open Service Mesh or Traefik Mesh or Istio, and many other service mesh solutions out there will actually automate the process of that certificate management for you so that you can get that true pod to pod security between your different pods running across different hosts, different worker nodes inside of your Kubernetes clusters.

11:04So that's really the main topics around Microsoft service vulnerabilities that you'll want to be aware of for CKS. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[AUDIO LOGO] Hi, guys and welcome back. So in the last video, we talked about some of the topics that you want to be familiar with in the Certified Kubernetes Security exam as it pertains to microservice management and vulnerabilities. Now let's shift gears a little bit and talk about supply chain security. So when we talk about the software supply chain in general, what we're typically talking about is getting your application source code that you're writing.

0:37It could be a Python application, Golang, web API. It could be some backend component that you're writing in Ruby, or PowerShell, or some other language out there. But basically you've got your application source code, right? And then typically you go through some kind of testing process, you've got unit testing, you've got integration testing, you've got performance testing, load testing that you need to do. But basically in order to do certain types of testing and deployment on that application, if we do want to do load testing on the app, then generally we would need to package up that application first and then deploy it to some environments so that we can actually run the load tests on that environment.

1:15Or if we're deploying it to a production environment, or to a QA environment, or manual QA processes where actual QA users are actually interacting with the application themselves using their keyboard and their mouse to actually play around with a web UI or something like that, then we need to actually package up our application and get it deployed out to some environment so that those users can do their QA processes on the application. So the whole software supply chain is basically everything from getting your source code actually authored, and tested, and packaged up, and then deployed on dev, QA, and then finally production environments.

1:55That's the entirety of the software supply chain. Now during different periods within the software supply chain, we can hook into that process and perform certain validation steps. So one of those steps that we want to perform is to minimize the base image footprint. And what this basically involves is making sure that the base images that your application developers are basing their application components on are going to be as small as possible. There's multiple reasons that we want to ensure that we are minimizing our base image footprints.

2:30Because we want to ensure that we have as small of an attack surface as is necessary. So we don't want to be installing lots of software packages into our container images. Or for that matter, on the worker nodes in our cluster, we don't want to go through and install a bunch of software packages on those that we don't actually truly need. And so same thing for the base images that we're using for our container images. We want to make sure that those are lightweight container images because we don't want potentially vulnerable software components and dependencies to get installed into those images, and then increase the attack surface that an attacker could use to get unauthorized access to the applications running on our Kubernetes cluster.

3:15Another reason that minimizing the base image footprint is important is because when you're talking about scaling applications and creating multiple replicas for high availability for your different application components that are running on your cluster, then you want to make sure that those applications scale as quickly as possible. So imagine that you had a cluster that has maybe 10 different worker nodes. And let's say that you've got a microservice that you deploy and you only have three replicas of that microservice. So basically it's going to create three separate pods on your cluster.

3:47Well, if you have 10 nodes in your cluster and you have three replicas of this service, then only three out of those 10 worker nodes are going to download the container image for that particular application, and then actually run a container from that image. Now let's say that you took that container or that pod, I guess, and replicated it additional times. So you've got three nodes that currently have that container image and you have seven nodes that don't have any awareness of that container image.

4:18So if you were to increase the number of replicas in your Kubernetes deployment controllers from let's say three up to eight. Well, now an additional-- potentially five additional worker nodes within your cluster are going to have to go out and download that container image from your container image registry. So that they can scale up and run additional pods and containers for your application to scale it up and basically horizontally scale it so that there's additional replicas of it running.

4:49Now if you have a very, very large container image, then the amount of time that it takes those five additional worker nodes to download the container images from the container image registry, extract it from its zipped format, and then actually create the new container from that image, it's going to take a lot longer. But if you slim down those images, you have small base images, and you're only installing the necessary components inside of those container images that are necessary for the application itself, then you're going to ensure that the application scales much more quickly than it otherwise would.

5:24Now when we also talk about supply chain here, we're talking about what registries are actually being allowed. In a production application that if you're working at a very large company and you really have to pay close attention to the security of your software supply chain, we want to make sure that our developers are not able to deploy container-based applications from just any registry. If I had to guess, I would bet that if I went out to Docker Hub or if I went out to Amazon Elastic Container Registry, ECR Public, those are both public registries that anybody can just upload their own container images to.

5:59Also Red Hat has a managed container image registry called Quay, Q-U-A-Y.I-O, I think. And anybody can just sign up for Quay.io and they can host publicly accessible container images on that service. And I would bet you that if I went out to any of those services, that I would be able to find container images out there that have malicious code. And so if we had a developer that was not paying close attention and they just went out to Docker Hub here, and they find some malicious container image to use as a base image here.

6:35So let's say maybe they do a search for MySQL. But rather than using the official MySQL container image to run their database, maybe they scroll down and go a couple of pages in, maybe go six pages in here, maybe they find some random person's MySQL container image. Well, that could very well be a malicious user. I don't know if they are or aren't, but it could be a malicious container image and have some kind of Trojan Horse, some kind of backend malicious component pre-installed into that container image.

7:05And so if that developer deploys this particular container image, then it could potentially have malicious software in there. If they don't do the appropriate validation of who the user is, and looking at the actual packages that are installed into the container image, and looking at the entry point of the container image to see what's actually being spun up with this MySQL image, then they could potentially introduce vulnerabilities into their Kubernetes cluster. And that's something we want to avoid. So by whitelisting certain registries, we can actually exclude these public registries like Docker Hub, or Quay.io, or Amazon Elastic Container Registry Public.

7:44And we can only allow container images that are hosted on a private instance of a container image registry to be deployed to our Kubernetes clusters. So let's say that, for example, you spin up this open source harbor registry. So let's say you deploy a private instance of harbor on your Kubernetes cluster and you set up a DNS name to route traffic to this harbor instance, you could actually create a policy and apply it to your cluster to only allow that particular registry to be used by your application development teams.

8:20Another important concept is digitally signing and validating digital signatures on container images. And so I've got a skill that actually talks about how you can use a utility called Cosign. So this is actually a utility from something known as Sigstore, which is part of the Linux Foundation. And Sigstore creates this utility called Cosign that allows you to digitally sign container images as part of your supply chain, and then publish those digitally signed container images to your container image registry.

8:52And then you can validate those container image signatures before you actually allow your Kubernetes cluster to deploy new containers onto your cluster. And by having that signing process in place, you can again, help to ensure that development teams aren't just randomly going out and deploying just any container image, that there has been certain approval processes that have been gone through in order to digitally sign those images and then validate them before you actually create those containers. And tools like what we talked about in the last video, like Kyverno and OPA Gatekeeper.

9:27Those types of policy management engines allow you to enforce things like verifying images. So we can say, verify an image using Cosign. Or we could basically validate the registry that it belongs to using-- restricting the image registries that we are allowing based on things like DNS names, for example. And so by implementing these types of policy management tools, we can ensure that our supply chain security is secured. There's also something known as container image vulnerability scanners.

10:00And I've got a skill that talks about how to use Trivia. And use Trivia in GitHub actions, for example, to automatically do vulnerability scans. You can integrate Trivia with other components that your organization is using like Slack, so you get automated notifications if you want to from vulnerabilities that are detected in your supply chain. And so that's a really important concept, to know how to use these vulnerability scanners that can look at databases of known vulnerabilities. Basically what they do is they look at the packages that are installed, the dependencies that are installed in your container images, and then they'll query those against a database of known vulnerabilities.

10:38And then it'll give you a report that shows the severity level of these vulnerabilities and potentially how you can either remove that software package to avoid the vulnerability, or upgrade it to a newer package version so that that vulnerability no longer exists. Something else that's an important concept is static analysis of user workloads. What this is referring to is there's different utilities out there that will actually allow you to inspect your Kubernetes clusters and you can actually look at the workloads that are running on there.

11:09And so rather than doing a policy management perspective where you are using admission control to allow or reject certain API calls, you can actually just go out to the cluster and say, hey, show me-- inventory all these different pods for me. And then I want to look for certain rules on those actual running pods. Rather than actually injecting yourself into the API pipeline and approving or rejecting requests. So it's basically similar to using those policy management tools. But rather than injecting yourself into the admission control pipeline on the Kubernetes API server, you're actually just doing an analysis of what's actually running on the cluster itself and then making determinations of what kind of action plan you want to take to rectify any potentially vulnerable configurations that you see on those running workloads.

11:56I hope this has been informative for you, and I'd like to thank you for viewing.

0:11Hi, guys. And welcome back. So, so far we've talked about cluster setup and hardening, we've talked about system hardening as well, microservice vulnerabilities and supply chain security. All of those are very, very important topics. But our final topic here is also a really important thing as well, which is around monitoring your Kubernetes clusters so that you have observability into just the behavior, the performance. The-- any odd behavior, anomalous behavior that could be occurring on those clusters, logging also.

0:43As we previously talked about, Kubernetes audit logging is a really important concept that you can see exactly what's happening in your Kubernetes API server, and then runtime security as well. Let's talk about some of the different topics that we might encounter in here. So "behavioral analytics of syscall processes and file activities at the host and container level to detect malicious activities." So I know that that's kind of a mouthful, but let's break that down and better understand what we're talking about in this learning objective.

1:14So what we've got here is syscall process and file activities. Well, the way that the Linux kernel works is that the Linux kernel itself exposes a variety of API calls that are known as system calls or syscall calls for short. And, basically, when you spin up a container or any other process, even if it's not containerized, on a Linux kernel, then that application is going to be making a variety of these syscall into the kernel. So that it can perform different functions, like creating network sockets, or opening a file for reading or writing, or things of that nature.

1:51That's why we're talking about process and file activities is maybe spawning off a thread, or creating a new process, or anything that's file system related or socket related on the system. And so as those activities are occurring on the hosts, the actual worker nodes on our Kubernetes cluster, we want to be able to observe what's happening inside of each of those nodes. And then we also want to be able to create some kind of rule set so that if something happens that looks potentially malicious, we want to be able to get notification of that potentially malicious behavior.

2:25And we may also want to log it to some kind of external system, like an Elasticsearch cluster or to a cloud-based logging service like Amazon CloudWatch Logs, for example. Or some other kind of logging system that you may implement like gray log, for example. And so there's actually a third party open source utility out there called Falco. And Falco is basically-- let's just do a search for Falco open source here. And so this is basically a third party open source utility that was originally created by a security company called sysdig.

2:57And this allows you to basically install this Falco agent on the worker nodes in your cluster. And you can create these rule sets to look for certain types of behaviors running on those worker nodes by inspecting things like syscalls through EBPF or E Berkeley Packet-- Berkeley Packet Filter in the Linux kernel. Which is basically just a mechanism that exposes what's going on inside of the Linux kernel. So Falco is a very powerful utility. It also allows you to inspect, besides just Linux syscall, it also allows you to look at Kubernetes event logs, audit logs, to determine what's going on.

3:35You can create rules that specifically target those audit logs so that you can find potentially malicious behavior. So let's say that maybe, for example, just as a contrived example here. Let's say that you create a namespace on your Kubernetes cluster, or maybe a couple of different namespaces that have highly privileged applications running inside of them. So maybe one of them is a logging mechanism, and maybe one of them is a metric system, or something like that. And so you want to make sure that these systems that are doing logging for metrics and application and infrastructure logs are highly secured.

4:08And that if anything happens inside of those high security namespaces, you want to get notifications about it. Well, with the Kubernetes Event Audit Log Monitoring, if somebody has unnecessary levels of access to a cluster, like cluster admin level permissions, and they actually go and manipulate one of the running workloads inside of one of those privileged namespaces. Then you can actually capture that from the audit logs, and then receive some kind of notification, whether it's sending it to a syslog channel, or sending a notification to Slack, or sending you a text message.

4:43There's lots of different ways that you can actually develop integrations to send those notifications to some kind of endpoint on the right hand side here. Additionally, Falco also supports monitoring, it says cloud activity right here. So there is actually a plug-in that allows you to look at AWS CloudTrail logs. And so AWS CloudTrail is an audit logging mechanism for your AWS accounts. So if you are an Amazon Web Services customer, and you're using AWS to deploy various types of workloads, whether it's EKS, Elastic Kubernetes service, or whether it's just virtual machines running on Amazon EC2, or maybe you're just using Lambda.

5:20Or whatever it is that you're using, you can monitor your cloud trial logs, and Create Policy rules inside of the Falco rules engine so that you can determine what potential behavior is occurring in those CloudTrail logs that could be malicious. So Falco is a really interesting open source project that allows us that level of observability. Also a threat detection within physical infrastructure. So this is really just around intrusion detection, looking at audit logs, looking for anomalous behavior that could be occurring.

5:53Falco is one utility that could help you detect those types of things. If there's a particular user that's performing a particular action on one of your nodes, and you want to get notification about that, Falco can really help with that kind of thing. But also, more general purpose logging can help with that. So if you've got log files from your infrastructure that are being sent up to a third party service like Amazon CloudWatch Logs or Sumo logic, or any other kind of enterprise monitoring solution out there, that's going to help you detect what those potential intrusions into your infrastructure.

6:28Also we've got attacks. And so we want to-- again, this is referring back to intrusion detection systems. We want to be able to detect attacks that are occurring, and then lock down access to applications so that we can mitigate the attacker's access to our cluster. You could do that through things like network policy. We had previously talked about Cilium and Calico. Those are a couple of network policy engines that allow you to have pretty advanced control over the access that you have against your pods running on your cluster.

7:01Also, service mesh solutions like Open Service Mesh or Traefik Mesh or Istio and others can actually help with this as well because those allow you to do things like rate limiting. So if an attacker is trying to do something like a dictionary attack or just a brute force-- brute force password attack against one of your applications, you can actually plug-in a service mesh solution that locks down access based on rate limiting. So that you can block those brute force and dictionary attacks against your applications. So all of this is really around things like Kubernetes audit logging, using things like Falco to do monitoring, using enterprise monitoring solutions to gather logs from your clusters, to get metrics from your clusters, and things like that.

7:43One other thing around immutability of containers at runtime. If you're not familiar with the term immutable, it basically just means unchanging. So if you have an immutable container, it basically means that that container once it's spun up, can't actually change. And so one of the things that in Kubernetes, a security feature of Kubernetes is-- allows you to create containers with "read only root file systems." That's a feature within the security context of your pod configuration. I would encourage you to just go out and Google "read only containers on Kubernetes."

8:15And this will show you basically how to create-- here's somebody's blog post that shows how to create "read only file systems in Docker and Kubernetes." But I do know that they have some documentation out here under "configure a security context for a pod." And there is a property called "read only root file system" that allows you to set the container's file system as read only. So basically when you spin up a container from a container image, normally you have the ability to just write data into the container's file system. But if you set the read only root file system property to true in your containers, then you can actually prevent any data from being written to the container's file system.

8:53And that's just one less way for attackers to inject malicious code into your applications if they find a way to run unauthorized code inside of your application. In any case, that's kind of a good summary of all of the different learning objectives that we have in order to pass the Certified Kubernetes Security Exam here. So I hope this has been informative for you, and I'd like to thank you for viewing. And good luck on your CKS exam.