Understand and Deploy S3

0:00[AUDIO LOGO] Welcome to the content on simple storage service or S3 for short. I don't have the metrics in front of me. But if I were a gambling person, I would place a bet right now, the S3 is the most popular resource in all of AWS. If I'm wrong, then it's probably EC2. But the reason I say that is that S3 power so much content distribution and storage around the world.

0:34It's amazing. But it has so many other use cases too for storing objects. Now what are objects? Why is S3 so powerful? When should we use it? And how do we start actually administering it, securing it, making it resilient, and redundant? That's what this set of content is all about. So in this set of videos, we are going to dig into S3, perform some really cool tasks like encryption, like understanding policies to allow access to S3, and then even talk about things like replication.

1:07So without further ado, let's talk about the simple storage service. Let's go.

0:00[AUDIO LOGO] We've briefly talked about this already in this set of videos, when we were talking about storage options, when to use S3. S3 is all about object storage, at the end of the day. So really, what it comes down to is, what is an object? And what is S3 capable of versus what it's not capable of? The biggest comparison to this is going to be between S3 and EFS.

0:31So without further ado, let's talk about what S3 brings to the table. Let's go. So if you're going to understand S3, what you really going to understand are objects. S3, or Simple Storage Service, is all about storing objects. Now, objects at the simplest form are files. But I want you to think about how you've been interacting with files traditionally up until this point. Whether you're working with Linux-based machines, macOS, Windows, whatever, you have a hard drive, maybe, or solid-state drive, or whatever it is.

1:06And on that hard drive, you install an operating system. And then the operating system manages a file system. In Windows, we know this, pretty much, as NTFS. And NTFS is really where permissions to access folders and files are managed. So you may have a root folder, which contains many subfolders. These are going to be the world's worst folders. They look like boots or like the state of Louisiana.

1:36And inside of those, you have all of your different files. The way these file systems, and really, the way the EFS service, is designed to work are that the operating system and the file system protocol allow you to put permissions on each one of these objects. And they're inherited as you go down the tree. We can put things like ACLs, which are going to be allow statements or deny statements.

2:06We can apply permissions to groups, and so on. And ultimately, that's what determines access to these files. And really, the metadata of all of this, of all of this folder structure and everything, is baked in to each one of the objects that you create. So when you create a file here, that file is going to have all of the metadata about its NTFS permissions or whatever file system you're using. All of those permissions are baked into it. That's not what S3 is designed to mimic or emulate.

2:39That's what EFS is designed to do. EFS created a file share that you can mount onto something like an EC2 endpoint and then use the EC2's operating system to create a file system on EFS. That's not S3. S3 is just a bucket. And we put files that we call objects inside of it. One of the primary use cases for this are to hold and store contents, in particular static content.

3:13And we've seen this in action already. We've gone through several sets of videos where I created a basic React plus Node application. And instead of having the EC2 instance or the Beanstalk instance store the content, we put them on an S3 bucket. And whenever people needed to download the images or any static content, the website itself would redirect the browser to download it from S3. We love this because offloading that workload onto S3, first of all, saves us a little bit of compute and optimizes the effort needed by the server hosting my React application.

3:56I don't need to have my server hosting static content anymore, when the S3 bucket can host it for me. The S3 bucket, being platform as a service, is effectively infinitely scalable, both in terms of storage as well as in terms of handling the requests that come in looking for it, or rather, looking for that content. To put this another way, the likelihood that my EC2 instance goes down because of my mismanagement or whatever is significantly higher than the likelihood of the S3 bucket going down.

4:32In this particular case, I can clearly see why I may have static images, or videos, or any other assets. And the file system permissions themselves are irrelevant. I'm expecting anyone from around the world to be accessing this S3 bucket and downloading this content. And this really is one of the primary use cases for S3 buckets. But it's not the only use case for S3 buckets. What are some other use cases for S3 buckets?

5:03Well, we also talked a lot about this-- ETL. S3, because it's so scalable and we do have the ability to manage permissions on things like the bucket itself, makes it a perfect target for extract, transform, and load. This will be my S3 bucket right here. And what you should think about this is I'm going to have applications running all around the world, doing all sorts of cool things. And each one of these applications might generate and store their own data according to this.

5:38Maybe I have an application that's for new user signups. I'll put signups right here. Maybe I have an application that's for orders. Maybe I have an application that's for applications, for applications to talk to other applications. The point is all of these different apps are generating data. And this data is valuable to me, especially as someone who wants to try to analyze what's happening across all of my apps and all of my ecosystems.

6:10So using other types of ETL tools, their entire purpose is to extract the data, E, out of these databases, transform them into a standard format on the fly-- that's the T-- and then load them into a destination, like an S3 bucket. Because S3, again, is scalable, it can handle this ingestion point. It can handle a huge volume of data. And it can handle a huge velocity of data.

6:44And it can handle a huge variety of data. This is what we call the three Vs of big data. And this is what makes S3 a great example of a data lake. As data gets pulled out of all of these different databases, throughout all day long, they're probably going to get converted into some type of file format, like a CSV or like a Parquet file. It'll then get dropped onto this S3 bucket-- a.k.a.

7:17a data lake-- so that it's now ingested into the AWS ecosystem. These databases could be running in AWS already. But they could also be running on premises in a data center, colocated, in my HQ, in Azure, or in Google. Once these files live in S3, I can do a whole bunch of things. I can use my analytics tools to query S3 files directly. Or I might want to move these files further into the ecosystem, into something like a data warehouse, like the AWS Redshift resource.

7:55The Redshift resource is a data warehouse, which is a relational database, like a SQL database, that's been optimized for analytics or read-heavy workloads and storing a humongous amount of data. Now, you might be thinking to yourself right now, now, wait a minute, Knox. In the previous example, we were talking about content. Somebody goes to download my website. And they're redirected to an S3 bucket, meaning that anybody can access that S3 bucket. And that sounds fine for static content.

8:26But when it's my critical business data, how then do I want to stop people from downloading that data, especially if there's no file system permissions? Well, that's a great talking point. Right here, when we create this S3 bucket right here, we still have the ability to manage permissions to the bucket itself or specific objects within the bucket. The difference is those permissions are not set on the files or the objects themselves. They're set by AWS.

8:59And as requests come in to the S3 API that is managed by AWS, it evaluates, based on IAM policies, if the requester has permissions to access this S3 bucket or if they have permissions to access specific objects in the S3 bucket. That's right. We use IAM policies and roles to manage access to the specific bucket or the objects in the bucket. And instead of allowing a file system to evaluate if the permission is going to go through, we're allowing the S3 API as the request comes inbound in the first place to manage this.

9:41Now, I don't know what your feelings are around the IAM policies. S3 buckets being very, very powerful and very, very important, now you're being forced into understanding these IAM policies and specifically JavaScript, JSON a little bit better. So that's a big thing need to wrap your head around. And if you're spying ahead, you can see we're going to talk about bucket policy examples in an upcoming video. But consider some other use cases for S3 buckets. One of the best use cases that I particularly love to see and use S3 for-- let's change the color here-- are logs, log shipping.

10:19That's right. Think about it. Well, let me go ahead and rewrite that. Logs. I have apps all over the place. And I may be generating system logs, OS logs, or even application logs. And a perfect use case for this are to dump those logs into S3. We can actually use things like the Kinesis Firehose, which is a file-streaming or data-streaming service, to stream our logs in real time into AWS, who will then write a copy of those logs to an S3 bucket.

10:53You could almost think about this as a live streaming queue for ingesting real-time data feeds into AWS and then directing those to various different places once it's in the AWS ecosystem-- again, like S3 buckets. In fact, I'm doing similar things with this Beanstalk S3 bucket right here or even the mag store UI. You see these job tasks? It creates a folder-like structure right here. And then we actually see what could actually be created right here in the actual file themselves.

11:29Now, one of the final things that I'm going to point out about S3 is one of the more confusing points when you first learn about it. See, I'm obviously here in the S3 section right now. But what region am I operating in? Global. But then what the heck is this right here, AWS Region? How could I be global but also have a region? Well, ultimately, when I want to write a file to a disk in Amazon, I have to tell it what region those disks need to live in. That's what this is all about.

12:00But every S3 bucket, every single one that has ever been created, has a globally unique fully qualified domain name that comes with it. So when you create an S3 bucket, that S3 bucket has to have a globally unique name. That way, traffic from around the world can be routed to S3:// or https://. So it shows up as global because it's going to be a globally resolvable resource.

12:35The DNS of your S3 bucket is, in fact, global. But there is a region that you choose where the buckets live. And the big thing to understand about that is regarding around your cost. While, of course, the amount of data or the volume of data that you store is going to incur a cost, you can choose what disk subsystems your data lives on. So you might be able to put your data on very, very fast hot disks.

13:07Or if it's older data, you could put it on colder disks that cost less money. Or if it's the oldest data, and you really only want to keep it for archive purposes, you can migrate your data all the way back there. And we're talking fractions of a penny per gigabyte at that point. We're going to talk more about the risks that come along with that in a later video. But it's not just the storage that racks up a cost with S3. Importantly, when people access the objects within your storage and download them, you are going to incur egress costs.

13:46As the data is sent out of the data center, that's where you rack up a bill. So if you're starting to upload really large 4K videos that you want to serve publicly around the world, when they start downloading those 4K videos, guess what, that's where you're going to incur your real bulk of your cost. So at the end of the day, use cases for this are going to be static content-- big use case-- or ETL for data ingestion and data lake.

14:24I'm going to put data lake over here. Or application automations. If we're doing things like writing log files or temporarily generating data that other applications need to pick up and read, this is going to be a perfect use case for this. I'm also going to put in here CI/CD pipelines, specifically uploading artifacts. Now, if you're not familiar with the DevOps world of CI/CD pipelines and artifacts, the idea here is all about I've created an application.

15:01I've made some changes to that application. And now I need to automate the process of testing that application and deploying it into various environments, like a development environment, a test environment, or even the production environment. That's what CI/CD does. And lots of times when we're creating an application, we have to compile it. Or we have to upload supporting dependencies to make our application run. And those are known as artifacts. A perfect place for doing local development and then getting my artifacts into S3 so that a CI/CD pipeline can pick up those artifacts and ship them, S3.

15:42That's the go-to use case here. See, we don't need file system permissions to pull that off, do we? We just need a place to store the files. And this is what S3 brings to the table. Now, S3, even as it says Simple Storage Service, it's simple. It can be as simple as you want it to be. Or it can get pretty complicated. And we're going to dig into the more complicated aspects as we progress through this set of videos. For now, this has been understanding when to use S3. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[AUDIO LOGO] Now we begin by actually diving into it. In this set of videos, we're going to walk through the process of creating a bucket. Importantly, we're going to explore in detail some of these different options and how we might want to configure our bucket at the time of creation. This is actually going to require a lot more elaboration as we go along. So in the next set or in the next few videos, you'll learn more about things like bucket policies.

0:31So without further ado, let's create our first bucket. Let's go. So let's dive into it, right? OK. What we're going to do first is we're going to walk through the process of creating an S3 bucket, but exploring the different possibilities as we go through the console of creating the bucket. Right here I'm in the S3 section. Notice the region does say global because right out of the gate when you create the bucket, one of the very first questions they ask you is, where do you want this data to live when it gets written to disk? That's an important thing to get under your head.

1:03Now I'm intentionally going to name this mayc-east because I'm going to put this data in the East US right now, US East 2 data center. The ultimate goal that I have is I'm going to create replication from this bucket to a different bucket in a different region later, something we're going to talk about at the end of this set of videos. So just bear with me as we chug along through this. So do note right here, this is where you get to choose what region you want to work with.

1:34Now, notice a lot of these are actually grayed out for me because some of them are not-- what's the right word there? Not experimental, but you do have to actually apply to enable some of these regions for you to actually write it. So having showed that, we'll leave this right here. Now underneath this come all of the different settings for configuring your bucket. If you have a dollar that you want to copy, you can just choose the bucket that you want to copy from the dropdown right here, and it will mimic the exact types of settings. What kinds of settings would we be looking at anyways?

2:04Well, it starts out with object ownership. What are we really talking about here? Remember, objects are the files that get written inside of a bucket. By default, the setting is to say that the person who uploads the file is not the owner of the object. Instead, by default, with this ACLs disabled setting, we're saying that the creator of the bucket owns all of the contents of the bucket.

2:36This would make sense where I Knox create a bucket. And then I have a fleet of applications that are writing data into the bucket itself. I want to be the one who has control over what happens to the data that gets written into the bucket. I don't want the application to be the one responsible for managing each of the file permissions that come inbound. This works particularly well when I only have one account in the mix here.

3:07I have my AWS account and my applications are all part of that same AWS account. And they're also writing data to it. However, if I have multiple users spread out across multiple accounts and we want to give them autonomy over what happens for each one of the objects that they upload, we can use access control lists. You can kind of think about access control lists as an off-the-shelf canned permission that you can specify.

3:39Now as it says right here, objects in this bucket can be owned by other AWS accounts. Access to the bucket and its objects can be specified using those ACLs. This is different from over here where ACLs are disabled. This goes back again to me the owner and creator of this bucket will be specifying what is allowed to happen in this bucket using only policies.

4:09And this is where it starts to get really confusing. In this case, I'm going to be writing JavaScript Object Notation policies to define what can go on inside the bucket, as well as for a group of objects or even specific objects-- a specific object itself. I'm going to be writing JavaScript to define each and every permission to go on for each of the files that can come in, or importantly, what AWS users or roles are allowed to have object or access to the bucket, the objects or specific objects.

4:48You should think about this policy as sitting in front of your S3 bucket. So here is my S3 bucket right here. With ACLs disabled as a request comes in, it's going to get screened by my policy first before it's allowed to progress on to perform the exact action against the exact object that we specify in the policy. Where it gets even more confusing is when we start to throw public access into the mix, scroll up just to here, right here under object ownership, I've said the ACLs are disabled.

5:24That makes you think, OK, well, I'm not using ACLs. Well, when we start to throw in public access, we can actually define ACLs in our policy. Now block all public access is the default setting because, of course, you want to start off with the principle of least privilege. We only want to slowly open up public access whenever we need to. So when we uncheck this, we see OK, well, we've unchecked block all public access.

5:55Now we have to select what pieces we want to block. So if you're looking at what's going on here, it's basically new, any, new any. And we start off with new ACLs or any ACL, new policies or any policies down here. That's really what we're trying to block access to. So what we're saying if we were to check this box off, block public access to buckets and objects granted through new ACLs, if somebody uploads an object.

6:27So let's say we check this off, create the bucket. Then somebody uploads an object. Then they create an ACL to make that object public access, this would block that from happening. Let's say we create it-- let's say instead, we'll move on to the next one, we check off this box. And let's say we already had an ACL on this bucket that allowed any public access. This is effectively saying we'll ignore it. So any files that get uploaded after the fact will be ignoring this particular ACL.

7:02Then we repeat the process again. But instead of ACLs, we're using policies. So this is effectively overwriting any ACLs or policies that might grant public access to this bucket or objects on the bucket after the fact. I'm just going to go ahead and leave this at the default, block all public access for right now. Versioning is something we're going to talk about in an upcoming video that's pretty exciting. Tags, basically key value pair that helps you organize your AWS resources.

7:33And then, of course, encryption. Now I'm intentionally going to use a KMS key for this. I already have KMS keys created. We've already talked about encrypting data using KMS already as well. So I'm going to go ahead and choose for my AWS KMS keys, important to remember these are going to be the KMS keys in this region, US East 2. I'm not able to select a KMS key that's not in US East 2. So when I click this dropdown, these are my KMS keys that I'm looking at.

8:06I'll choose my demo key that lists right here. Now let's go ahead and talk about this real quick while we have a chance. You see this SSE, that's Server Side Encryption. This is basically saying that we are going to be uploading a file like so. And as it arrives on the server that hosts the disks, it's going to be getting my KMS keys, and then encrypting it, gobbling it up like this before it gets written to the disk.

8:40The S3 resource here is going to be managing access to the KMS key, encrypting the data as it's written to the disk, and decrypting it as it's written from the disk. This is all happening on the server side. There is a potential for there to be client side encryption. And in this case, our calling application would be doing things like retrieving a KMS key, encrypting the data before it gets uploaded.

9:11So I'll put Enc right here. I'll just go ahead and write encrypted. And then uploading that into the S3 resource. This again, is typically handled all in the application code itself, which is why we're not seeing any type of options here. The thing to understand is that S3 can absolutely handle any type of encrypted data, whether it's being encrypted on the server side or it's being encrypted on the client side. So in this case, I'm choosing to encrypt it with my own KMS keys.

9:43And I'm choosing to do this for a very specific reason, to demonstrate how this is going to work whenever we actually go through replication again. So with this being said, we can also explore things like in the advanced settings. Really the only advanced setting we have is with object locks. And it's basically for something like an in place legal hold or some type of immutable storage that you need here. We see this as write once, read many.

10:14And it helps you prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. So if we are uploading something that needs to be legally held and unaltered for a specific amount of time or indefinitely, this is what the object lock does. I click Create the Bucket and boom, now I have my bucket created. Weirdly I created this bucket and an old bucket that I had deleted is now showing back up. But showing is not found.

10:45When I actually click on this bucket, it shows you that it's not found. And if I try and delete this one more time, it shows you is not found. So I don't know what's really going on there, why it's trying to show me this old bucket that I had already deleted. But nonetheless, my mayc-east bucket is now created. Now we've got some more work to do with this bucket as we start to explore things like the policies themselves. When we scroll-- oh-oh, that's under permissions right here. Wrong section. When we talk about the policies, we're going to pick it up in the next video to talk about that. I hope this has been informative for you.

11:15And I'd like to thank you for viewing.

0:00[AUDIO LOGO] This is where the simple storage service becomes a little less simple when it comes down to object policies. Whenever we're creating an S3 bucket, the thing that we keep in mind is the objects themselves don't have a file system to attach permissions to. Instead, we're going to use the AWS platform to manage access into the bucket and to each one of those objects.

0:31And as we've seen throughout this set of videos so far, when we're defining permissions or privileges or access rights in AWS, we're really defining things like what feels like an IM policy. Put this another way when we're defining access to the S3 bucket itself or the objects within them, we're defining them with JavaScript. So without further ado, let's explore some of the many different examples the AWS gives us for the types of things that we can manage with a bucket policy.

1:03Let's go. So you might want to spy on this access section right now because I see three different things right off the bat. And this really all comes down to whether or not I check that block public access button whenever I was creating the bucket in the previous video. First of all, I see public right off the bat. In this case, I uncheck that box so that block public access wasn't created, then I created a policy to actually allow this to be a truly public bucket.

1:38Then or we could say objects could be public if I create a policy to allow it, or bucket and objects are not public. So if we're looking at it, this whole bucket is public. If we're looking at this one, maybe an individual object or a group of objects inside this could be public, but I have to create a policy to allow it or nothing in this public-- in this bucket is going to be public. Now let's start to dig into each one of these to explore what we've done here.

2:11In this particular bucket, I'm going to go into the bucket, then go into the permissions and immediately block all public access bucket settings. You can see right here it's off. Scroll down just to here, and the individual checkboxes here are also missing at this point. So then I start to explore the bucket policy that allows this to be a truly public bucket. If I'm looking at here, the principle is anyone. The action is get the object or get the specific version of the object, and the resource that this is allowable is the bucket name slash, and then a wild card, which effectively means all objects in this bucket.

2:54So we are allowing anyone to get the contents of this bucket. That's effectively what this policy does. And if you want to create a public bucket, this is how you do it. This is, again, with ACLs disabled right here. So what we're seeing right here is with the bucket owner is going to be the one who is enforcing the policies above. These ACLs down here-- they're a list of ACLs, but they're effectively irrelevant.

3:25We've just granted the bucket owner full permissions to control what goes on in this bucket. So let's look at some other buckets right here, like my Elastic Beanstalk bucket. This was created by the Elastic Beanstalk CloudFormation service whenever I deploy Elastic Beanstalk. And this is really for logs of what's going on within Elastic Beanstalk. And understand and also uploading the archived artifacts to ship my website into Elastic Beanstalk. So as I deploy new website code zipped up, this is where it's going to stage these files before it could ship to EC2.

4:02If we explore these permissions right here, block all public access is turned off, but we control specific operations for specific resources in this S3 bucket. In this case, you can see things like managing the actual S3 Beanstalk logs. So, in this case, we're allowing this specific principal, in this case, an EC2 role to create new objects in this specific path right there of this S3 bucket.

4:36See, pretty interesting. Same thing with the EC2 roles down here. We're allowing this EC2 role to perform these actions against this specific bucket and these specific objects inside of the bucket. So this should be highlighting immediately how you control permissions into the bucket and into specific objects of the bucket and when we might want to use this. In this case, we're giving permissions to EC2 instances who've been granted this role the privilege to look at the contents of these specific paths.

5:11That's how the two roles were able to access-- the workers on the EC2 instances were able to access this S3 bucket and pull down the new content that they needed to deploy into the actual website or the web hosting service itself. Pretty cool. And then lastly, the bucket that I just created right here where nothing is allowed to be public. That's because we've turned on block all public access, and currently, there are no policies. Only I, as the bucket owner, has full permissions to this particular bucket.

5:45Nobody else has permissions to this bucket. No other resource has permissions to this bucket. Now bucket policies can get quite complex, and this is why they've given us bucket policy examples to start building off of. Look at all of these right here. We can create policies that require client-side encryption. We can create policies using canned ACLs. We can create policies that allow it-- allow access based on tags or specifying what tags are allowed to be created.

6:17We can create policies to whitelist specific IP addresses. We can create policies whether the protocol being used was unencrypted or encrypted. We already kind of saw access to specific folders whenever we were looking at that EC2 instance trying to access things like environment logs. And we can even require multifactor authentication for specific operations that take place on this S3 bucket, which I think is pretty cool too.

6:48We'll actually take a look at that later. So I highly encourage you to find this document and take a look at how this is going to work. Understanding the logic of these policies can also be tricky. Seeing things like Require server side encryption for an object to be written is going to be a must. So if I'm exploring the logic here, I'm saying deny the put object operation on the specific bucket when the condition is null for this server-side encryption KMS.

7:24So that's kind of an interesting thing. We're basically saying the condition here is when this is the key that's being used for understanding server side encryption. If that is set to null under any instance than deny the put object operation. Do you see how it's kind of a weird logic to understand how specifying this key being set to null and a higher up tier. I don't know, it's just one of those things you have to wrap your head around when you start to see it. It's kind of a similar logic right here for denying a request if they're saying-- if they're not using the correct server side encryption KMS key.

8:04We're again denying the creation in this bucket if this key right here is not equal right there. So if the request comes in and the ARN is not equal to this exact key, then deny the objects. Again, it's kind of a weird ordering of the logic there, but it's one that you really want to make sure you wrap your head around. Here's where we can actually create policies using the canned ACLs. So when we scroll down here.

8:35We can see how I'm granting access to two different accounts. Multiple accounts can access an S3 object to create objects while also giving them public read. This ACL right here is the named canned ACL that allows public read. So in this case, we're allowing these two accounts, the write operations, while also setting this to be a public read ACL-- kind of crazy. A similar policy right here that grants cross account permissions to create objects while ensuring that the bucket owner has full control.

9:12Very similar construct right here where we have we're specifying the principles that are allowed to create an object, but we're also adding a conditional ACL and the name of the ACL is bucket owner full control. It's again a canned ACL. Here's where we can allow a user to have read only access to a specific object that has been tagged with a specific key and value. So here we specify the user based on the principle. They're allowed to get object or historical versions of it in the specific bucket if the S3 existing object tag is a key of environment with a value of production.

9:55See again, it's kind of this weird syntax for managing tags and conditions. But once more, it's really important to see these things and just wrap your head around how this is going to work whenever it comes. And this is why I really want you to take the time to explore these bucket policy examples and try to understand how these conditional statements work because they're really, really important for working with S3 buckets. So now we've explored how policies work on S3 buckets and how they enforce permissions for authorization into the S3 buckets and the individual objects themselves.

10:34I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[MUSIC PLAYING] One of the critical things that you, as an administrator, need to understand about S3, are how to track changes within our files. What happens if a user overwrites a file with malicious data or corrupted data? What happens if they accidentally delete the file? This is really where you start thinking about backups at the end of the day. And S3 implements something called versioning.

0:30But versioning sets up an even bigger conversation surrounding the amount of storage we use, and how we actually want to store all of the different versions of the same file. So first, let's talk about versioning. And then in the next video, we'll talk about file lifecycles. Let's go. So here's what it wants you to see. I want you to see versioning in action. I'm in the bucket that I just created, mayc-east. Let's go ahead and start by uploading a file into it. I'll click on Upload File.

1:01I'll add my file. And then I'll choose something like the CBT Nuggets artificial intelligence policy that just came out. I'll choose Upload. And just like that, it gets uploaded pretty quickly. Right here, I can close this dialog box. Because right now, this is just the Upload status page. And I'll click Close. There. Now the policy exists. Pretty cool, right? I can explore this object right here. Understanding that when it comes to actually coding this, this is actually going to be a key. That's really an important thing to understand.

1:33When we were to actually access this particular file programmatically, using code, it would come back to us like a JSON object. And the key would be the name of the file. Other metadata associated with that file would also come back as part of the object that's contained here. It's another reason why we actually look at these objects, even though they're files, we look at them as JavaScript objects. I see things, like the object URL. But knowing that public access is actually restricted-- when I click on it, what's it going to do?

2:07Access denied. That's exactly what we expected it to do. Let's jump on back into the console. One second, though. My bucket properties immediately tell me that this file does not have versioning enabled, nor is this file being replicated, which is an interesting thing-- nor is object locking going on. We are using server side encryption. And it's currently being stored on the standard access class. This is all setting up a big conversation to talk about what a lot of these things, if not most of these things, actually mean.

2:40So let's see what happens now, when I jump back into my bucket, and I try and re upload the file. I'll go to Add files. I'll choose my CBT Nuggets artificial intelligence policy. I'll click Open. And I'll click Upload. Well, it goes and uploads just fine. I'll click Close. And I still see the file here. When I click on the file, and I click on the versions, I see-- wait a minute. There's no version here. Hmm. What's that really mean? That means that I just overwrote the file that I had originally uploaded.

3:14The old version of the file is gone. You want to see it done another way? Watch this. I'll call this version 1 file right here. And then I'll immediately go and say, file, and Save As. Let's go File-- if it's going to let me. Why can't I click on File? Hang on one second. Let's you Control-S, or Command-S. There we go. Now the dialog pops up. I'll put this on my desktop. And I'll call this File 1, like this. Click Save. And there we go. Version 1 File is now there. So I go back into my buckets. And I'll go back here into mayc-east.

3:45I'll upload the bucket-- or I'll upload the file. I'll go back to my desktop. And upload it, just like so. This is, again, to drive home this concept of what's really happening here. So with File 1 RTF now uploaded, I'm going to reopen that file on my desktop. So let's reopen it here. Oh. Came up on the other screen. I'll now call this version 2 of my file. Command-S to save it. And now I've got a new version of my file, right?

4:17So I'm going to upload this new version of my file called File 1. Click Open. And upload. OK. So now I've uploaded that file one more time. I'll go into File 1 RTF. I see there are no historical versions of it. So when I download this file, and open this downloaded file, it's version 2. So I've effectively overwritten the original version of my file. So what I can do, if I want, is I can enable versioning on my bucket.

4:53So I'm going to go into mayc-east here. Go into Properties. And turn on bucket versioning. This way, whenever a file wants to be overwritten, what it actually does-- so here was-- let's call this version 1 of my file. When I'm ready to work locally on my computer here, like so-- this is my laptop-- And I create version 2 of my file. When I want to upload it into the bucket right here, what's going to happen is S3 will actually store version 2 of this file separately from version 1.

5:29So with bucket versioning enabled, I'll click Save Changes, like so. Bucket versioning is now enabled. I'll go back into my objects. I'll reopen my version 1 file right here. File 1 RTF. And I'll call this version 3 of my file. Command-S to save it. Close it all out. And I'll reupload that file. So let's go back into Add the files. There's file 1 more time. Click open. Hang on. One second. One second. One second.

5:59That was wrong. Hang on. We're going to go back into upload the file here. Add files. Find the correct one, where I just created this version 3 of my file. And click Open. We'll upload it. It gets uploaded successfully. So now, see. I still have file 1 RTF. Only one file 1 RTF. But when I go into versions now-- aah. There's the old version of my file. Now, when files get uploaded, they're going to be uploaded with a version ID.

6:34So if I'm using code, I can say, go to this S3 bucket, get this particular object based on its key. But then get this particular version of it. So one more time, just to see it all in action. I'll bring the File Explorer back up. We'll go back to my desktop file. Open this one up. I'll call this version 4 of my file. Close this out.

7:05Close it out. Do one more upload here. Just to highlight this, I want to show you-- I want you to be able to see exactly what this is doing. Hang on. There we go. Now I got it fixed. Version 4 of my file. Upload. There it goes. Now, when I go into file 1 RTF, and I see versions, I see these different versions that are popping up. I see which one is the current version, and I see the old version here. When I click on the old version, I can see details about this.

7:35Importantly, like when that version was last modified. Let's jump back here into that file, and go back into versions. Also check this out. When I click on one of the older versions, I can download that older version. I can open that older version. I can delete that older version. Or, under Actions, I can do some cool things. I can make this older version public. I can give that older version a different tag. I can share the older version with a pre-signed URL so that somebody can access it.

8:10Or I can download the older version as a different file or a different format. Pretty cool, right? So if I wanted to be like, oh, you know what? This version broke something right here, so I need to restore to my older version. I would just download it, and then reupload it. And then that would become the current version. Now, what's the big takeaway from this? Each one of these versions is a separate file. At this point, you should see how my one 384 byte file is actually times three at this point.

8:45Doesn't seem like a huge deal when it's a simple text file. But all of a sudden, if we're uploading different versions of images, or large Excel documents, or anything like-- Power BI files can get pretty large. Or Tableau files can get pretty large. Data sources, Parquet files. Things like that can chew through a lot of space really quickly. You could see the volume, or the amount of data that you're storing go up by multiples very quickly.

9:19And therefore, your bill goes up by multiples. This is the challenge that comes along with versioning. And that's why it's really important to understand that if you're going to be implementing versioning, you probably also want to be implementing lifecycle policies. But that's what we'll talk about in the next video. For now, this is understanding what bucket versioning does. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[AUDIO LOGO] In the previous video, we implemented versioning, and now we understand that when I want to overwrite or change an existing file, what it's really doing is creating a new version of the file and I can have the ability to roll back to previous versions if that's what I need to do. But that came at an expense. We're now consuming more storage. So what can I do about that?

0:30One of the cool things that AWS brings to the table for S3 are lifecycle policies. This is where we manage the lifecycle of our objects themselves and their are many different versions. Put another way, we're figuring out when it's time to archive an old file or an old version. Let's talk more. In the previous video, we introduced the challenge of versioning, and that's when we start uploading files, then editing those files, then uploading a new version of those files.

1:00They could start to chew through a lot of space because we could effectively have hundreds or thousands of versions of the same file. And if that one file was just 1 megabyte in size and it was 1,000 copies of it, but now we've used through a gig of storage on just one file because it gets frequently edited and re-uploaded. Do you see how this could be a bit of a challenge? This is why we want to start to explore something called bucket lifecycles.

1:31Now, what are lifecycles all about? At the end of the day, this S3 bucket lives on disks. And some disks are faster or more performant than others. And those faster, more performant disks, they cost more money. But it's really, really useful when we have particularly hot data, data that's getting accessed or updated a whole lot. But what happens if I have the current version of my file, it's hot, but the 100th version of my file hasn't been accessed for months?

2:08Wouldn't it be ideal to move the older files to something like cold or archive storage? This is what a lifecycle policy does. And where we go to explore these are under the Management tab of our bucket, in particular, you see lifecycle rules. What we're really defining here are what are the rules or policies for moving old files to slower, less expensive disks? A great example-- and if you're the blog writer of this, I can never remember where I found this, but this is a really common example that's spread around with blogs is let's say I have a business expenses app, and I'm a salesperson.

2:50I travel to a different city and I take a client out to a nice steak dinner that costs something like $350. I take a picture of the receipt, and then I upload it into our business expenses app. Behind the scenes, that picture is going to be living in an S3 bucket. So for this particular payroll cycle, that picture is pretty important because the accountant who manages payroll or business expense reimbursement, they're going to access this picture and validate that my expense report meets up with what the receipt says.

3:28So within the first 30 days, that's a pretty important picture. But then let's say 90 days have passed, am I accessing that picture anymore? No. Why does it then need to live on premium SSDs? We probably want to rotate it back out to slower disks. And this is what we define with lifecycle rules. So here let's create a lifecycle rule real quick. We'll give this a name like liferule, and then something like mayceast to specify the bucket.

4:01You then have the ability to define the rule scope. Put another way, this is a filter defining what exact kinds of files you want this to apply to. As it says, filters can be applied to file prefixes, like every file we have begins with dev- Maybe we want that all of the lifecycle rules to apply to every file that begins with dev-, that's what the prefix does. Or maybe we want it to apply based on tags, or maybe only if objects are of a certain size, or any combination of these that suits our case.

4:41Filters are highly advisable and right below here's where we get to see how to specify these. Here's the prefix, here's the tags, and here are the object size rules that we can specify. Or you can just say apply this to all objects. Now, it does tell you if you want this rule to apply to all objects, we need to acknowledge that this is going to be moving all of our objects after a certain time has gone by. If you want to filter these out, change this up from above. So you have to acknowledge this.

5:12Now, underneath this, whether we choose the filters or not, we then have to choose the exact rules that we want to take this in action to. This is effectively asking-- let me change the color here because that yellow is not showing up very well. This is effectively asking what kinds of versions do we want these actions to take on. Do we want to move the current version? Do we want to move the non-current versions? Do we want to expire versions? Do we want to permanently delete non-current versions, or so on?

5:47So it's not just about moving them to a different tier of storage disk, we could actually delete old versions if that's what we want to do. Let's say I just want to move non-current versions of objects between storage classes. So I want to keep my current version on the hot disks and any older versions of that, I want to move to slower disks. Now, how do we choose which tier to go to? When we click the dropdown, we actually see we have a lot of different options, and intelligent tiering is pretty cool.

6:18Amazon basically uses machine learning to move data between tiers because we don't know the kinds of access patterns that we may be having on a specific type of object that's being uploaded. For instance, think about that expense reimbursement ticket. I can pretty easily predict that ticket, that picture is going to be accessed within the first 30 days because accounting has to reimburse that expense within that first 30 days. After that, it's probably not going to be accessed anymore.

6:52But if I'm uploading objects where I have no idea how frequently older versions are going to be accessed or not, I'm going to choose intelligent tiering and let Amazon determine which standard class this needs to go to. Now, if I knew, for instance, that that picture was going to be moving to an older tier after 30 days, I might want to use something like standard tier like standard SSDs, but under the infrequently accessed tier. This still guarantees me to have 30-millisecond storage access, but I might only access to it once a month or so.

7:30That's what the standard-IA does. If I want to reduce my cost even more, I might use One Zone-IA. This is the same thing as standard, except for the data is only going to exist in one availability zone. So I'll say non-current-- right here, non-current versions are going to be moved to One Zone-IA after a certain number of days occur. Let's say after 20 days have passed, I want to move non-current versions of my files into One Zone-IA.

8:04Now, the number of newer versions here is basically saying I want to keep my non-- like up to a certain number of versions still in the hot tier. So if I put something like 5 right here, it would be keeping 5 non-current versions in the hot tier, all other non-current versions will be moved. So in this case, I'm just going to delete this out because it's optional. Now, this moves them into One Zone-IA, which still has 30-- [STAMMERING] what am I trying to say here-- with millisecond access 30 days minimum storage duration.

8:43That's what One Zone-IA does. Is it's still relatively hot, it's not frozen, but it's not super hot anymore. It's what's in between, it's cold. Now, what happens if I want to move data out of that and into true archive storage because we're not going to use it anymore? We like to call this write-- write once, hopefully read never. This is the WORN strategy. The idea with archive storage is we are being forced to access it for some reason.

9:17Maybe we have some legal reason that we need to archive data, but this particular storage is going to be super cold. It can take a long time to access it because it has to be rehydrated. And the cost of rehydrating could be extremely expensive, but nonetheless, the cost to store data in archive storage is extremely cheap and we've then met whatever our compliance goal is to store this data. So I can add another transition underneath this one by clicking Add transition right there.

9:52Within this, we can start to explore the different types of glacier archive storage that us offers. When I click this dropdown, we see glacier for flexible retrieval requires 90 days minimum storage, but retrieval can take minutes to hours. Glacier Deep Archive can take a retrieval of-- allows you to retrieve data less-- this data is access less than once a year with a retrieval of hours and requires 180 days minimum.

10:22So in this case, I may specify something like after 90 days living in One Zone-IA may move it then to Glacier Deep Archive. When we put this down here, we have to acknowledge this lifecycle rule because we're using glacier. Down below this, it shows you to review the transition of what happens. When we have non-current options on day 0, they're just going to stay non-current. On day 20, they're going to be moved to One Zone-IA. On day 90, they're going to be moved to Glacier Deep Archive.

10:55So fire off Create rule-- oh and, of course, it does say you have to have 30 days required before transitioning to that. It did tell me that. Didn't it? Of course, it did. Right there. So we'll go ahead and create this rule right there and boom, the rule is now being created. Give it a simple refresh and now my rule has been brought to life. So now we have the ability to manage versions, and all of this historical data by moving it through to older cheaper storage based on how frequently it gets access.

11:27This is the importance of understanding tiering options in S3 buckets. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[MUSIC PLAYING] One of the final things that we're going to talk about is tiering. Again, remember, this is a really important thing to understand. The fully qualified domain name for our S3 bucket is globally accessible. However, where the files actually live are stored within a specific region. And there may be reasons why we need to replicate data from one region to a different region. Or, importantly, from one account to a different account.

0:33So in this video, we're going to explore replication. This is going to be a big video. Because replication can be big to set up. So without further ado, let's implement an S3 bucket replication to a different region. Let's go. The final thing that I want you to see in experience is replication, because replication can be a big undertaking. We absolutely have the ability to take a source bucket, and replicate it to a destination bucket.

1:04Importantly, there are some caveats, or some things that you can understand. The source bucket can be in a different account of a destination bucket. Or they can be in the same account. They can also be in different regions. Or they can be in the same regions. They can also be encrypted with KMS keys. Remember, KMS keys are specific to their region. So if I'm encrypting data in a source bucket in US East 2, and I want to send that data to a different region that's going to be using a different KMS key, we're going to have to manage that process, telling it what keys are supposed to be used to decrypt the data, and then re-encrypt the data upon arrival.

1:50There are a bunch of different reasons why you might implement different strategies of replication. In the S3 documentation, they actually have a section of why, or when to use replication. There's different reasons when you might use cross-region replication. First one is to meet compliance requirements. Second one is to minimize latency to your customers. Or operational efficiency, which is really about if you have different compute clusters in different AWS regions. But there's also reasons to use same region application, like aggregating multiple buckets into a same bucket, or configuring replication between a production and a test environment.

2:31There's even some great reasons to build in two-way replication. I love the particular use case for managing failover in keeping data consistent as it moves from one region to the next. So how do we actually implement this? Well, first of all, you have to have a source bucket and a destination bucket. I'm going to go ahead and create a destination bucket right now, like I said. I'm going to call this mayc-west. I'm going to put this in the US West 2. Replication requires, importantly, that bucket versioning is enabled, which means you probably want to consider lifecycle policies on your buckets as well.

3:08Also, to demonstrate this and understand the complexities of using KMS keys, I'm going to choose to use a KMS key. Now, since I've chosen US West 2, that, of course, means I have to use a key that I've created in US West two, which I have right here. West KMS 3, which was what was just disabled right there. I'll go ahead and create this bucket right now. So just remember, bucket versioning must be enabled. And I've already got a KMS key in the West that I'm using to encrypt this data. Now my goal is to establish replication between the US East 2 bucket and the US West 2 bucket.

3:45So I'm going to go into US East right here to be my source bucket. I'll go into Management. And right underneath my Lifecycle Rules, I see Replication Rules. We're going to create a replication rule, and get this thing started. I want to say right now, this process is so much easier than it used to be in the old versions of eight of AWS. Let's go ahead and call this mayc-replication, just to go ahead and call that a replication rule. The status of this is going to be enabled. And it's going to be a top priority.

4:16We can have multiple replication rules. And this really matters. Because if an object has already been replicated to a different region, then it will not be replicated by a second rule to a different region. Once it's been replicated, that object is no longer available for replication. So setting the priority of your replication rule effectively dictates what's going to be the target region that you try to go to first.

4:48All right. Here we go. We start off with our source bucket. Well, that's the bucket that we're working on. Just like the prefix-- or just like the lifecycle rules, we specify filters of the types of objects that we want to replicate. This can go based off of prefixes or tags. Notably, not file size. In this case, I'm going to choose replicate all objects. Now I'm going to choose my destination bucket. From here, I can choose a bucket that exists in this account. But notice, I could specify a different bucket in a different account.

5:21I would need to know the account ID, and then the bucket name. That's also a good thing to know. Also keep in mind who are the owners of the objects when it goes to a destination. We can change the ownership to be the destination bucket owner. And therefore, they control the policies for those objects when they come inbound. That's a big security thing that you want to keep in mind. You may have a file that's private. The destination bucket might be a public bucket. So do keep that in mind. Since my bucket was created in this account, I'll go ahead and just choose to browse S3.

5:55And I'll choose mayc-west right here, and choose that particular bucket. Now, the IAM role. There's going to be, the AWS S3 service is going to be the one reading the objects out of your source bucket, and writing them to a destination bucket. This is a really important thing to understand. And that's why they're asking you to choose an IAM role. Now, I can tell it to create a new role from scratch for me, but that's going to give me a little bit of a problem with the KMS keys when I get started.

6:29Because right now, this newly created role doesn't have access to the currently in use KMS keys. So I could pre-create this role, and then specify the role here, while also going into KMS and granting that role permissions to the keys. For instance, I want you to see, this is what that particular role would look like. I'm going to go ahead and collapse it here under my code samples for SAA C03. You'll see the S3 folder.

7:00And this is what the role is going to look like. We're going to allow the allow process for get replication configuration. List the bucket for the specific bucket. Then we're also going to allow get object version for replication, object version ACL, object version tacking for all objects in the source bucket. Then we're going to allow replicate object, replicate delete, and replicate tags for the destination bucket.

7:30So we'll give that particular role a name, then grant that role permission to the KMS keys. Now, I'm going to go ahead and fix this after the fact, since I'm already here. I'll let it create a role for me. We're also going to specify when you write data to that target destination, it needs to be encrypted with the encrypt-- So first of all, when we choose encryption, we're going to replicate with encryption. The first thing it's asking me is KMS keys for decrypting the source objects.

8:01I used my demo key for my source bucket. Next, in this option, is what is the KMS key that's used for encrypting destination objects? I'll choose from the dropdown here. And I'll choose the West KMS S3 key that I created for the destination topic. I also can choose a different storage class. So if I want to write from standard to, say, something like 1 Zone IA, I can override the destination target, and write it to a different storage class for storage purposes.

8:33Pretty cool, right? I'll go ahead and uncheck that right now, and use the defaults. Now, the last things that I want to enable here. Replication time control is all about aggressively replicating. This says, when you enable this, it replicates 99.99% of new objects within 15 minutes, and includes replication metrics down below. Replication metrics are all about streaming replication task data to the CloudWatch monitor. So you can monitor the amount, or number of objects that are currently queued for replication, the amount of data that's been replicated, or even diagnose replication failures with this.

9:13For my example, I'm going to go ahead and check this off. It is important to know that in some cases though, replication can take up to two hours for replication to actually occur between your source and destination. So with that being said, I'll click Save right here. It asks me if I want to replicate existing objects. But since I don't have any existing objects in my S3 bucket, I'll just go ahead and leave this at No. And click Submit. So now when I see replication rules, I see I now have replication enabled.

9:48But I'm not done yet, am I? I still need access to those KMS keys. So I'm going to go into my replication rules here. And from here, I'm going to check this out. We're going to go ahead, and actually, under Actions, click Edit. When I scroll down to the IAM role, I see the name of the role that was created. S3 CRR-- that's Cross Region Replication-- role for Mayc East undercsore 1. That's the role that needs access to KMS keys.

10:20So I'm going to jump over to KMS. Let's go to my kiloohms keys. That way, this role that's performing the replication can decrypt and encrypt the data. I'll go into my key here. I'll go to my key users. I'm going to add that user. I'll search for CRR. And if I expand the name, I see the newly created role. And I'll add them to be a key user. I'll change my region to be the Oregon region right there.

10:53We'll go into the customer managed keys, West KMS. And under the Key Users one more time, I'll search for CRR. Expand the name. Check off this role. And now click Add. Perfect. So at this point, that role now has the access to the S3 buckets, and the KMS keys that are used for encryption and decryption on those S3 buckets. So my final task here is to do some testing. I'll go ahead and upload a file. Let's go ahead-- Oh, I did have files in here.

11:24Duh. I'll go ahead and add a new file here into the mix. We'll go into something like Recents. And I'll upload my Mayc image. Here we go. We'll click Upload. And there we go. It's now been successfully upload. Within a few minutes, this mayc 5013 JPEG should be replicated over to mayc west. So if I jump over to mayc west, like I said, it could take a little while for this to kick in, but it will eventually appear. Because I also enabled metrics, I can jump into the metrics.

11:57Scroll on down to the replication rules. Choose my replication rule right here, and display the charts. Now, like I said, it's going to take a while for the replication service to come to life. You got to give it some time. It's not going to kick off right away. It could take up to a few hours for the first replication to occur. But once replication is up and running, because we enabled RTC-- right there, the Replication Time Control. Right there. Oh. Right there.

12:27That means it's going to aggressively replicate data once the replication jobs get kicked off. So jumping back to buckets. Now that a few minutes have passed, I'll go into mayc-west. And there it is right there, replicated. So that's how we configure replication across our S3 buckets. I hope this has been informative for you, and I'd like to thank you for viewing.