Describe Types of Core Data Workloads

0:06We are collecting a lot of data these days. There's no way around it. It doesn't matter what industry you're in, what business you're involved with. If you are in IT, there is data collection going on. Everyone's walking around with a little data collection device right in their pockets. And we all have data collection going on with IoT, our refrigerators and our cars. And even our light bulbs now are data collection devices. So there's a lot of data. And we have to deal with it. And it's no longer really feasible to deal with all this data instead of our own data center. There's too much cost associated with developing and building out a big, robust data center to store this data.

0:40So increasingly, many of us are choosing to migrate and move our data processing and our data storage into the cloud. Now, when we are talking about data storage in the cloud, there's some things that we need to consider. We need to consider our data storage models. And we need to consider our data processing models. And if we're going to ingest all this data from all these different places, well, we need to understand what's the appropriate way to handle all that data. So in this video series, I want to talk a little bit about that, some of the different standard data models that you might run into when you're deploying to the cloud, as well as some of the different standard data processing or ingestion models that you'll run into.

1:14I'll finish up by talking about a few of the different services that some popular cloud offerings have that allow us to handle the data the way that we want to handle it. So stick around. This should be interesting.

0:00[MUSIC PLAYING] I mean, look, there's probably as many different ways to store data in a computer system as there are people on the planet. Everyone, if they had to sit down and think about it, would likely come up with a different way. So certainly, there are a lot of options when it comes to data storage. But by and large, the technology community has settled on two very broad buckets in which our data storage technology, our databases, rely on in order to architect and store the data. And those two buckets are what we call NoSQL data, which is a little more loosey-goosey unformed data, and what I want to talk about here, relational data stored in relational databases.

0:39Now, relational data is this concept that was developed in the 1970s, and it's all about how we can logically structure the data storage on our computer systems so that we can rely on the database engine software that's already written and compiled and running in order to handle that storage for us. And we can simply ask questions of that database, typically using something called SQL, Structured Query Language, and get back results quickly and efficiently, while also being comfortable and confident that that data is being stored in an efficient way, so that everything's hunky-dory, and we're really happy, and we're really good with how our data is being stored in our computer systems.

1:17So relational data-- this is one of the more traditional ways of storing data. It's one of the more traditional database systems that you will run into. And it's really about storing data in a relatively fixed structure, and one that super emphasizes efficiency, data integrity, and something I call transactionality. So efficiency is efficiency in storage space to make sure that we don't take up a whole bunch of extra storage space inside of our computer systems, we don't duplicate a bunch of data, we don't have data that's too large or too many indexes.

1:48Data integrity is what we talk about ensuring that the data lines up, that it matches. If we have a table that lists a bunch of user IDs in it, for example, well, data integrity ensures that those user IDs are actually represented in a user table. Those user IDs tie back to a real user. We don't want to insert a fake user ID or a user ID that doesn't exist. That would make our data not make sense, so that is data integrity. And transactionality, what I call transactionality, is the fact that relational data is very good at handling transactional data. Think about entries in a bank account or a ledger.

2:20There are just one after the other. They add or they build on the previous entry itself. Relational data is very good for that type of data and that type of data system. Now, you've likely heard of relational databases. They are pretty much the most common and most popular databases out there. They have been for a long time. We're talking about Microsoft's SQL server, Oracle, Postgres, MySQL. These are all relational database systems. And one of the things that you'll notice in almost all of their names are those three letters, SQL, which I mentioned before, stands for Structured Query Language.

2:54And we'll take a look at an example of some SQL in a second here, but I just want to talk about this a little bit. Structured query language is a different standard that was developed alongside of relational databases, but certainly not limited or contained to them in any way. And it's a standardized way to insert data into our databases, retrieve data from our databases, and even modify and manage the actual databases themselves. I have a whole course here at CBT Nuggets on SQL, both inside of my course on Microsoft Access, as well as on some courses on the Oracle database system.

3:27So if you want to learn more about SQL, you want to know more about it, check out those courses. I really dive deep into it. I'm just going to touch on it here. So hopefully, we can understand a little bit about how it works. In practice, I think relational databases are very easy to sort of comprehend. It's very intuitive. When you look at a database that is stored in a relational data system, you're going to look at collections of data called tables, and those tables are going to be filled with rows of data, and that data is split into fields. Now, this is similar to-- well, the analogy that's often made is a spreadsheet.

3:58So think about a spreadsheet full of, say, customers like I have up here. You've got this worksheet, which is analogous to a table. This is like a table of data. And it's split up into rows. They are numbered down the left-hand side. And each of those rows is split into separate individual cells, or fields that we call them, inside of relational database. And the key here is that each little field contains this sort of little nugget of data in it. And it's all related to one another. All of the data that's on row 5, for instance, right here, is related to the same customer. It's their phone number. It's their email address.

4:28It's their last name. It's their first name. But the columns are also related to each other. All of the data that's in column D here, as an example, is email addresses for all the different customers. So we have this relationship inside of our table between the different rows and fields, so that we can reference specific individual data points inside of this. And this really is what it looks like when you work with a relational database. For instance, here is an example of an Access database. I have a whole course here on Microsoft Access. It's a great starter relational database if you want to learn about working with databases. This is an excellent place to start.

4:59But here, I'm looking at the customer's table. And look, it's laid out exactly the same. It looks just like a spreadsheet. We have our different rows. We have our different columns. In the inside of the individual cells, or again fields, as they're called here, we have the individual values. So what makes it a relational database? Well, there's a couple of specific properties I think you find universal to relational databases, regardless of whether it's MySQL or Postgres or SQL Server. Whatever you're using, you're going to run into these features. The first thing is your data is what we call atomic.

5:30This is kind of a database term, a little bit of database vocabulary. And atomic data just means that the cell or the field contains the smallest unit of data that you would ever reasonably or practically want to look at or collect. So oftentimes, how that works in practice is a cell does not have a whole customer name or whole user name or whole, I don't know, employee name. But rather, you have a first name cell and a last name cell. You split those up because sometimes you only want the last names. Sometimes you only want the first names. If you need the whole name, you can retrieve them both, and then add them together using your software-- connect them up using software.

6:03But the fields are atomic. It's got, well, the first name. That's the smallest value I would ever want. And there's some kind of wiggle room here. There would be people who point out, hey, a letter is smaller than a name. Why don't you put each individual letter in its own cell? And that's because, typically speaking, you don't need to select or grab an individual letter of a person's name. I very rarely need just one letter of a person's name. On the few times that I do, I can select out the entire name, and then just use some code again to strip out the individual letter I need. So there's always a little bit of sort of wiggle room there as you figure out what exactly makes sense for your system and how you're going to use your data.

6:38But by and large, you can count on your data being atomic within those cells, and that is really a key property of relational data. Also, a key property is that each row is unique. For instance, we were just looking at those rows of customer data. You would not want to have two different customers listed. Or rather, you would not want to have the same customer listed twice inside of that table. That could cause some real confusion. Why? Well, you might end up changing the information on one row and not on the other. Now you have two rows that represent the same customer with different data. Which one's right? That's how you enforce integrity.

7:09And one of the ways that you enforce data integrity-- by having unique rows, by having data and information only in one place ever at one time, you can help to ensure that you don't get conflicting data. You don't get ambiguous data. The data exists in one place and one place only, so the system can enforce integrity on that data. And then lastly, like I talked about, SQL is the language that is really typically used to interact with relational databases. I mean, it doesn't have to be. But 99.9% of the time SQL, Structured Query Language, is in fact, the system or the language that we use when we want to interact with these databases.

7:41If you're wondering what SQL looks like in practice, well, it looks a little something like this. Here, you can see a simple SQL statement, and it's almost read like English. I'm saying select first name, last name, and phone from customers where last name is Like F*. You can almost read it out like a sentence. What this is saying is, I want to grab the first name value, the last name value, and the phone number value from the customer's table, but only when the last name value begins with the letter F. That's what this little Like F* is. That's saying, hey, anyone whose last name begins with the letter F, pull that data out, return that data to me.

8:15And inside of whatever system I'm using, this would allow me to get those records out of that database really, really easily. And again, because SQL's kind of standardized, this statement, this query right here, can be applied to a lot of different databases. It helps make your data and your query language a little bit separated, a little bit unique from one another. So all of this together is relational data. And it's a lot we just covered. I realize that. If you want to know more about it, check out those courses I recommended at the beginning of this video. Here's what your main takeaway should be. Here's what you should understand about relational databases.

8:46They store the data, the information, in a relatively fixed, rigid, enforced structure that helps to maintain data integrity and efficiency, and it helps to support the SQL query language so that you can structure your queries against that database with relative ease. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[MUSIC PLAYING] As I mentioned before, there's a lot of different ways that we can imagine storing data inside of a computer system, but THAT doesn't mean that all of those methods are going to be, well, a good idea. They're not all equal, and they're not all going to serve our needs. So it's probably unsurprising that we have kind of naturally fallen into a few different standardized data storage models over the years. One of the first and earliest data storage models that was developed is something called the relational database. That was in the 1970s, and it's still widely popular and widely used today.

0:33You may often hear referred to as an SQL database. And I talk a lot about that in my video on relational data models. But with the rise of the internet and the world wide web in the '90s and early 2000s, we started to develop a need for a different type of data storage model, something that has come to be known as the NoSQL model. And the idea was that, well, not all of our data fits the model of a relational data model. Sometimes the data that we get is not rigid. It's not structured, particularly when we're collecting it from the wide diaspora of the internet.

1:03Data can be coming from any different place, from any different system, from a different language, or any different geographic location. So we need to be able to be a little bit more flexible, a little more loosey-goosey and handle some different things that go on or might happen with the data. We also need to deal with a lot of data. It's no longer the case that we have small really constrained portions of data. We are collecting a ton of data. And so we need to be able to maybe split our database up over multiple servers. And traditional relational databases don't deal well with being split up over multiple servers. They expect to be located on one single server, one single computer handling all of that workload.

1:37So enter the NoSQL model, and the NoSQL model is really about valuing speed and flexibility over consistency and efficiency. So consistency and efficiency are two things that I would generally consider to be features of a relational data model. When we talk about NoSQL, we're really talking about speed and flexibility. You're going to hear the word ad-hoc a lot. And there's a lot of popular NoSQL databases that are around and available now. You may have heard of some of them, MongoDB or Cassandra. Redis is a very popular in-memory caching system as well as a key-value pair database, a NoSQL database.

2:12DynamoDB is Amazon's AWS cloud-based NoSQL database service. And in relation to that, CosmosDB is the one from Azure, their cloud-based NoSQL model that's served up on the Azure platform. So these are just some of the different databases. There's a lot of different NoSQL databases that you might run into. And what makes NoSQL important, again, is its speed and its flexibility, its ability to really deal with a lot of different stuff. Now, NoSQL was a phrase that was originally developed to mean no SQL at all, but really today, most people will indicate that it means not only SQL.

2:45This is really the terminology that's been adopted. And that's because usually you're not going to have an SQL interface to NoSQL databases, but you still can. People are very familiar and comfortable with SQL. So a lot of NoSQL databases, while they don't store in a relational model, will still use the relational SQL language or some variation of it to allow you to interface with that database and put data in or query data out. So you still run into some SQL. Not only SQL is what the NoSQL typically stands for. And NoSQL databases like to handle ad-hoc data ingestion from a lot of diverse sources.

3:21Think about IoT, right? We have all these different devices that might be scattered all over a warehouse that might be scattered all over the country or even all over the world, and they're pulling in all of this data. And a lot of things can happen. They might miss data values. They might report the wrong data. The data may not quite fit the standard definition of the data that you expected inside of your data schema. NoSQL databases are ready to handle that. They're ready to say, OK, I can take on a lot of this different data, and I'm going to be able to be flexible enough to know that even if it doesn't fit what I'm expecting, I can still store it, capture it in some way that can be understood and read later down the line by future systems so that it can either be discarded or cleaned up or otherwise addressed in the way that it needs to be addressed.

4:01NoSQL databases also take advantage of public cloud offerings to really fully realize their potential. I think this is very, very important because one of the things that we lose with NoSQL databases is efficiency. That means we need more computing resources, more memory, more CPU power. Well, that stuff is expensive, or at least, it was expensive until the rise of public cloud offerings. The ability to simply pay for what we need to use to take advantage of a big, already installed managed data center like Microsoft Azure to run our NoSQL databases means that we can really get a lot of the advantages of NoSQL and minimize the impact of that loss of efficiency.

4:40Now, the relational model is fairly straightforward and fairly understood. NoSQL is kind of a catch-all bucket for a lot of things that are not the relational model. So there's-- well, again, a million and one different ways that you might store data and call it NoSQL, but by and large, we've settled into a few different standardized data models. The really popular ones that you're going to run into are going to be something called key-value pair NoSQL data models. And this is just like it sounds like where you have a key and a value paired up together. So I can look up inside of my NoSQL database, and I can say, hey, what is the value of the name?

5:11The database will shout back to me, oh, that value is Ben. That's a key-value pair. You see key-value pairs a lot in memory caching. Redis Cache is a key-value pair database. You also see it in things like configuration files where you just have a text file with a list of different configuration variables and what each of their value is. That's a key-value pair database. The other common NoSQL data model that you'll run into is something called a document database, and a document database is kind of like a collection of key-value pairs that are all bundled up. So you'll see here, I've got name Ben, but that key-value pair is actually part of a larger entity or a larger object that we call a document.

5:47And those key-value pairs may not just be simple standard values. For instance, here I have a key called order items, and its value is actually another document with a series of key-value pairs. So in a document database, you may well have nested documents or nested lists, things of that nature, more complex data structures. A great example of a document database is JSON. Have you ever worked with JSON data? That's basically a document database. Another good example is an HTML document. You may have heard of the DOM, the D-O-M. That stands for Document Object Model. That's a way to look at your HTML document and treat it like a NoSQL data document.

6:23So some other less common NoSQL data models but still very, very important and you'll want to understand them are going to be things like a graph database where you take those documents and you actually create relationships between them so that you can understand how those different objects, how those different documents relate to one another, column databases, time series databases, object storage, which is how you store binary blob objects. Think about things like video files or media images, large binary files that don't have a lot of discrete data but are rather large big chunks of objects that you just need to store somewhere efficiently.

6:54So all of this comes together to form the backbone of what we call the NoSQL data model, not only SQL. And these are the ad-hoc, fast, less efficient, but more responsive data models that are really frequently used for a lot of data operations inside of the web and inside of the cloud these days. The most common data NoSQL data model that you've interacted with is the search engine. Every time you interact with a search engine on the internet, that data is stored in the back end inside of a NoSQL structure. That is not inside of a relational database.

7:24If it were inside of a relational database, you would never be able to get the responsiveness. You would never get the speed out of those results that you see. So the NoSQL model sacrifices some things. Those databases take up a lot of space. They are not especially efficient with space usage. But in return, you get very fast search results. So that is the NoSQL data model. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[MUSIC PLAYING] So we've talked about relational data. And we've talked about NoSQL data. And the question now is, well, which one do I use? I don't know. The answer to that is going to depend on your workload and your requirements. I think you understand that the different pros and cons to these two different models can apply in different scenarios. And I would venture to guess that in most modestly sized organizations, you're definitely going to run into the need to use both of them for different types of data that you might be collecting. For example, maybe your customer order data is being stored in a transactional MySQL database, whereas feedback, customer feedback, coming in over a web form, might be getting dumped into a NoSQL database.

0:40So you'll have these different database models used in different systems and different scenarios, depending on the type of data that you're receiving and how you want to store and manage that data. Now, one important thing I think to keep in mind is we're talking about this in the context of public cloud providers because these cloud providers have really opened the floodgates for NoSQL data. One of the number one limiting factors for adopting a NoSQL model in the past has been the cost for setting up the computing infrastructure to support it, right? It's not especially efficient with our computing use. That's why we need a lot of power behind it.

1:12But public cloud providers take a lot of that cost off our shoulders. By offering us the ability to pay for what we want to use on their service, we can simply spin up these Platform-as-a-Service NoSQL services and only pay for the amount that we need to use without worrying about the upfront cost of setting up these large data centers. So how do we compare and contrast relational data versus NoSQL? Let's just get right down into it and start talking about how these things are the same and how they're different. And I'll start by talking about core storage.

1:42In relational and NoSQL data, the core storage is actually, I would argue, exactly the same. It's some discrete data value. This is good news. Regardless of which data model you choose, at the end of the day, you're trying to do the same thing. And that is store some individual discrete value of data. Maybe it's an order date or a customer number or a latitude and longitude. Whatever it is, that's what you're trying to do. You want to store it, save it somewhere inside of your computer system with the ability to relatively easily retrieve it when you need to retrieve it to include it on a report or output it in some other fashion.

2:15So both of these different data models ultimately boil down to the same thing-- storing data. Oftentimes in a relational model, we call that core data value a field, whereas in NoSQL, it's frequently referred to as a value. But certainly, that vocabulary can change. It's at that point where these two models start to diverge or differentiate. Relational databases store the structure, the data structure, instead of tables. Those fields are gathered up into rows that are collected inside of a table. And a table is a collection of rows that are similar to one another, say, customers, right?

2:46Your customers table will have data values, fields, for all the different information that you collect about a customer. And all of those different customers will be on individual rows that are grouped collectively inside of the customers table. Your orders will be in a different table that will have all the different data values about orders. In a NoSQL model, you're looking more at things like, well, maybe keys or entities or documents, right? You can have these different values collected up and organized and structured in some different ad hoc ways and some different vocabulary that's used.

3:16So you can have something that looks a lot like a table. A document is oftentimes very similar to a relational table. And they can be somewhat compared to one another. But they're not the same thing. So we have a lot of divergence here. And one of the big ways that they diverge is in schema enforcement. On a relational database, this is strong. On our NoSQL database, really, it's very weak or, in fact, nonexistent. What do we mean by schema enforcement? With a relational data, our tables and our rows and our fields are very strongly defined. We will indicate for a given field what type of data it has to store, how big or how long that data can be, whether it's a date or a numeric field or a text field.

3:55There's going to be a lot of rules about the type of data that can go into that field. And the reason we do that is because if some data comes along that does not meet that requirement, well, the database system itself, the relational database system itself, will kick that data back and say, hey, that data is no good. It will bubble up an error, so to speak. And then in our overwriting system, whatever is collecting that data, we can then choose how we want to actually handle that error and what we need to do when that data comes in bad. Typically, that's not the case with a NoSQL database. With a NoSQL database, we can handle bad data values that come in because, well, the schema is not necessarily enforced.

4:29If someone types in some letters in a field that's meant to store the phone number, we can probably just take that in and store it inside of that particular entity. That particular value can have numbers in it-- or letters in it, rather, even though every other value for that entity has numbers in it because it's a phone number field. Same thing with the actual specific values themselves. What happens if your database does not have a specific field defined for middle initial or middle name and yet you get some data from a customer that has their middle name included? Well, in a relational database, there's nothing you can do. Your choices are to either stick that middle name in with the first name or in with the last name, or to just throw it out entirely and not store it.

5:05With a NoSQL database, you can choose to let that data value get inserted into that entity or into that document-- and only that document. It doesn't matter that you've got 500 customer documents in your table and not a single one of them except for this one has that middle initial. It's allowed. You can have that kind of flexibility, that loosey goosey answer to the data that's coming in. So a lot of times, for a lot of people, that sounds a lot more intuitive, a lot better. You're like, hey, I don't want to lose data if weird data comes in. I want to collect it. And that's a good reason, a good choice, for choosing NoSQL databases.

5:37But that also means that any data validation that you really rely on, you probably have to develop an overlay on top of the NoSQL database yourself. It's another piece of software that you have to write. It's more code that you have to maintain. It's more work that you have to do. As opposed to the relational database, if you know that you really want to enforce these strong rules on your data, the relational database is already doing that for you. Once you set up your database and establish that those are the rules for the data, errors will get bubbled up to you when the data does not meet those requirements. So there's pros and cons to both of these.

6:07But that's how the schema enforcement differs between relational and NoSQL databases. Relationships in a relational database, we always have relationships, specifically between our tables, and usually something called normalization. I cover normalization in another skill in another video. I would look that up here at CBT Nuggets and watch that if you're curious. Normalization is very cool. But it's really the rules about how the tables should relate to one another inside of a relational database. Inside of a NoSQL database, well, sometimes you have relationships, but not often, right?

6:37A key value pair doesn't really have any relationships in it whatsoever. A document database, yeah, there might be some relationships between the two different documents. But they're probably, like we said before, because the schema is not enforced, not going to be very strongly enforced. You may have them, you may not. You can, again, force relationships inside of NoSQL database. But you've got to write your own software. You've got to write your own layer to do that. With a relational database, those relationships are enforced for you by the actual database engine itself. Finally, ease of use-- I would consider most relational databases to be moderate to expert ease of use systems.

7:10There's not really an easy way to get up and running on a relational database. They to be fairly robust, complex, enterprise-level database systems. There are some smaller, easier to use databases, such as Microsoft Access. But even that has a bit of a learning curve as you come to understand how it should be used. NoSQL databases, on the other hand, cover the whole gamut. There are some NoSQL databases that are so easy to use and set up and understand that it can be one of the first database systems that you ever work with. But you can also have very robust, very expansive, difficult to use NoSQL systems.

7:41Think about Cassandra or HBase. These require a lot of technical knowledge to use appropriately and use correctly. So you can really get any kind of level of complexity that you want out of your NoSQL databases. Again, the takeaway here, you need to look at your different data collection, your data storage operations, and decide which of these is right for you. And be sure that you are paying attention to opportunities to mix and match when it makes sense. You don't need to store all of your data inside of a relational database. You don't need to store all of your data instead of a NoSQL database. Your data is going to fit different models at different points in time, both within your organization, as well as in the life cycle of that data.

8:16So pay attention to that. And make sure that you make the right choice. I hope this has been informative for you. And I'd like to thank you for viewing.

0:00[CHIMING MUSIC] It's a common complaint-- at least here in the United States, it's a common complaint-- about how slow our financial institutions are. You get a deposit made into your bank account, and it takes two or three business days before that's actually reflected inside of your bank account-- which is crazy to most of us, right? We live in this digital, connected age. We expect things to happen instantaneously. Well, I was fortunate enough-- or perhaps, unfortunate enough, depending on how you look at it-- to actually have worked for a bank. And I got to see a little bit about how the sausage was made.

0:32And the reason that it takes so long for something like your deposits to show up in your bank account is because banks still rely heavily on what we call batch data processing. So batch data processing is an older form of data processing, but is still widely used. And the key to what batch data processing is lies right there inside of the name. Batch data processing is all about processing data in a batch-- in a big lump sum, right? Think about if you-- I don't know, you make cookies in a batch. What do you do?

1:03Well, you make a whole bunch of dough, and then you put all the little cookies on the trays. And then, if you happen to have like a cool industrial oven, you put all of those cookies inside of your big oven at once. And you cook them all at once. A bakery would do this, for instance. So that they have nice, fresh cookies available for all of their customers, all day long. They cook a big batch of cookies. Well, batch data is the same thing. You are cooking, or processing, a big batch of data. And it's important to understand how to use batch data because, with the overwhelming prevalence of the cloud and big data, batch is, again, a really relevant model for processing data.

1:37It's really important to know how to do this. So first of all, when and how is batch data processing used? Well, the number one way you're going to see it is when you have to interface with older legacy systems. Like that example I just described at the bank, they still rely on the mainframe a lot. The mainframe is still a popular choice of system because it's so reliable, and because it handles such a large volume of data well. They don't really have a good way to stream or to do real-time data processing on the mainframe. That's not what the mainframe is built for. The mainframe is built to gather up all of its data into one big file-- one really big file-- and then turn it over and process it all at once.

2:13And then maybe wait 12 hours for data to all pile up again, and do another big processing-- process another batch of data. So if you're going to be interacting with any sort of older legacy systems you may very well run into the need to process your data in a batch format. Batch processing is also great for high volumes of data. Instantaneous, streaming data processing is great-- Processing all the data as it comes in, that's fine. But, look, if you've got a lot of users, if you have a lot of data, if you've got a ton of stuff coming in, you may not have the computing power available to handle all that data in real time, as it comes in.

2:46You may need to lump it up into batches and process it inside of batch windows, which I'll talk about in a second. So high volume in data can often indicate the need to look at batch data processing. You'll also run into batch data processing when you work on something called Extract, Transform, and Load. I'm not going to dig into this here, but ETL processes, as they're called, are all about taking data out of one database, performing some operation on it, and then loading it into another database. And it is almost, by definition, a batch data processing process. ETL is pretty much always using batch data processing.

3:18That's simply how it's laid out, and how it's designed, and how it works. And it's still a very common operation that a lot us do in a lot of systems, these days. And batch data processing is great for things like media manipulation. Maybe you need to add captioning-- closed captioning-- to a bunch of video files. Or you need to change your crop, or otherwise adjust a bunch of images. When CGI is made for movies, they do all batch data processing. They make the adjustments to their individual frames-- their individual scenes. They load those up into the cloud, and then they let them batch process and render each of those frames over a period of time during their batch window.

3:53And then the next day, or two days later, they look at what they got out. They look at how their changes are reflected inside of those frames. So media manipulation is a really common, frequent occurrence for batch data processing, as well. Now, as far as how you actually do batch data processing-- well, I think it's fairly straightforward. I think you probably understand it pretty well. You've got to load your data up into a batch, right? You've got to sit there, and wait, and let your data pile up. And you need to think about things like your batch window. The batch window is the window of time when it's appropriate to run your batch processing. So a lot of times, your batch window is going to be like, overnight.

4:25You say, oh you know what? No one's really using our system from midnight till 6:00 AM, so we will run our batch processes at that time-- during a period of lower usage of our system. That's our batch window. And maybe you're an international company, and your batch window's at a different time. But otherwise, you need to somehow identify what your batch window is. You've also got your batch size-- how big of a batch do you want to process? It's all well and great to say, hey, I've got a nice 6-hour window, here, where I can process my batch. But if your batch size is so large, that it's going to take eight hours to process, well, now you've got a new problem.

4:56So you've got to focus on your batch window and your batch size, which in turn feeds into the processing time. You also want to think about scheduling. That's, of course, very important for your batch processes as well. Are you going on a fixed schedule? Are you always processing at midnight every day? Or are you waiting until your batch reaches a certain size before you process it? So once it hits a certain threshold, now you know you need to process, and you perform that operation dynamically, depending on when that data reaches that particular point. Now, of course, batch processing does introduce its own challenges, its own questions, of course-- the number one being that sometimes you will have stale data in your downstream systems.

5:32So if data is piling up into a batch, and it's not going to get processed until midnight tonight, all the systems that rely on-- or they're expecting those changes to be processed and input into them, are going to be stale. They're going to be out of date until that batch process is actually run. For instance, the bank-- They were only running their batch once a night at midnight. That meant that the system would run its batch, send the data over to system B. Then system B would wait and not run its batch until that night. Hence, it would take two days for that deposit to be reflected inside of the bank account. It took time for each of these batch processes to occur.

6:03And in the meantime, the data downstream was stale. You also run into resource utilization. It can be expensive to process large volumes of data. I talked about CGI for Hollywood. That's a really expensive thing to do. It takes a lot of computing power to process those huge batches of CGI that they build. Obviously, they have the money to do it. Not everyone does, so you need to consider what the cost and the resource utilization implications are going to be if you batch process data. Otherwise, you still run into batch processing quite a bit in a lot of different ways, like I talked about before.

6:35So it's important to understand how it works. And it's important to recognize when it's appropriate to use it. I hope this has been informative for you. And I'd like to thank you for viewing.

0:00[MUSIC PLAYING] If batch data processing is the old-school way of processing data, then the new greatness is streaming data. Our interconnected world means that streaming data processing is a lot more viable, and a much more relevant option than it was in the past. I mean, after all, we didn't always have computers and devices that were interconnected on Wi-Fi and the internet 24/7. It used to be that devices were relatively standalone and relatively isolated from one another, and while there was some network communication, that communication was slow.

0:37It might be intermittent. Maybe you even had to dial in and connect to the network only for a temporary amount of time. So piling up data, waiting, and then processing it in a batch format made sense. Doesn't make sense anymore. Why not? Now we can process data in real time. We can stream it to where it needs to go as it comes in, and we can reduce the lag in processing data and getting the answers and the questions and the results out of that data by processing the stream of data, as opposed to waiting for it to come up in batch. I previously used the example of a batch cooking.

1:08Well, a stream is more like handling the drips of water as they come out of the faucet. You don't wait for the drips to fill up the sink before you do something about it. You're going to handle those drips as they come out, as they drop out. Think of stream data as the drip, drip, drip of data coming into your system. Now, it's all well and good that data streaming is a cool thing that we can now handle, but it does come with some additional complications. I would argue that handling streaming data is quite a bit more complicated than handling batch data. Those complications come in a lot of different forms. A lot of different reasons to be a little more cautious and a little more proactive about your structure and your rules around streaming data.

1:44For instance, streaming data is going to be dynamic and continuous. You might be getting it from a lot of different places, a lot of different sources. You have to be always ready to handle, to accept the incoming stream. The continuous part is particularly important. What happens if your receiving system-- your processing system-- is down for some reason? What happens to the data that's being generated and is expected to be processed during that time? Does it get queued up and wait somewhere, or does it get lost in the ether? Do you care, or do you not care? So that's a really important part. And of course, because it's dynamic, you don't necessarily always know what the structure or the format of that data is going to be.

2:17I mean, you may well have control over all these devices that are providing your data, but it's coming in in real time. If you're not doing any preprocessing or any assurances to assure that it meets the structures that you exactly expect, it might be a little more dynamic. Might be a little different, and a little more flexible than you're used to dealing with in a structured batch data file format. And that, of course, is the same thing that we're talking about here, when we talk about more diverse sources. Typically, streaming data is coming not just from a single device. It's coming in from lots of devices. Think IoT-- the Internet of Things.

2:48With IoT devices, you might have dozens, or hundreds, or even thousands of little sensors or devices out there all producing data at the same time, as well as lots of other different things-- people's phones. You might have a website that's building streaming data. All these different sources, logs, social feeds-- anything you can imagine might be producing streaming data, which means your processing needs to handle all these different sources. There are a lot of different types of data that might communicate in different ways, and might structure their data in different ways. You've also got to deal with and understand the lag that happens between data processing and what I'll call data finalization.

3:23And what I mean is with a batch data feed, well, you know when you have all the data, because you get all the data at a certain scheduled time, or when it reaches a certain point. That is the entire data set. As you're streaming data and processing data in real time, you don't necessarily know when you've gotten to the end of the data. Just because there's a small gap, or data wasn't sent over a short period of time, maybe there's still additional data to come. It just hasn't shown up yet. So it's a little more blurry. It's a little more difficult to identify where that demarcation line is. When are you done getting data and you need to consider it finalized?

3:55When is the end of the day of data, or the end of the group of data that you need to process, and does it even have an end? Maybe it doesn't. Maybe you're just constantly continually streaming data and analyzing it like you would with, say, logs. Streaming data is also very latency sensitive. And this is important. We get a lot of help from this data point with public cloud providers. They offer a lot more resources. They are much quicker and faster, and more resilient than our own data centers. But the simple fact of the matter is if are streaming data, you need to be very concerned about the latency of your responsiveness of your systems, because data is continually coming in.

4:26You're not waiting for a batch to be finished. You were just getting the data as it comes, and you've got to process it quickly, and move it along, and do what you need to do. So any sort of latency or intermittent breakups can be potentially very dangerous for your data stream. Now, typically when we talk about data streaming, we really look at it in two buckets-- two processes or models for handling streaming data-- and that's going to be a push queue or a pull queue. A push queue is kind of like that faucet that I just talked about. The systems, or the devices that are generating the data are pushing it out as they get it, right on time-- just bink, bink, bink.

5:00They're constantly throwing that data out there to the processing system. This would be something that's happening over a network communication, maybe a web server as the processing system. And it's sitting there with a public endpoint. And all of your devices are sending their data to that public endpoint as they receive it in real time, just getting thrown or pushed to the processing system. The other way-- the pull queue-- relies on an intermediary system, where the data is stored temporarily. So the data is put into the queue-- into the pull queue-- and the processing system then, on its own schedule, on its own terms, pulls those data pieces off of that queue one by one.

5:34This is something like the pub/sub model that you may have heard of. It's a little more complicated, but it gets you a little more robust and a little more predictable data processing options out of it. So when you're talking about setting up a streaming processing system, you want to understand if you're going to be pushing or pulling, and what systems you're going to rely on to do those things. So that's a little bit about how we process data in a stream, as opposed to in a batch. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[MUSIC PLAYING] When we're talking about all of this stuff, it probably makes sense to get down to a little bit of the practical matter of it and how does this apply to our relationship with data in the cloud. Well, when you roll out your cloud services, you need to start to think about what services on which cloud platform or which cloud provider you are actually going to take advantage of. And all the things that we've talked about here-- relational databases and streaming versus batch data and NoSQL-- it all ties into those decisions.

0:30And now I'm going to help you make that decision. So let's set a few baseline ground rules about the cloud first. I want to talk about three different ways that cloud providers offer their services to us. Now, not everything falls neatly into these buckets, but they are relatively good measures or metrics for how you can look at the different cloud services that you're going to run into. First, there's infrastructure as a service. I think of Infrastructure as a Service, or IaaS, as kind of the baseline, beginners' all-purpose introduction to the cloud. With infrastructure as a service, the cloud provider, whether it's Google or Azure or Amazon or whomever you're using, controls all of the physical components of what you are deploying.

1:11They are in charge of the building that the data center's in. They're in charge of the data center itself. All the networking equipment, all of the computers-- they install all of those bare-metal computers. They are in charge of almost no software. The only software that they're in charge of is the hypervisor that's running the VMs. At that point, it's handed off to us as the consumers of the cloud to install whatever operating system we want in these virtual machines, whatever patches we want installed in those operating systems, whichever versions of whichever software that we want. We call this infrastructure as a service. Now next in line after we go up a level or so from infrastructure as a service is what they call Platform as a Service, PaaS.

1:47And in the platform as a service model, the cloud provider takes over control of not just the underlying hardware, but also the underlying platform that's running on that hardware. So they are going to control the operating system that's installed on those virtual machines. They are going to control the memory management, the CPUs, how much patching levels or what security levels are on those operating systems. So there are some things that they will expose to us as the users of that platform. So we can make some choices from dropdown menus or using command-line interface options. To adjust some of those features, we can choose from a limited list of features.

2:21But really, the cloud provider's in charge of all of that. We are just spinning up instances of that platform in order to deploy our code. What we are deploying out there is our own customized code, usually something like JavaScript or Node or maybe Python. And those services are going to host that code and execute that code for us. But we don't have control over the underlying platform itself. Then the top level, the most abstract level, is what we call Software as a Service. This is where us as users-- we just take advantage of fully functional, fully running software. We don't really have to do much configuration, if any at all.

2:54Think about the way you might use something like Hotmail or Outlook.com, right? You just sign up for account, and you've got a full-fledged email client available for you to use. That is a software as a service. Office 365 in the cloud or Google Docs, right, all of these different Google applications that you might use in the cloud-- these are all software as a service options. Now, there's one additional one that I do want to mention here. And that comes a little bit in between the software and the platform as a service. It's kind of a little bit of both. And that is the database as a service.

3:24And you can probably guess why I think that's important. A lot of the database tools that you look at in the cloud are actually a database as a service. They're more like the platform as a service and software as a service. But really, they share some of the features of both of those things. So let's talk about what some of these options actually are when we're looking inside of the cloud. To begin with that exact database as a service level that I was talking about, you'll find relational and NoSQL database options available to you. And I'm just talking about three very popular cloud providers here-- Azure from Microsoft, AWS from Amazon, and Google from, well, Google.

3:57And they all offer something-- if you are using a different cloud provider, they probably have something in this realm as well, at least the larger ones do. And this is going to be where you simply spin up an instance or multiple instances of a specific server. If you go to use Azure SQL DB, you will be spinning up a relational SQL server database. Azure will control everything that underlies that platform. You don't have any control over the platform, the bare-metal hardware. You are spinning up individual Azure SQL databases. So I call this database as a service because the management happens at the database level.

4:30You spin up individual database instances. There are also NoSQL options available on all three of these platforms. On AWS, it's DynamoDB. On Google, its Datastore, which is being renamed to Firestore. And on Azure, it's something called Cosmos DB. These are all database-as-a-service NoSQL databases. So that's likely what you're going to run into. I think these are the most common database deployment options. Inside of the cloud, these are the ones that are going to be easiest for you to get your head around and understand when you want to move to cloud-based database providers.

5:00But there's other options. For instance, there is a software as a service storage model on all of these. Azure calls it Azure Storage, AWS calls it AWS S3, and Google calls it Google Cloud Storage. These are largely NoSQL providers. They are not relational database provider engines. They are NoSQL engines. Although they are fairly unique and fairly typical, they're somewhat object- or entity-focused. But they're really very interesting software-as-a-service NoSQL options, really great for storing large blob or binary data files. If you need to store a bunch of images or media files somewhere, these are great options, things that you might use.

5:36Azure your has one interesting, I think, somewhat unique service that's basically a relational platform as a service called the Azure Managed SQL Instance. This is relational. It is an SQL server database. It's not quite a database as a service because you do have some control, some managerial control over the actual instance itself, but not as much as you have in an infrastructure-as-a-service option. And then lastly, we do have that infrastructure as a service itself. And this is-- well, this is virtual machines. This is when you can simply spin up whatever operating system and whatever database software you want.

6:08That's why I've listed it as either relational or NoSQL because, really, you can choose any database software that you want. You have complete control over what software is running on those machines. So the infrastructure-as-a-service level is maybe where you migrate initially your on-premise database applications to before you deploy them or before you migrate them over to one of these more abstract, more heavily managed models. So all this together makes up some of the different cloud data store options available to us, both relational and NoSQL. There are also going to be options about data ingestion that will impact whether or not you choose to use streaming data options versus batch data options.

6:45Now, unfortunately, when we talk about stream processing versus batch processing, it's not nearly as straightforward and simple because, well, there's a lot of different options that play a lot of different roles in the workflow. Whereas relational versus NoSQL is really about which particular tool you're choosing to use, processing data is about an entire workflow. And there can be a lot of options. Take this example from the Azure documentation that I just pulled up. Notice all of the different tools that are available to us to use when we're batch processing. We're talking about data lake store, maybe Cosmos DB, maybe not.

7:16There's things like Apache Spark available to us inside of the Azure cloud. And from there, we might go into a HBase or SQL Data Warehouse, so a lot of different things that can be used at different levels. There's a lot of different examples available for both batch processing and stream processing. And unfortunately, it looks like this all over the place. If you look at, say, the documentation for AWS, it's just as complicated. The thing to remember is that each of these tools has their use cases and has their purpose. And if you're going to engage with them, you want to make sure that you read the documentation and understand that they are the appropriate solution for the problem that you are facing.

7:48So understand what challenges you have, understand whether or not you're using stream or batch processing, and how you need that data to be handled and processed on what schedule, and with what constraints. So this covers a little bit about how we can leverage these technologies inside of the cloud, just an introduction to the concepts. Obviously, we have a lot more to talk about in future skills. I hope this has been informative for you. And I'd like to thank you for viewing.