00:00:00 - Okay, welcome to Nugget 103.4 where we're going
00:00:03 - to talk about using streams, pipes and redirects for, well, basically
00:00:08 - text that comes out of and goes into a standard program. So
00:00:11 - we're going to talk about three things. We're going to talk about standard input,
00:00:14 - standard output and standard error. Now, if that seems a little
00:00:18 - bit confusing, don't worry. We're going to explain those as well. But then we're going
00:00:22 - to talk about what we can do with those things. We're going to
00:00:24 - learn to pipe it using the pipe symbol which is on your keyboard.
00:00:28 - We're going to use the tee which is just like a plumber would
00:00:31 - use a tee.
00:00:33 - Well, I don't think a plumber would use a tee that I drew. But basically, it's
00:00:37 - where one thing goes in and several things come out. And then
00:00:41 - we're going to learn how to make the output of one command
00:00:44 - the input of another command using xargs. Alright, so before
00:00:48 - we do any of that, let's take a closer look at what standard
00:00:50 - input, standard output and standard error are all about.
00:00:55 - Alright, it's pretty simple. The standard input is the stuff a
00:00:59 - program chews up. For example, if it's a program that manipulates
00:01:03 - a bunch of texts, this is a way that you can send that bunch of text
00:01:06 - to it. Then the standard output is the stuff a program
00:01:10 - spits out. If you type ls, it's the results that it puts on the
00:01:13 - screen. So that's the standard output. And then standard error is
00:01:16 - something that you generally don't see unless something goes
00:01:19 - wrong. If something goes wrong or, or it wants to spit
00:01:22 - out an error, that is the way that it complains. It's like another
00:01:26 - output file. Generally, it goes right to the console, but you can
00:01:30 - redirect that separately from the standard output when there's
00:01:32 - not something that goes wrong. So let's look at those things
00:01:35 - in action.
00:01:36 - Alright, first let's start with standard input, alright. Now, you
00:01:40 - might remember the wc command is word count. But let's see
00:01:43 - what we have in here. We have file1, file2, file3 and they each have
00:01:47 - like one word or something in it, just a little bit of text in
00:01:49 - each file, all right. So let's do the word count command. We did learn
00:01:53 - that a nugget or two ago on this. Now, we see there's one word, one
00:01:58 - line, eight characters in file1.txt, alright. Word count will
00:02:02 - tell us what file it's working on so we could actually do star
00:02:07 - and remember file globbing, and it's going show us every one of
00:02:10 - them and then the total at the end, alright. So it tells us the,
00:02:14 - the characters, the lines, the,
00:02:17 - or I'm sorry, the lines, the words, the characters, and then the
00:02:20 - name of the file. Well, we could actually instead of giving it
00:02:23 - a file as a command, we could feed it standard input and it will work
00:02:28 - on whatever we, whatever we send it. So if we do the standard
00:02:31 - input command, a less than character, file1.txt, now
00:02:36 - what this is doing, the wc command isn't going to know
00:02:40 - that it's working on file1.txt. It's just gonna
00:02:43 - see in the standard input that it's getting the contents of that.
00:02:46 - So how does that look, how does that work differently? Well, the
00:02:49 - results are still the same. I mean it still has the same data,
00:02:52 - but again it doesn't know that it was file1. So it doesn't
00:02:55 - report any file name because it's just standard input. It's
00:02:59 - like that file1 was kind of like shoved into its mouth and
00:03:02 - it didn't even know what it was that, that it was working on, alright.
00:03:05 - So that's how you give something standard input. That's
00:03:09 - how you direct data into a program with standard input, with that
00:03:12 - less than sign. And it's a little bit confusing, but instead
00:03:16 - of an argument, it's just a way that you can feed some programs.
00:03:19 - Now, some programs don't accept data from the standard
00:03:23 - input, alright. So it's important to know that it doesn't work
00:03:26 - for all files, and we're going to learn later on what you do
00:03:29 - when those type of programs
00:03:31 - don't accept standard input, how you do that. But anyway, that's
00:03:34 - how standard input works.
00:03:36 - Alright, next I want to show you how standard output works.
00:03:39 - Let's clear the screen. I'll see what files we have in there. Now, we can
00:03:43 - redirect the standard output. See. what happened is we, we typed
00:03:47 - ls and the standard output was for it to say file1, file2
00:03:50 - file3 right on the console. Now, if we wanted to redirect
00:03:53 - the standard output into a file like ls.txt,
00:03:58 - what we're going to see here is, now there's two ways to
00:04:01 - write this. We can just do ls > means redirect standard
00:04:03 - output into this file, ls.txt. When we do that, we'll get
00:04:07 - no output on the screen. But if you look, now there's a file called ls.txt
00:04:12 - and inside there are going to be the files listed
00:04:16 - right there that it saw, alright, so that it redirects the
00:04:20 - output into, the standard output into there. But there, you can
00:04:24 - also do this, ls 1 > ls.txt. Alright,
00:04:30 - now what that means is one signifies standard output and the
00:04:34 - only reason you need to, need to know that is that ls 2
00:04:39 - greater than
00:04:41 - ls.txt. This is going to redirect something to, that is
00:04:46 - going to redirect the errors, but there were no errors. So
00:04:49 - right now if you look,
00:04:52 - that file is empty. It has zero length, ls.txt, because while
00:04:57 - it did show the, the standard output on the screen it
00:05:01 - redirected the error or standard error to this file but
00:05:05 - there were no errors. If we want to see something go into
00:05:08 - there, what we could do is say, ls bob. Now, there is no bob so
00:05:13 - it should give us an error. And if we redirect the error into
00:05:17 - ls.txt,
00:05:19 - nothing appeared on the screen. However, if we look inside
00:05:24 - that, again we used cat from a previous nugget as well, concatenate,
00:05:27 - it will type this out on the screen, if we look inside ls.txt,
00:05:31 - we'll see that it stored this error, okay. Now, it didn't
00:05:36 - come on the regular screen like ls did. It went into the
00:05:39 - standard error and we redirected that into a file. So again,
00:05:43 - you have, up here this is a, whoop, I didn't mean to do that.
00:05:48 - This shows you two different things. This one redirects the standard
00:05:52 - output when there's no errors, just whatever the program outputs
00:05:56 - into a file called ls.txt. We did that and it, it will
00:06:00 - put these contents just like up here into that file. And then
00:06:04 - ls 2 > is going to redirected it in, just the
00:06:08 - standard error into ls.txt which is what happened
00:06:11 - down here, alright. It's important to know that there's like two different avenues
00:06:15 - a program can output stuff. Either the standard output when when
00:06:18 - it does what it's supposed to and nothing's going wrong,
00:06:21 - and then it's almost like a different channel on TV that it sends
00:06:24 - the errors when something goes wrong. Now, by default they both
00:06:27 - go to the command or to the, to the console, right. So if we
00:06:31 - didn't redirect anything, they both go to the console. Ls goes right
00:06:34 - to the console,
00:06:37 - and the error, standard error goes right to the console, see.
00:06:40 - But we redirected those to files and you have to specify which
00:06:43 - you're redirecting, alright. So that's, that's how those two things
00:06:47 - work. I guess the only other thing that you want to know is that
00:06:52 - if you just use the >, it's going to overwrite
00:06:54 - the file every time. Let's say we wanted to keep a log or
00:06:57 - something, then you would want to
00:07:01 - do the greater than greater than (>>). And what that's going to do is
00:07:03 - append onto the end of ls.txt, alright.
00:07:07 - So if we do that a few times,
00:07:10 - now we'll see that it's going to append onto the end of ls.txt
00:07:14 - instead of writing over it every time. So see, it just keeps adding
00:07:18 - to the end if you do >>. And the same
00:07:21 - thing with.
00:07:23 - ls bob 2>> ls.txt.
00:07:27 - Now, what this is going to do, let's walk through
00:07:29 - this, ls bob, we know that bob doesn't exist, right? So we're expecting
00:07:33 - an error. Well, what it's doing is taking, redirecting the standard
00:07:37 - error. But instead of just one greater than sign, we have two
00:07:41 - so it's going to append it to the existing ls.txt.
00:07:44 - So we'll do that a couple times, three times. And now let's look at
00:07:49 - ls.txt, which is getting significantly big now.
00:07:53 - And you'll see it just like it was before, but then we added these errors onto the
00:07:56 - end. Now, one last thing I want to show you is how you can redirect,
00:08:02 - well, I'll show you two more things. One, alright, let's clear the screen.
00:08:05 - Let's get rid of our ls.txt. So we're back at the
00:08:08 - beginning. We just have these three files, right? Now, you can
00:08:12 - redirect these things separately, alright. So you can say,
00:08:16 - ls bob. We know that's going to create an error. And we're
00:08:21 - going to put the standard error into or the standard output.
00:08:27 - We're going to put that into results.txt and we're
00:08:32 - going to put the standard error into errors.txt, alright.
00:08:37 - So ls bob.
00:08:39 - And now nothing comes out because we're redirected both of these
00:08:41 - separately. But if we look,
00:08:44 - ls bob, now the results has zero file length, right,
00:08:48 - because there were no, there was nothing in standard output.
00:08:52 - But there was a lot in the standard error which we've redirected
00:08:56 - into errors.txt, alright. So what if
00:09:00 - we instead of saying ls bob
00:09:03 - we just did ls, right? So now this is going to take ls and
00:09:07 - it's going to redirect the standard output into
00:09:10 - results.txt. Now, we would expect that to actually return
00:09:14 - results and we would expect results to get stuff in it. And
00:09:16 - errors we would expect nothing to show up in there. So
00:09:19 - let's see.
00:09:22 - And that's exactly what we would expect to happen. So if we cat errors,
00:09:28 - yes, there's stuff in there. But remember that's from the first time
00:09:30 - we did it, okay. We didn't erase it afterwards. So that's the
00:09:33 - very first time we did ls bob. That's where it actually
00:09:38 - put it into the errors.txt.
00:09:42 - Now, if we looked in results, it's going to be those files again. So
00:09:46 - that's the standard output that was redirected, alright. One last thing
00:09:50 - and then we'll, we'll be done and you're going to be professionals
00:09:52 - on the whole standard output, standard error and standard input, alright.
00:09:57 - Okay, the one thing you can do is what if you want both standard
00:10:01 - output and standard error in one place, alright? Now, you could
00:10:05 - just redirect them both to the same place. But what you usually
00:10:07 - see is, now let's see. What do we have in here? We just have, okay,
00:10:12 - I've erased all the other stuff. We just have our original three files, okay. So let's
00:10:16 - say that we want to do an ls bob. Now, we know that's
00:10:21 - going to create an error. See, no such file or directory, alright.
00:10:24 - But let's say we want to redirect the standard output into
00:10:29 - output.txt. Now, again we're, I'll run it just to show you, step by step,
00:10:34 - okay. Nothing is going to be in output.txt because
00:10:38 - that was standard output that we redirected, and this is a
00:10:40 - standard error, alright. So output.txt is here zero length,
00:10:45 - nothing in it, alright. Let's get rid of it
00:10:49 - so that, you know, we just have these three files again. So what
00:10:51 - we're going to do is the same thing. We're going to do ls bob. We know that
00:10:54 - that's going to create output into standard error. We're going
00:10:58 - to redirect standard output into output.txt, but here is where
00:11:02 - we're going to get tricky. We're going to redirect standard error
00:11:07 - into standard output. So both of them end up in standard output, alright.
00:11:13 - It's kinda like we're just going to send all the errors into our
00:11:16 - standard output too so they all end up in that output.txt. Now,
00:11:20 - you can't just say standard error into one because what that's
00:11:24 - going to do is going to, going to create a file called 1 and
00:11:26 - that's what want. We wanted to go into standard output.
00:11:30 - So if you put &1, that tells it that, okay, you
00:11:34 - don't want me to create a file called 1. You want standard
00:11:38 - error redirected into standard output. So if we do this, what
00:11:42 - we're going to end up with
00:11:44 - is the standard error redirected into the same place as standard
00:11:48 - output. So here you should see
00:11:51 - output now
00:11:54 - has an error in it. Now, again we tried it up here and it had nothing
00:11:58 - in it, right, because the error, the standard error wasn't redirected there.
00:12:02 - Well, here it's standard output, it's output.txt, but
00:12:05 - we've redirected the standard error into standard output.
00:12:10 - I know that's a lot of confusing lingo. But hopefully, if you look through the
00:12:13 - examples, you'll see exactly what it is that's going on, alright.
00:12:16 - And that's how standard output, standard error work. Now, let's
00:12:20 - move on and talk about manipulating some, with pipes
00:12:24 - and xargs and stuff like that.
00:12:27 - Alright, the first thing we're going to look at is the pipe symbol.
00:12:30 - Now, if we do an ls, the output that we get is file1, file2,
00:12:33 - file3. We get a listing of the files in the directory.
00:12:36 - Now, what you can do is use this pipe character. This is
00:12:39 - generally right above the enter key on your keyboard. It's
00:12:42 - like shift back, yeah, shift backslash usually. But the pipe
00:12:46 - command will take the standard output from the program that you're
00:12:50 - using and put it as the standard input for whatever goes
00:12:55 - after the pipe. So one of the things that you'll commonly
00:12:58 - do is use grep
00:13:00 - and let's see. We're going to grep which searches for something
00:13:03 - in the, in the string that it's given. So we know that the output from
00:13:07 - ls, the standard output is going to be this list of files.
00:13:10 - So let's grep the number three. So what we should see now
00:13:15 - is that what it does, it took the standard output from ls,
00:13:18 - these three file names, and it searched for the number three
00:13:23 - and it found that in one line, the one called file3.txt.
00:13:26 - So what it's done again, it's kind of easy to visualize. You
00:13:31 - just take the output of this and pipe it into whatever is after the pipe symbol.
00:13:35 - And the downside is this doesn't work for every application.
00:13:40 - Not all applications accept standard input as a way to get
00:13:45 - information. Now, grep does thankfully because this is a really handy thing
00:13:49 - to do, you know. You can search for a big list of files or,
00:13:53 - you know all that kind of stuff. Pipe it into grep and search for what you're
00:13:55 - looking for. But some programs don't accept things on standard
00:14:00 - input, and that's where the next application we're going to
00:14:03 - look at comes in and that is xargs.
00:14:07 - So let's do a similar situation, but instead of using grep,
00:14:12 - let's try to use echo, alright. Now, we know how echo works. We say
00:14:16 - echo hello and it says hello. But let's say we wanted to echo what
00:14:22 - came out of ls. Well, if we just use the pipe, we would expect
00:14:27 - that if we did this, it would echo the list of file names. But
00:14:32 - it doesn't and that's because echo doesn't accept things on
00:14:36 - standard input. So basically, all it did was echo this blank, you know.
00:14:40 - We put nothing after it. It didn't accept anything on standard
00:14:42 - input. So it just printed a blank for us. So what would you want
00:14:47 - to do if you wanted to do that? Well, that's where the xargs
00:14:50 - command comes into play. So what you do is you do ls
00:14:54 - and you pipe, but this time you pipe into xargs, okay.
00:14:59 - And what this does, and then you do echo. So what this does, we're
00:15:03 - piping in standard output into xargs standard input this
00:15:08 - list of files. And then what xargs does is it runs whatever command
00:15:12 - you tell it to and then it, it kind of plunks in what it took on
00:15:16 - standard input as arguments for that command. So if we do this,
00:15:22 - sure enough we get the list of files just like out of the ls
00:15:25 - command, alright. So ls gave us this, you know, this output. We piped
00:15:29 - that into the xargs command and then xargs in turn ran
00:15:34 - echo with that as an argument and then it printed it out. So see the difference
00:15:38 - here. We just piped it directly into echo and it doesn't accept standard
00:15:41 - input. But xargs kind of does a nice little work around and it will
00:15:47 - put it as arguments at the end instead of trying to put it
00:15:49 - into echo, the echo program, the standard input. So again,
00:15:53 - this is kind of a confusing,
00:15:56 - you know, concept to wrap your mind around. But kind of look
00:15:58 - what happened and what the differences here and hopefully that
00:16:00 - will make sense because understanding standard input, standard
00:16:04 - output, standard error and then xargs and pipes; it's kind
00:16:07 - of important if you're going to work on the command line a
00:16:10 - lot. And hopefully this has cleared it up quite a bit. Now, there's
00:16:13 - one more thing to look at and then we're going to move
00:16:16 - on from a redirection.
00:16:18 - So in this instance what we want to do is we want to both
00:16:22 - display things on the console like we type ls and the standard
00:16:26 - output comes right here in the console. But we also want to log it to
00:16:30 - a file. Well, it's actually more difficult than it seems. You could
00:16:33 - run it twice. You could do, you know, ls and then ls
00:16:37 - and then redirect it into a file standard output. But what if
00:16:40 - you want to do both which is something that you would want to do
00:16:43 - if you're writing a program and you want to see what it's doing
00:16:45 - but also keep a log. And that's where the nice little program tee
00:16:49 - comes into place. So what we do is ls. Now, if we just redirected
00:16:53 - that into output.txt, we would see
00:16:59 - in output those files, right? Well, let's get rid of output
00:17:04 - and, but when we did that, see,
00:17:06 - there was no output on the screen, right? We didn't see the actual
00:17:10 - ls command because we redirected it. But if we were to do this and
00:17:14 - we pipe it,
00:17:16 - the standard output into the tee program, okay, so just like
00:17:20 - we piped into the xargs program, now we're going pipe the standard
00:17:25 - output into the tee program,
00:17:28 - and we'll make a file output.txt. Now remember we just
00:17:32 - deleted this. So there is no file called output.txt
00:17:35 - right now. But if we do this, it's going to do two things.
00:17:39 - It's going to show us the results of that ls command but also
00:17:46 - it's going to add that to a file, see. So it seems pretty simple,
00:17:52 - but in practice it's rough to do without this nice little command
00:17:55 - called tee which does, it both puts it into a file and displays
00:17:59 - it on the screen, alright. So that's tee in a nutshell and that pretty
00:18:03 - much finishes up our section on file and text redirection, alright.
00:18:08 - So let's go back over, make sure we covered anything and
00:18:10 - then we'll move on to the next nugget.
00:18:12 - Alright, so looking at the topics we need to cover in this nugget, we went over standard
00:18:17 - input, standard output, standard error in some depth quite
00:18:21 - a bit. Hopefully, you understand what those are. We talked
00:18:23 - about pipes and how that takes the output and makes it the
00:18:26 - input of another program. Again, the standard output and the standard
00:18:29 - input. Then tee, we actually just finished learning about tee which
00:18:33 - basically just takes it and sends it in two different directions. Again,
00:18:37 - forgive my graphics here. And right before that we learned about
00:18:40 - xargs which is kind of a neat program that just gets around
00:18:43 - those certain programs that don't accept commands on standard
00:18:47 - input so that you can just make them arguments on a command
00:18:50 - that otherwise wouldn't be able to do those. So this is how
00:18:53 - this works, and I hope that this has been informative for you.
00:18:57 - And I would like to thank you for watching.