“Everything is a File”

Today, I have dived into this interesting UNIX command line environment. To start it is completely case sensitive so a command like “cd” which is used to change directory and “Cd” or “CD” are not the same. Other commands ive started looking at are “mkdir” for creating new directories, “ls” for listing what is within my current directory, the “who” command to see who is logged into the system. There is also the email system accessed by the “alpine” command with commands for working around it. And most interestingly the screen irc chat system that is always active and when I want to attach to it the “screen -r” command is used, then to detach I press “ctrl-a then d” These commands make sense more or less easy to understand but it may be a bit of a challenge to get used to working purely with commands.

Today, I got deeper into the syntax of commands and their uses in practice. First off it seems many commands are 2-3 letter versions of the English meaning of what they do, for example “mv” is for move and “rm” is for remove. The “ls” command for listing can have an argument added such as “ls -l” that shows all data on the listed items. Some new commands ive learned started to test are…

“cp” or copy, used as “cp source destination”

“mv” or move, used as “mv filename destination”

“rm” or remove, used as “rm filename”

“ln” or link, used to make a symbolic link to a file and used as “ln linkingdirectory linkname”

There is an important argument for the link command “ln -s” that makes it a soft link which is much like a shortcut, changing something in one effects the other. And since there is no rename command the move command can essentially do a rename by putting the new file name in instead of an existing directory. Lastly my favorite is the “man” command. This is used to get a manual of another command, used as “man commandname” and can have arguments to reference sections. It doesn't make that much sense to have to use the move command to rename a file but at least its fairly simple. remembering these command names isn't too hard but its the arguments that will take time to memorize.

Today, I worked with archives and compression of Unix. There is more to it than on windows but its not bad. using the “tar”, “gzip”, and “unzip” commands I was able to compress and decompress some files and it seemed fairly simple. Compression using Unix takes an extra step when compared to windows it must first be “tar” archived then “gzip” compressed. Using the “man” command I was able to see a comprehensive list of arguments that made this more clear to understand but I still want to get better at the combinations of arguments. There are a lot of good reasons to archive and compress data including ease of transfer and storing, but getting a complete grasp of all I can do with this in the Unix environment might take a little time.

Today, I got into the file systems hierarchy and options on files themselves of Unix. Im familiar with the windows hierarchy but this Unix one is fairly different. Pathnames are obviously important when navigating through files, the kind I have been mainly using up to date are “relative” pathnames. This is essentially typing “cd” and the next directory that is located within the current directory. Interestingly taking a look with the “ls -a” command there shows some additional directories that are usually unseen. These include the “.” and the “..” and they have special meanings for relative path movement. “.” indicates the current directory im in and “..” points to the directory that my current is located within. This is all great to know but if I want to move quickly to another path then I found using an “absolute” pathname is better. An example of this is “cd /usr/bin” and no matter where I was before ill be moved to this new location.

Besides this I also learned more about the information that “ls -l” shows me about files. The 10 symbols at the beginning of the file description represent the file type and permissions. For file types “d” is for directory and “-” is for ordinary files, there are a few more but these will be most common. As for permissions the first 3 bits are for the user that owns the file, the second 3 are the group that owns the file, and the last 3 are for everyone else. “w”,“r”,“x” in order of positions they can be in stand for.. write.. read.. and execute, There can be a special bit but im not in full understanding of it yet. Also if id like to modify any of these permissions I use the “chmod” command. This can be used in both symbolic method and in an octal version. 4,2,1 is the octal values for each position and the combined total for each triad is how it is input. I think I for the most part got the hang of everything tonight but I feel there is more to it that I need to know.

	root
         |
	home
	 |
      cbenne17         <---- an example of the hierarchy to my home folder and stuff within :)                                              
     /   |   \
    /    |    \
   /     |     \
 lab2   src    bin
       

Today, I got a good look into how file naming can make working with them difficult. This includes using spaces in the file names, a very common occurrence. Other characters that should not be used in file name include “&”, “#”, “$”, “!” “?”, “\”, and even quotes.. and that's just for starters. Ive learned some tricks to use these horribly named files and have put them to the test in todays exercises. proper use of the “*” is one method, its a wildcard for more of “any” character. In todays work I tested this method to open 3 files, for the first file named.. “just a simple file.txt” without the quotes, I just did “cat just*” and it knew the file I was looking to access. Same went for the other 2 files.. compress “this” file.data using.. gzip compress* and for.. one, 'two', three, “four” and again using the “*” I was able to open it using the.. cat one* setup and it was all good.

Now the next method I tried was with the “\” symbol.. its purpose is to escape the following character. Using this to remove.. ??? can you delete me ??.abc it was just a matter of placing the “\” symbols like this.. rm ???\ can\ you\ delete\ me\ ???.abc and it was removed. The other 2 were.. * * * watch out! * * * and it was done like this.. cat * * *\ watch\ out\!\ * * * and then.. $USER VALUE$ done like this.. cat \$\USER\ VALUE\$ and it works just like that.

Lastly the simple quotes method, by placing the file name in quotes it works very well. 3 more examples of how this worked.. `ls` * HI! * then to.. cat “`ls` * HI! *” for access, next.. ( parenthesis & other odd things ) then to.. cat “( parenthesis & other odd things )” and lately to change the file name.. change my\ name.file I used.. mv “change my\ name.file” newname.file and tada it changed.

There was a final “challenge” problem but it was fairly simple, it was to remove the file.. - challenge round - it wouldnt work with just using the command and one of my methods. I had to make it so it knew I meant from that directory so I used this command setup.. rm ./-\ challenge\ round\ - As a final note to bad naming if I begin to type a file name and press the tab button it will fill in a proper way to call the file. This is much easier than figuring out how to do it myself but it was a great exercise to learn how this all works. Now I wonder why people dont correct their most common mistake of using the space by using underscores instead.

Today, I did learned a few things.. starting with using the “cat” command, which I used a little in my last activity. The “cat” command is used to concatenate files and print on the standard output. It makes it simple to print out a files contents for me, and I think ill get a lot of use out of it. There are also the similar commands “head” that displays the first “n” lines of a file, and “tail” that displays the last “n” lines of a file, you can fill on the “n”. Very interestingly was a little test I did with the “tail” command and the “-f” argument. It allows for me to detect and auto print any new appended data to a file.. really cool stuff, It would be good for getting messages in real time. I see some potential in this!

Ive also got to use the vi text editor, well actually the vim to be exact. This is very different from nano and the others and I must say it is a bit confusing to me. However ive grasped a good amount of the concept behind it and why it can be better for certain things. The big thing is that it switch modes between “insert” and “command”. Insert is for text input with absolute usability. Command mode is for absolute (and kind of confusing) control. In the end I think ill understand enough of its ins and outs so that im comfortable with it.

Today, I messed with the idea of misnamed files, or rather files with the wrong file extensions. This is not unheard of, its less likely in a windows environment but in this unix system it can happen and probably does. The command “file” can be used to detect the contents of a file and attempt to decode what the file should be. Its very accurate and in the case of a compressed file it tells what style of compression and the former name of this file before compression. I got to really put this and my class knowledge to date in the puzzle challenge and I must say it was a challenge.

I took the misnamed abcd.txt and used the “file” command, this told me it was actually gzip compressed data and that it was a “tar” before that. I tried to unzip at first but it would not allow me. Eventually I decided to rename the file with the “mv” command so that it was able to be extracted. I had to repeat this to extract the archive version of the file, and eventually I got something named “making.waves” and from that unix.text… This final file was the true text file that gave me instructions to complete this assignment. I must say it was actually fun, even with some frustrations. I know that this exercise was useful not only for making sure a files are named correctly but also for proving to myself that im getting the hang of unix/linux.

Today, I really got into lot of new concepts and commands. First off “ls -a” reveals all of the “dot files” that are hidden. These include the .bash_history, this is a log of all commands ive ever used in order, with each numbered. Using a certain command directly from the history can be done by.. !history_number for example.. !524 to run the 524th saved command. Also using the up and down arrows allows for scrolling through the recent command history, it auto fills them to the command line for potential use.

This is a reminder or the tab concept that I briefly mentioned in an earlier log. Using tab for completions is an easy and very useful concept. This can be great for typing an obvious large command or even directory, but in those cases it works by filling in even the smallest parts before the tab is pressed.. kind of confusing to explain but an example is by typing /u and then hitting tab it will auto fill to /usr. It can keep going from there unless there are too many possibilities, in this case it will beep and if it fails again it will actually list all of those possibilities and you will have to be more specific.

Then I looked at environment variables and command aliases. A “$” followed by all caps is an environment variable. By using the “printenv” command I can get a list of all environment and their contents. $SHELL is the environment variable for the shell, mine is /bin/bash. Others are $MAIL and mines at /home/cbenne17/Maildir, and $PATH which contains paths to directories that ill automatically have access to. to easily view the contents of one of these variables the “echo” command is used, like.. echo $PATH will show me the paths. I can create some of these as well.. varname=“content” is the format, it will be seen as $varname.

Lately commands can have aliases, and some already do for me it seems. by typing the command “alias” I can get a listing of all current aliases. the “ls” command already was set as an alias to “ls –color=auto” and so it always has that effect. I can remove said alias with the “unalias” command, and remake or make a new alias by this format.. alias nameofalias=“utility -options –options”. All in all I everything I went over today makes sense to me but only to a certain degree. I see a lot of room learning the complexities of those variables and even meaningful aliases.

Today, I explored some of the social features of unix. Commands such as “write” “talk” and “ytalk” are used to interact with other users on the system. By using “write” and typing a users name a message connection can be made.. with the permission of the other user. Both users could potentially send messages back and forth using this method but its not the most efficient way I found to do it. But in this case I can proceed to type anything I want and send it to be seen by the other user right on his/her interface. This does tend to get messy and so if needed the “CTRL-L” command will wipe it nice and clean. Keeping in mind that messages can only be sent to a user that has messages set to allow. The “mesg” utility can turn on and off messages for your user account and so when its off someone trying to connect to you with “write” will see the error… “write: user has messages disabled.” Similarly if a user whom has messages off tries to send to a allowed user it will fail with the message… “write: write: you have write permission turned off.”

But for a more efficient and multiple-user chat setup.. “talk” or “ytalk” is best used. It is recommended to use “ytalk” because it has extra features and can still communicate with those using “talk”. To start a chat type “ytalk” and then the name of the user you want to start with. If this user accepts.. both will be taken to a different interface where live chat can begin. In this chat, potentially many users can all get together and anyone can send an invite to another user to join by pressing escape to access the menu and pressing “a”. This menu has many options but most interestingly is the option to open your unix shell right into your chat box. This all only works if users have messages set to “yes” otherwise.. “ytalk Error user@lab46.offbyone.lan refusing messages” will be all you get. I wish when i'm on the server more people are on too because I feel it would be fun to jump into a quick chat about something whenever it suites us.

Today, I toyed with a bunch of features, both old and new. For starters, ls can have an argument of a file name or part of a file name so to list all applicable files. Mixing this with wild card characters can allow for detailed searches in a directory. The ? replaces one character spot, “*” replaces 0 to any amount and stuff within [ ] will replace with those inside.

EX: ls ???? will list all with 4 characters within the directory.

EX: ls [cab]* will list all that start with c, a, or b, and with any amount of characters after.

EX: ls [abcd]*[rst] will list all combinations that start with a,b,c,or d, and end with r,s, or t, with any between

Some more quote play was experimented with today as well. Iv'e said before that when referencing a file using “ ” double quotes will allow me to use even a poorly named file. But there are some files that this still work quite work with. This includes files with expansion in the name such as $PATH which would expand to show the variables contents and not be literal. The fix for this is the ' ' single quotes, these are meant for completely literal file name reference. I tested this with a long file name with $PATH in it. The files name was “Long File $PATH for Shell Lab.text” and to use it I had to make sure to set it in single quotes and type it perfectly.. caps and all. typing cat 'Long File $PATH for Shell Lab.text' it works like a charm.

Lastly I looked at the I/O streams in unix, standard input, standard output, standard error. These streams can be redirected using symbols such as.. > for STDOUT 2> for STDERR < for STDIN. One can redirect to say a file using these, but be careful because it rewrites the file, to add on/append to the file » is used for STDOUT and 2» for STDERR. Using the | symbol one can pipe data and therefore create additional functionality. I tested this with the “wc” or word count utility. To count lines of the good old “motd” file I piped it into the “wc” with the lines argument “-l” and it looked like this.. cat /etc/motd | wc -l this resulted in 19 being displayed, telling me it counted 19 newline characters.

I did a little messing with the page utilities “more” and “less” that I piped the /bin long list into, and ill say that its true, “less is more than more” as I was able to see and scroll through the data showing all it can but moving one line at a time with much control for this “less” setup.. “more” on the other hand scrolled in bulk using the space bar and it cant go back, leaving little control and organization.

Today, was about webpages.. or rather basic webpages done in unix. I used the text editor “nano” for this process, mainly because it takes html tags and turns them blue. This web development process is very similar to using a notepad in windows. The html tags are used exactly the same, but that was to be expected as html is its own thing regardless of the platform. My experience in html is very limited so ill log a few details for future reference. All webpages must be between the <html> </html> tags. In most cases all tags have a start and end, with the end tag starting with a “/”. also important is the set of tags <body> </body> for the primary information. I only worked with a handful of these tags today but it was enough to modify some stuff.

For example.. to make a selection of letters/words “bold” the <strong> </strong> tags are used. Or to center any amount of text.. the <center> </center> tags work well. I actually worked with 2 different tags that can work solo (with no end tag), the line break (new line) which is <br /> and the horizontal line <hr />.

After a couple exercises I made a homepage of myself.. I included a picture that I linked from my facebook, talked a bit about myself, made a list of the top things im currently looking forward to this year (using the <ol> </ol> (ordered list) tags combined with the <li> </li> (bullet in list) tags), and pumped in some of my favorite quotes.. A success if you ask me. In any event I again got a taste of the html setup but this time using the unix system, very cool.

This Week, I have been annoyed and excited at the same time, I have dipped into Unix scripting. Im no stranger to this concept and to the structures of coding so that helped, but I must say that it took some work getting everything right. Scripts purpose are to execute commands to make a process happen, they tend to be made to make a process easier. But as for Unix a couple things should be mentioned. Anything that can be typed in the command-line can be part of a “shell script”. When I say shell I refer to what is actually processing and executing the commands that are within a script. In the case of our class we are using “bash”, and so to make sure bash is used every time a script is run, #!/bin/bash should be typed on the first line of all scripts that intend to use bash. Next mention is when a new script is created, its permissions only allow you to edit it. In order to fix this a quick “chmod” command is needed.. chmod 744 file.sh is what I tend to do.

Now to look at some of the code ive been using, “echo” displays to the screen and when used with “-n” can make a response to an echo'd question appear on the same line. To get input I used the simple but effective “read” command, and fed the data into a variable. The “let” command gives variables a numeric data type and allows mathematical calculations. And there are a ton of operators that can be used, this includes testing values with “< = >” symbols or even redirecting outputs with “»”.

Here are some of my works, starting out basic and leading into more complex:

There is more to be done but this a great start I believe when it comes to scripting in Unix…

Today, I created more questions than I answered… I got more into unix files themselves but this time device files. There are 2 main types of device files.. Character and Block. Ive found that there are many more character device files, the /dev folder is full of them. Some of the character device files I found were.. loop0, xvda1, xvda2… and some of the block device files were.. tty, null, port. The way to tell what kind of files they are is to do a “ls -l” command and look at the first letter of the file permissions part, there will be a “c” for character and a “b” for block.. fairly simple.

I did a little experimenting with these concepts and found some useful concepts. Firstly the “tty” command shows me what virtual device im connected in as (aswell as the folder its located within), and even when i'm logged into 2 interfaces at once each gets its own unique number. But most interestingly was when I played with the “mesg” command. When changing my ability to receive messages to no, I found that it actually changed the listed write permission off on my terminal device file itself. This means that although I myself cant force changes onto these files, I can use utilities to do some of these mods for me.

I also did a little messing with redirecting data output, when redirecting to “tty”.. I found that nothing changed, this is because it is the default location of standard input, output, and error. However when directing to “null” nothing appeared, this is because “null” sends anything it gets to oblivion.. never to be seen again. All in all I see uses for these concepts but don't see much practicality.

Today, Is my 23rd birthday.. and I also have a lot to log.. as I did a lot and learned a good amount. First ill start with the concept that the Unix system runs many users at once and is considered multitasking, each user is essentially a process. Each user can also run multiple processes, this ability to multitask within your terminal is called Job control. “ps” is a command that can be used to display all of your running processes. Then there is also “top”.. an auto updating list of processes being running resources on the entire system(with the highest use being on top), seemingly multiple users irssi sessions jump to the top here and there. The “ps” command can also be used to view all processes running on the entire Unix server, this is done with “ps -A”. A look at my running processes using “ps” below…

I took and compiled some code that I used to do some tests. I used “time” to count how long the program took to run its course and ran it into the output like this.. time ./count > output. This reveled 34.5 seconds in real time and 33.5 seconds in user time. Then I used the “&” operator to do something cool.. It allows me to send a task to run in the background. So my input looked like this.. time ./count > output & and while it worked on it I could do other things. The result was also a little faster with 33.2 seconds real and 33 seconds user. When I did this I checked the “top” command list of running processes that are using resources and I found that “count” program taking 99-100% of the processing power.

Next ill talk about 2 command writing concepts that I learned. First is the “;” which has the purpose to write multiple commands at the same time to be processed. And the other is using parentheses to group up multiple commands that are separated by semi-colons and make them into one recognized process by the machine, by doing so I can send many processes to the background with them counting as one.

In a task I was asked to do I created this command line… (sleep 8; ls /var > multitask.lab.txt) & and it worked great. The “sleep 8”, means for it to wait 8 seconds before running. “ls /var” was for it to list the contents of the “var” directory.. but I used the “>” to redirect the output to a file it will create called “multitask.lab.txt” and the “&” means to do it all behind the scenes. I also did an experiment with running “jobs” which can be viewed with the “jobs” command, that proved.. when jobs are suspended on my list I seemingly can not log out until I finish them. I used “ctrl-z” to suspend and I can use “fg” followed by the process number to bring it back. Lastly for processes I should mention.. The “kill” command. This command has many styles of use but with the “-9” argument.. it will kill a process instantly.. no questions asked. It also used “-15” to properly terminate a process, “-2” to interrupt, and “-1” to hangup (restart basically). Heres a picture of the “who” command with 2 instances of me online… right before I “kill”ed it… heheheh

To finish this day out I did some searching of specific processes using a combination of the “ps” command and the “grep” command. One example is to find the process owned by the user “statd”.. I make this command… “ps -A | grep statd” which took all processes and piped them into grep so it could find and output anything with “statd”. Similarly I did this with “init” to find its PID # using… “ps -A | grep init” and found it to be 8356. Lastly I checked to see who owns “cron” and what its PID # is… “ps -A | grep cron” and I found that the “root” owns this and its PID is 512.. Interesting. For a birthday of School work it was rather nice. I feel I covered a lot.

Today, I spent way more time messing with some concepts than I expected.. and unfortunately I am still having some trouble. I started by making a bot that I can use in chat channels of irssi. It can do all sorts of things like math or even search google. I named my bot “Bigboss” after the greatest soldier of the 20th century (From the amazing metal gear series), and it worked like a charm. I documented some of the process of obtaining the generic “phenny” bot on the class opus. I also wrote a script that checks to see if my bot is active and will start it up if its not on.

I was asked to read into “cron” and “crontab” which is for scheduling tasks to be executed. This is where I started to really get confused but after some time I came up with something that sort of worked. I essentially took the script I wrote to start my phenny bot and set it to run every 20 mins.

The “at” command was next and the most frustrating of all tonight. It is like “cron” but it is for running something once “at” a certain time, this can be very specific. I worked this by piping a mail to send into the “at” command, im not sure this is right but its my answer to this. I had it set to run 5 minutes from when I set the command.

A little more to mention is that “atrm” is used to remove an “at” job before it runs. but again I spent many hours on this and I must say i am exhausted. Ill look more into all of this but at a later date. Perhaps others will have some input to assist in this.

Today, I was taken back to some old C programming concepts, but in unix style. Mainly it was about compiling code, both the fast way and the long way. I started with some different types of source code and I had to compile each of them slightly differently. First was a simple C source file…

Next I compiled the higher level C++ file…

Then a much lower assembly language…

When looking at the types of files each one is, and how the different levels of programming differ.. I did a few “file” command checks…

A mention should be made that.. it seems like the lower level languages take up less space overall in their compiled form, even when they output the same thing.

I then took things step by step to create a final c file, to show what we all take for granted. The first step was “gcc -S helloC.c” outputting a file named helloC.s. Its now assembly and so the next step is “as -o hello.o helloC.s” outputting the resulting hello.o object file. For the final linking step I used a slightly different method than the earlier assembly code. I took a script called link.sh and gave it the object file as an argument. Last mention is “Makefile”s these are prebuilt files that give instructions within on how to built an output from a source file or multiple source files. They are usually used like this “make helloC.c”. I once covered these concepts in my C programming class last semester and its been a good refresher especially with the Unix touch.

Today, I got another refresher from C programming. This time with data types and capacities, being they can be slightly different between platforms its good to know how Unix sees them. The main data types are Char, short int, int, long int, long long int. Lets start with the “char” data type, which is 8 bits in size. This allows for 256 different values, or 0 to 255 in an unsigned setup. But if a sign is added (creating negative values) then the range is -128 to 127. Short int doubles this at 16 bits which is 65536 possible values.. far more than double the total values of a char. This is because of how binary works, the formula is 2^(number of bits), and so it increases exponentially. All versions of long int are 32 bits on unix it seems and therefore its max unsigned range is 0 to 4294967295.

That being said ill go into the details of what I did to toy with these.. first I had to create the dtypecode.c data file which I got off the study…

Then I compiled it with this easy compile code…

And the output to this was…

After that I made a couple mods to the source so that I could see into this a bit more. The first mod was to line b = 0, I changed it to the maximum positive signed short int value. The second change was altering y = b - 1 to y = b + 1. This program looked like…

The resulting output looks as shown below…

Lastly to show a bit more on max and min values possible in unix int data types I made one last program to show the top and bottom values of signed and unsigned long long ints. I feel like I might have done it wrong but it was not cooperating with me no matter how much I changed things around. In any event it seems to give a proper output even though I feel the actual program is far from what Id hoped to make it. So heres the C code…

And once compiled the result shows these top and bottom values…

Again I am versed in some of C programming however I feel rusty and I am also used to using Windows with C versus Unix and so this could be part of it. Still cool as always! :)

Today, was the beginning of a much in depth look at regular expressions. Well at the very least, I put a lot of practice into getting its basics down and I feel its working out well. The good old “grep” command is for the regular expressions and there is indeed much to it. I took a nice reference chart for future use…

I started using these with grep on a file called passwd. Just a few simple commands but it was a good test of some of the regex features.

Also here is a search in that file for anything that starts with my first and last initials.

Regex can be used to replace a bunch of words with other words too. This is done on the syntax :[address]s[/pattern/replacement/][options][count] And its easier than it looks. A big mention is to watch for special characters in replacing words. In these cases use a “\” to make it regular again.

:%s/<centre>/<center>/g :%s/<\/CENTRE>/<\/center>/g :%s/<b>/<strong>/g :%s/<\/b>/<\/strong>/g

This fixed up the file nicely and with each replacing around 40 words.. id say this words very well.

Lastly I did a bunch of other regex styles on a big file of many words, I searched and got counts of results (the count is from piping the results into “wc -l”.. Must say I think im getting a feel for this…

Tomorrow ill be doing a bit more with grep.. And I might be looking forward to it.

Today, I did some more with grep and its regex expressions. But mostly I went into detail of the “egrep” also took a peak at “fgrep”. The “egrep” means extended and its got a few more commands to it. I got another chart that shows the each of the extended options…

I tested the difference of egrep to grep on a file called pelopwar.txt which contained some repetitive writings. No screenshots this time.. but i was able to search and count occurrences of words in certain positions and styles. Also instead of piping I used the “-c” option to give a count… First with just grep..

grep -c 'coast' pelopwar.txt 9 grep -c '\<coast\>' pelopwar.txt 8 grep -c 'Dorian$' pelopwar.txt 1 grep -c '^Again' pelopwar.txt 2

And then the more advanced methods of “egrep”..

egrep -c '(^Athens | ^Athenians)' pelopwar.txt 2 egrep -c '(Corinth$ | Corinthians$)' pelopwar.txt 5

Lastly I was asked to do 2 different tests with grep. The first was to take the last command and mix it into grep to show the ones that start with my initials, but only the first 4.

Then there was the big and somewhat complicated final test. I took “last log”, sorted its results, used grep to only show results that start with e,f, or g.. and made it so only 4 results show…

A final mention to “fgrep” is that it only accepts literal strings for searches and is unable to do what these other versions can, but it is by far the fastest if that is whats necessary. But after 2 days spent with grep I can say I feel comfortable with its functions.

Today, and some time after, I did some Data Analysis with Regular Expressions and Scripting. And I had a lot of road blocks along the way. To be fair ive hit a lot of fatigue with the semester coming to a close and the stress might be a reason that things haven't been sinking in as well. I started off by copying the file fall2013-20130417.html.gz and gunzip'ed it to its html form. I then opening this file with the Vi text editor and did a little searching to check the format. By typing /CSIT 2044 I was able to find the entry for that class, same with /ENGL 1020 and /MATH 1230 with each showing me a very similar setup with the html tags and formatting.

To follow up on this I had to find a part of the data for that was only included in each entry so that I can grep all classes. “ddtitle” was used in every entry but not on anything else and so this is what I have chosen. With this in mind I can make a command that will be able to view all of this data, however it will still be very messy and hard to read. To fix this I need to bring up a new command ive been working with.. the “sed” command which can match parts of the data and either erase or replace these matched parts. This was my first command line for this.. and it did ok with a little mess left…

To get rid of the end tags I just needed to pipe into another “sed” and match the remaining junk data, with good use of regex its too easy..

Now the list is clean and ready for use. First thing I did was some searching, and then a couple number counts of things like.. Total classes offered.. Total CSCS classes.. Number of ENGL classes ranked 2000 and above.. and to see if ENGL or MATH offered more classes.. I used the “wc -l” to get the line count for my questions. Below are the results…

And it looks like MATH is the winner! From here I copied a handful of different semesters and years to my home and began a process of writing a script that can accept these as input.. as many as desired and make comparisons depending on how many were input. Now this is far from perfect but it does work and is fairly effective.. I put a lot of time into this but I feel I just couldnt squeeze all of the functionality I should have been able to.. perhaps its just a mental block. In any event here are the results…

It is functional but I had some greater plans for it that didnt quite work and with time not on my side this is were it got.. but hey its nice!

To round all of this out I went on to make regex commands to work with “sed” so that I can rearrange the data for better management. I needed 4 different commands that effect a different part and then combine them all into a large command to complete it. I spend some time on this too and I came up with 4 working different commands, but when combined didnt seem to work no matter what I tried to change.. maybe its just me but ill still show the data because it was indeed an admirable effort. Starting the the 4 replacement commands…

The first: cat output | sed 's/ - ' | less ..then the second: cat output | sed 's/[0-9]…. - g' | less ..followed by the third: cat output | sed 's/[A-Z]… [0-9]… - g' | less ..and the fourth: cat output | sed 's/ [0-9]..$g' | less

Each worked in their own rights but combined didnt quite do the job.

cat output | sed 's/^\( - \) - \([0-9]…. - \) - \([A-Z]… [0-9]… - \) - \([0-9]..$\)$/\2:\3-\4:\1/g' | less

But in the end I learned so much.. even though I had some struggles. This may have been the hardest yet but I wont let that get me down :)

Today, I mostly messed with the “dd” command, but did mess with a couple of other commands that ill mention. “dd” stands for data dump and it works better than “cp” and similar commands, as it is very fine grained and allows complete or partial copies to be made. However, because this process technically creates a new file and dumps data into it, the result is that the permissions are different. To start my tests with this, I used “dd” to copy a file completely to my home directory…

Both the old and new file have the exact same data size, and with some tests I can see just how similar they are… First a “file” check what types of files they seem to be..

How about an exception check of each file to see the output works the same.. but to do it “chmod” is needed because of “dd” creating a new file to contain the data.

Seems like they are the same in this aspect, but what about a “diff” check..

Looks clean, but now a new command can be mentioned, one that I think is cool. The “md5sum” is a great command for checking a files water mark to see if 2 files are truely the same. So by doing this it seems they indeed are the same, and different than “cp”..

So besides the permissions, these files are exactally alike.

Next I made my own zero filled file of the size 8kb..

I then did.. echo “more information” » test.file to appended “more information” to the end of the file, after the dummy data. This was to test using “dd” to take a certain amount of data from one file and place it into another. By skipping the dummy data and counting the real data, I was able to come take just the “more information” part. As a side note “bvi” and “hexedit” allow viewing of a data type file such as these with a count of bits… I lost the picture for this command but I do have another example of this.. I took a file called “data.file” and worked with it for a bit, I managed to extract data blocks from it, each sorta hidden within. So here is the command line…

And this data within is shown below using “bvi”..

This day of “dd” exploration was interesting and I might add that this is cool and something I feel windows has never really shown me.

Today, I mostly worked with the concept of filters, basically filtering output. Many of these concepts have been introduced before but ive got a chance to see them in action more. Ill start by talking about keyword filtering, a simple filter using “grep” that can have multiple “grep”s if desired. I worked with a file called sample.db that has a list of some students and information relating to them, separated in categories by colons. Below is some “grep” filtering..

This is great and all but the “cut” command also very handy for filtering. This allows for data that is separated by a specified symbol(like a colon) to have only certain columns visible. It is a bit strange but it works well and can be powerful. I did some experimenting with some echo'd data divided by colons and it worked nicely..

That last command line made 2 different columns appear but got stuck with a separator on the output. This can be fixed with “sed” to replace the colon with a space. I also had a little more practice with these concepts and replaced all lower-case t with upper-case T, and then made the period into an asterisk..

Altough im talking mainly about filters today I should make mention of something else I went over, “ASCII file line endings”. Basically this is what creates new-lines and they differ depending on your operating system. For Unix its a “line feed” which is represented by a “\n”, for Mac its a “carriage return”, and for DOS its both a “carriage return” shown as “\r”,and a “line feed” together as one done by “\r\n”. This makes them essentially incompatible but with the “tr” command I can translate one into another. Here are some examples of switching operating system text compatibility…

Mac to Unix and an attempt to view it..

Unix to DOS and an attempt to view it..

DOS to Mac and an attempt to view it..

Now back to filtering, after this was done I looked at some ways to filter the student data from before but by certain criteria, showing both output and then line count. I sorted the outputs to be alphabetical using “sort” to show no duplicates using a command called “uniq” with its “-u” argument, and I also removed the headings from the results..

All unique students..

All unique majors..

All unique “favorite candies” and removed all end asterisks..

And lastly another type of filtering using “head” and “tail” to only show the first so many lines or last so many lines respectively. These can be combined to get any amount of lines from any spot on the file. For my practice I used the file “pelopwar.txt” with some writings in it..

First 22 lines..

Last 4 lines..

Lines 32-48..

The first 4 of the last 12 lines..

And so this has been a lot of filtering techniques as well as some thought into the differences of operating systems. I feel like my understanding of Unix concepts has reached a point where I can say i have become fairly comfortable as a whole.

Today, was my final lesson in this Unix class, a little sad. Anyway I looked at security and groups access, much of which was interesting information but I did do a few activities to supplement some of those concepts. Ill start by recalling that all files and directories have permissions on 3 different levels of a hierarchy. These start with “user” then to “group” then lastly “other”. The most powerful owner of permissions is “root”, a super user that can do anything. Here are the permissions of my home directory..

I was digging around in the passwd file looking at permissions and “user id” and “group id” numbers to see how they relate. I found that the root was most interesting as it had a 0 for both ids, meaning a 0 permission id is the best..

Now to talk about permissions, when a regular user makes a directory it is defaulted to have the octal permissions of 777 which is everything for everyone. Similarly with making a file it starts with 666 for permissions which is everything except execute/search. This can be bad to give this much access each time and so the “umask” exists to change that default though an binary invert and ANDing process. This can also be done by hand by subtracting the octal values. By typing in the “umask” command I found that mine was set to 022.

By making a file I confirmed that my “umask” makes them into 644 permissions. To do more testing I switched my “umask” to 000, and this made it so files had their default masks. I then tried to make a file with 640 permissions which was easily done by changing “umask” to 026.

Talk of all of these numbers is worthless without a few examples of how the ANDing and Subtract method work..

First to take a directory and AND it with a “umask” of 077, so to do this the “umask” will be first inverted to 700 for ANDing..

     111 111 111
     111 000 000
     -----------
     111 000 000
     

Next to take another directory and AND it with the “umask” of 777, so to do this the “umask” inverts to 000..

     111 111 111
     000 000 000
     -----------
     000 000 000
     

The last AND came out all zeroes and because of this, nobody has permission.

Now for a standard subtraction “umask”..

    666
  - 226
    ---
    440
    

Very simple that way, now for 1 more..

    777
  - 047
    ---
    730
    

Cool stuff and it seems straight forward and easy to grasp.

As for the category “group”, its a second level security that can allow many access to files that they need but not those who have no use for them. The command “groups” can be used to show what groups your account is a member of. There is also the “id” command that will list them with their “GID” numbers..

The last thing I did was search the directory “groups” within the public Unix directory..

I went digging around the files that I could access, to find some secrete codes. Some of the files were within the “root” group and I had no way to see them. But I managed to open 3 files, 2 of which had codes inside. Ill now list them below…

1st: The mountain ate the dog

2nd: Up Up Down Down Left Right Left Right B A

As far as I know that was it, but in the end I also learned more about “SSH” (Secure Shell) and the importance of permissions to prevent viruses and hacking. I feel this has really been a good class for me. And so I have to thank Matt for this structured approach that has taught me a lot, especially for an online class. :)