======UNIX/Linux Fundamentals Journal======
====MONTH Day, YEAR====
__August 26, 2014__
First day of class. Time to become a mystical powerful hacker through the use of a black background coloured terminal.8-) Yay for listening to the amazing speeches of Matt all class and going over the syllabus.
__August 28, 2014__
Second day of class. Had fun using wall on everyone to mask myself as a hacker. Also did some advertising of CIST club. Matt went over some very basic commands and then talked about the unix philosophy.
**The Unix Philosophy**
- Ordered List ItemSmall is beautiful
- Ordered List ItemDo one thing, and do that one thing extremely well.
- Ordered List ItemEverything is a file (almost).
**3 Types of Files**
- Regular/Ordinary
- Directory(links) - metadata
- Special
**3 Tiers of Ownership**
- user
- group
- other (world)
Permissions
* - means no permissions
* x means execute or search
* w means write
* r means read
__September 2, 2014__
Third day to the most amazing class ever! I get really excited to come to this class for some reason.
Started out class talking about projects. There is a new notes project where Matt will pick 3 people to put class notes on an official notes page for each class. There will be an author, designer, and editor. I remember that Matt had talked about doing this last semester, but it never actually happened. I am very interested to try it out this semester. I'm not sure how this will be better than just putting our own notes on our opus. We will still have to take personal notes down, which I do on my opus, to have a record for the main notes page. It just seems a bit redundant, but I see why he would want it for if someone is absent or for "the web" to google.
Next we went over Alpine. This is the email service that Matt uses for lab46. To be honest, I don't expect myself to really ever use it. I would much rather just use my college gmail.
We also spent a large amount of class time allowing people to set up their mercurial repositories. I got a good refresher on how repository cloning works with mercurial. To this point in day, I'm not sure if I like git or mercurial better. I tend to like git more, because github is a wonderful web-side application for git repository hosting.
__September 2, 2014__
All directories in unix/linux are denoted with a forward slash "/"
__Relative Path__ - a path that works off of our current location
__Absolute Path__ - a path that works globally
A wild baconfish has appeared! I choose you Pikachu!
0 STDIN
1 STDOUT
2 STDERR
">" redirects STDOUT to file (writes)
">>" redirects STDOUT to file (append)
"2>" redirects STDERR to file (writes)
"2>>" redirects STDERR to file (append)
tail command views end of a file and adding -f means don't end - usefull for viewing live changes to a file or for log files
cat command is useful to view file contents in terminal view
here is an interesting command: "cat file 2> ~/err" this makes the STDERR go to a file in my directory named err
another interesting command: "cat file 2> /dev/null" this sends the STDERR to a null place lel
__September 2, 2014__
Started out class learning how to check our current status in the course :)
lab46:~$ status unix
ATTENDANCE
[UNIX] 238 240 245 247
[cclay] 238 240 245 247
OPUS
1:opus:week1entry
1:opus:week2entry
PROJECTS
1:titlecustomize:customized Opus title section
lab46:~$
Learned about the super nifty calendar with the cal command.
lab46:~$ cal 04 2000
April 2000
Su Mo Tu We Th Fr Sa
1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30
lab46:~$
It is a little strange that the only holiday available is easter lel.
Also learned about a cool command called date
lab46:~$ date -d 20140803 +"%j"
215
The above code shows the amount of seconds since time began until 2014-08-03
Lastly, the command pom can be used to find the moon cycles :D
lab46:~$ pom
The Moon is Waning Gibbous (99% of Full)
Cool little command is tty, which shows my current terminal number the :/dev/pts folder shows all of the terminals
When using the who command, + shows who has messages on and a . means they have been active in the last minute.
rhenggel + pts/48 2014-09-09 16:04 . 21292 (pod01.offbyone.lan)
cclay - pts/49 2014-09-09 16:05 . 21281 (172.16.200.195)
You can view the specific libraries that need to be in a specific place for a specific command with the ldd command:
lab46:/lib$ ldd /bin/ls
linux-vdso.so.1 (0x00007fffad5ab000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f90ede17000)
libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007f90edc0e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f90ed865000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f90ed5fc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f90ed3f8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f90ee046000)
libattr.so.1 => /lib/x86_64-linux-gnu/libattr.so.1 (0x00007f90ed1f2000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f90ecfd5000)
ls -l for file/directory information :D
LELELELELELEL
lab46:/usr/local/bin$ ptt "sup noob"
_
.-./*)
_/___/ `.
U U 'sup noob'
Finished class today with some awesome games
__September 11, 2014__
Going over some stuff about archives for the archive project. Mostly stuff I already know. One thing I didn't know that I learned is that Unix/Linux does not register files extensions. It is just part of the name of the file (some times you might see 2 extension like parts in the file name). Some programs though may use parts of the file name.
Started learning VI (were using VIM) (ed is command line based line editor - cant really "see" the text)
the idea of insert text and commands such as save being split up (idea created by Bill Jay)
Insert Mode:
i,I
a,A
o,O
esc = command mode
Navigation - h=left,j=down,k=up,l=left (you can prefix these commands with a number such as 80l - will move 80 spaces to left)
word - w=forward by word, b=backwards by word (W,B treat things based on only spaces while lowercase treat other types of grammar)
line - ^=beginning of line, $=end of line, G=go t a certain line like 3G - goes to line 3
x=delete
X=delete backwards
p,P=paste
dw,db,dW,dB - delete by word
d^, d$, dd -del by line
s=substitue character
cw,cb,cW,cB - change word
y = yank ex. yw, yb, yW, yB, y^, y$
u,U = undo
r = redo
/ = search - n = next match, N = prev match ex. /hi 3n
:w filename = save as
:w = save
:q = quit
:q! quit without saving
:wq = save and quit
__September 16, 2014__
Went over the new puzzles project. Then started a wrap up of vim.
More commands:
:means current line
:co is copy
:2,4co9 copys lines 2-4 to line 9 (and moves down line 9)
:3m7 moves line 3 to line 7
:r filename - reads in a file
:w filename - writes to a file
:x saves and quits
:3,9s/lab46/LAB46 - searches for lab46 in lines 3-9 and will replace with LAB46 - adding /g on the end will replace every occurence on each line, without it will only replace once per line
:s/[a-z]$/Z/g -will add to the end any lines that end with a lowercase letter a Z
:%s/./DEADBABY/g - will replace everything with dead babies (dead baby joke lel)
:set number - gives you line numbers (finally)
:set tabstop=4 - sets tab to only go 4 spaces instead of the default 8
set shiftwidth=4
syntax on
set cursorline
set smartindent
editing the .exrc file you can add config to set things like tab number every time
the vim specific config file is .vimrc
make sure these files are in the home directory
__September 16, 2014__
Doing the archive project.
I started out making some directories for my unix/linux projects and then for the archive project.
I copied both archive files into lab46:~/src/unix_linux/projects/archives$
Had to read up on the command "file" It's pretty cool that it puts the file against 3 different tests (filesystem tests, magic tests, and language tests).
I got these results back when I ran file on each archive:
archive1.zip: Zip archive data, at least v2.0 to extract
lab46:~/src/unix_linux/projects/archives/archive1$
archive2.tar.bz2: bzip2 compressed data, block size = 900k
lab46:~/src/unix_linux/projects/archives/archive2$
I figured out that I can use unzip for the first archive. Unzip has a lot of cool modifiers. One example is unzip data1 -x joe => extract all files except joe from zipfile data1.zip
When I ran unzip on archive one, I notices that some things were inflated, while others were extracted. This is interesting.
Archive: archive1.zip
inflating: image1.jpg
extracting: image2.gif
inflating: image3.png
extracting: image4.txt
I figured out that the ones that were inflated were compressed, hence they were originally deflated, while the extracted files were simple just put into the archive and you are taking them back out. Pretty cool :D
Next I figured out that I can use the command tar to unzip the bzip2 files. This explains why there was bzip2 and tar in the name of the archive. I have to use the x modifier of tar to unzip. Oh wait... tar x filename didn't work... Looks like I actually have to use tar xjf filename. x for extract, j for bzip2, and f to use the archive. I can also add the modifier v to display the names of files as they are extracted.
Next I connected to lab46 via FTP and downloaded the pictures to see if any were corrupt when I viewed them (too lazy to put them up on my lab46 website :P)
So it looks like in archive1, image2 and image4 files are empty. I deleted them using rm. Image 1 is a bunny and image3 is a whale. In Archive2, image1 is empty. I also noticed that the whale picture was either corrupted with black, or that was supposed to be water. Then I remembered there was a complete whale picture in archive1, so I assumed it was corrupted and deleted it. Lastly, even though it was a txt file, it still had ascii art of a horse in it so I kept it (plus I knew I needed 4 files for naming purposes as said in the directions :P)
Next, I assumed the directions meant to rename the files based on the size of the animals in the picture :P
bunny is smaller than a horse is smaller than an elephant is smaller than a whale. I figured out that the command I used to move files (mv) is also used to rename files. Interesting.
Now I made an archive called myarchive.tar for all of the good files from both archives and put them in the archive.I used the command tar cfv myarchive filename filename filename filename to do this. C to create an archive, f to use the archive, and v to output the file names when each file is added. Of course I forgot to put the file name in the first time I did it so I broke my smallest picture. I fixed it the the second time.
lab46:~/src/unix_linux/projects/archives$ tar cfv myarchive.tar smallest.jpg small.txt big.gif biggest.png
smallest.jpg
small.txt
big.gif
biggest.png
lab46:~/src/unix_linux/projects/archives$ ls
archive1 big.gif myarchive.tar smallest.jpg
archive2 biggest.png small.txt
I used tar vtf myarchive.tar to make sure everything was in there (also shows file permissions, sweet).
lab46:~/src/unix_linux/projects/archives$ tar tvf myarchive.tar
-rw-r----- cclay/lab46 132216 2014-09-17 21:52 smallest.jpg
-rw-r----- cclay/lab46 2579 2014-01-26 11:01 small.txt
-rw-r----- cclay/lab46 4895 2014-01-26 10:54 big.gif
-rw-r----- cclay/lab46 52592 2014-01-01 16:17 biggest.png
Looks good so now it is time to compress myarchive.tar.
I read the man page for gzip and saw that 9 is the best compression level, so I want to use -8 (the default is 6 and lowest/fastest is 1).
lab46:~/src/unix_linux/projects/archives$ gzip -n -8 myarchive.tar
lab46:~/src/unix_linux/projects/archives$ ls
archive1 big.gif myarchive.tar.gz smallest.jpg
archive2 biggest.png small.txt
Now all I need to do is submit this! WOot!
Reflection:
1. I suppose tar works the way it does for maximum optimization.
2. The benefit of separating archival and compression functionality is that you can then create
an archive for transporting a lot of files without compressing them. This would be good if you
are worried that compressing the files might wreck them. Also it allows you to easily compress
1 file vs. having to compress a whole archive to compress something.
__September 18, 2014__
Started clas today going over some cool math tricks for parties lel.
Cat analyzes data of a file and splits it up into 8 bits (for ascii) wont work for binaries because characters in binaries cause weird sequences to occur. Cat will try to read the binaries as ascii files.
cat -n filename will add line numbers
cat -e filename will show $ at end of all lines ($ means end of line)
od filename will take an octal dumb of the motd file
od -x filename gives hexadecimal dump
Endianess
-little endian
-big endian
Intel is little endian (every 2 bytes are flipped)
grep - globally search for regular expression and print
by default, grep prints matching lines to a provided pattern
ex. grep pattern filename
grep "the" /etc/motd
grep '\' /etc/motd
\< start of word
\> end of word
-i takes away case sensitivity
-o only displays the matching text
-n of course will display line numbers in the output
cat ouputs, while grep inputs for processing
can use od and grep with a pipe
ex od -x /etc/motd | grep '0a'
pipe wc for word count
od -x /etc/motd | grep -o '0a' | wc -l //will give the account of new line characters
__September 20, 2014__
Starting the puzzlebox project. Not exactly sure what to expect.
I copied the puzzle.txt file into my src. I used cat to view its contents and it was a binary file. I then used file to see what kind of file it was. Looks like it was an archive, so I used the tar utility to extract it. After doing this, I found a "making.waves" file. I looked in the file with vim, and it says begin 644 leaf_on_the_wind.wav and then a bunch of gibberish. Not exactly sure what to do at this point, but I know a .wav file is a sound/music file.
I then decided to check the file type of the making.waves file. That came back with: uuencoded or xxencoded, ASCII text
I googled uuencoded and xxencoded, and it seems that they are used for binary-text encoding and are used for email operations. The "begin 644 leaf_on_the_wind.wav" means that 644 are the permissions and the file that being encoded is leaf_on_the_wind.wav I think that I am going to have to use a decoder now to see what the other characters are.
I found a uuencode tool. I read up on the man, and I decided to try uuencode making.waves. Didn't seem to do much.
Now I feel stupid, I decided to try uudecode to see if it was an actual tool and it is. This time I tried uudecode making.waves. This created a leaf_on_the_wind.wav file. Looks like we are getting somewhere, which is good. When I look inside that file, it seems like more binary stuff, but I do see one line that says dinosaurs_with_hats.mp4. Not sure what that is about. I'm just going to check the filetype again of the .wav file.
Looks like the filetype is a zip archive. When I unzip it, it inflates dinosaurs_with_hats.mp4. I looked into that file, and it is another uuencoded file. You have got to be kidding me. I immediately imagine the scenario of infinite going through this cycle. I really hope I am not missing the big picture. I am going to try decoding this now.
Now I have a file named results. I look it says I unravelled the puzzle box. Woot lol. I thought there was going to be something more than this that would make it really complicated O.o Now I just need to do the steps for submitting it. Yay!
Remember: submit classname checks projects that were submitted and such
__September 23, 2014__
{{:opus:fall2014:cclay:pipe.png?400|}}
lab46:~/src$ cat /etc/motd|grep "the" | wc -l
3
lab46:~/src$ who > who.dat
lab46:~/src$ cat who.dat
NAME LINE TIME IDLE PID COMMENT
smeas + pts/3 2014-08-29 11:52 02:31 1921 (cpe-67-251-64-18:S.0)
dsaunde6 + pts/9 2014-09-19 15:59 . 30058 (pod01:S.1)
mgardne8 + pts/11 2014-09-10 19:38 old 19834 (mosh [19834])
dsaunde6 + pts/14 2014-09-19 16:03 16:01 30107 (pod01:S.2)
Find the number of lines in who.dat
lab46:~/src$ cat who.dat | wc -l
57
break the file into 3 sections
lab46:~/src$ head -19 who.dat > section1.dat
lab46:~/src$ tail -19 who.dat > section3.dat
lab46:~/src$ head -38 who.dat | tail -19 > section2.dat
lab46:~/src$ cat section2.dat
putting all the files together and sorting them alphabetically
lab46:~/src/unix_linux/classwork/9_23_14$ cat section2.dat section1.dat section3.dat > who.all
lab46:~/src/unix_linux/classwork/9_23_14$ cat section2.dat section1.dat section3.dat | sort | less
lets find my username bin for my password
use getent to display entries in a database - queries the system for everything it has access too (local and remote), because our user accounts are not defined on lab46, but on an LDAP system.
lab46:~/src$ getent passwd | grep "$USER"
cclay:x:5798:5000:Casper Clay:/home/cclay:/bin/bash
passwords now are in etc/shadow instead of the password file. This is because people could get the encrypted passwords and crack them. Now only the system has access to them.
bc -l opens a handy dandy calculator
change base to hexadecimal obase=16
use the groups command to see what group you are in (as you can see im in 3 groups)
lab46:~/src/unix_linux/classwork/9_23_14$ groups
lab46 unix cclay
see everyone who is in the unix group
lab46:~/src/unix_linux/classwork/9_23_14$ getent group | grep 'unix'
unix:*:1730:abuck4,acarson1,ahoover3,avolino,cclay,ddewert1,dsaunde6,dshadeck,dsoutha3,jcliteur,jjacobs7,mgleaso7,mp010784,mquesad1,nsano,nvitull1,rhenggel,skramer1,ssmit133,stiwari1,tarnold4,tmosgrov,vgarfiel,wedge
__September 25, 2014__
Starting class today talking about how some kid who fork bombed lab46 un-intentionally.
Then we started talking about the recent bash vulnerability that has surfaced.
Just a reminder that ps creates a list of processes.
Flashback to the hilarious Vogue website easter egg :P
{{:opus:fall2014:cclay:vogue.png?400|}}
Continuation of the fun with getent, pipes, grep, tail, head, and now cut :D
Cut allows us to remove sections from each line of files :D
teames3:x:5944:5000:Timothy S. Eames:/home/teames3:/bin/bash
jfenchel:x:5945:5000:Jonathan South:/home/jfenchel:/bin/bash
lab46:~/src/unix_linux/classwork$ getent passwd | cut -d':' -f1,7
root:/bin/bash
daemon:/usr/sbin/nologin
filtering out the above to things with 'th' in them, and then taking the last 16 results. In this case we will see there are not actually 16 results to be displayed.
lab46:~/src/unix_linux/classwork$ getent passwd | cut -d':' -f1,7 | grep 'th' | tail -n 16
appelthp:/bin/bash
ssmith85:/bin/bash
rthatch2:/bin/bash
thakes3:/bin/bash
mhether2:/bin/bash
gerth:/bin/bash
bkathan:/bin/bash
dsoutha3:/bin/bash
athoma13:/bin/bash
lab46:~/src/unix_linux/classwork$
We can count 9 names, or count the number of matches with word count.
lab46:~/src/unix_linux/classwork$ getent passwd | cut -d':' -f1,7 | grep 'th' | tail -n 16 | wc -l
9
Quotes:
^ Quote Char ^ Quote Name ^ Meaning ^
| ' ex. 'test' | full quote | literal quote |
| " ex. "test" | half quote | Allow for expansion (variable)|
| ` ex. `test` | backquote, backtick | command expansion |
| ( ex. (test) | backquote, backtick | command expansion |
Quote examples:
lab46:~/src/unix_linux/classwork$ echo $USER
cclay
lab46:~/src/unix_linux/classwork$ echo "$USER"
cclay
lab46:~/src/unix_linux/classwork$ echo '$USER'
$USER
lab46:~/src/unix_linux/classwork$ echo "${USER}, you are logged in `who|grep "$USER"|wc -l` times"
cclay, you are logged in 1 times
lab46:~/src/unix_linux/classwork$
lab46:~/src/unix_linux/classwork$ uol=$(/usr/bin/who|wc -l)
lab46:~/src/unix_linux/classwork$ echo $uol
37
lab46:~/src/unix_linux/classwork$ t=`echo "There are $uol users on the system"` lab46:~/src/unix_linux/classwork$ echo $t
There are 37 users on the system
Semi colons aka ; will separate commands on the same line
__September 30, 2014__
Started class out today talking about puzzlebox2 and the new project. Then had a lecture on problem solving. If you are stuck, manipulate the unix world in a way where it works for what you need it for.
Need to look-up the command uniq for the new project.
Also will need the command tr, which will get rid of specified characters.
Example:
lab46:~/src/unix_linux/projects/puzzlebox2$ printf "\tHello\tthere\n" | tr '\t' ' '
Hello there
This got rid of the spaces.
You can use man man to view the different sections of the man file.
Use man ascii to see a complete ascii table.
Wildcards are like a watered down version of regular expressions. Regular expressions are used within files, while wildcards are used in the command line.
Wildcards:
^ Wildcard Char ^ Meaning ^
| ? | matches any single character |
| * | match 0 or more of any character |
| [] | match any one of enclosed characters |
| [%%^%%] | do not match any of enclosed chars |
Examples:
ls ???????
ls l*
ls *.zip
ls *.tar*
ls ?[aeiou]?
ls c[aeiou]*t
ls c[^aeiou]*t
__October 2, 2014__
Need to make sure I'm subscribed to the class mailing list.
Practicing commands today with wildcards. Went into usr/bin to count files :D
Shows the number of files with 5 or more characters in the name:
lab46:/usr/bin$ ls -d ?????* | wc -l
678
shows the number of files that have at least 4 or more characters, and whose first, second, and last character is oneof the jeopardy characters.
lab46:/usr/bin$ ls -d [rstlne][rstlne]?*[rstlne] | wc -l
43
All files whos names do not have vowels in them.
lab46:/usr/bin$ ls -d [^aeiouy][^aeiouy][^aeiouy] | wc -l
27
Looking for the number of all directories and symbolic links
lab46:/usr/bin$ ls -l|egrep '^(l|d)' | wc -l
203
__October 7, 2014__
Started out class talking about the dataproc assignment. Due date changing to the 22nd; however, if we want a bonus point, we can still turn it in by the original due date.
A **__process__** is a "**program in action.**"
Each **process** has a **__PID__** or **__ProcessID__**.
Running the command **ps** lists the current processes running and their PID's, as-well as other information regarding each process running.
In this example, we can see that when we run the command cat and let it sit, it can then be viewed by running the process ps in a different terminal.
{{:opus:fall2014:cclay:capture.png?1000|}}
We can kill processes with the kill command. Here are the different kill signals available.
lab46:~$ kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR
31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3
38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7
58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2
63) SIGRTMAX-1 64) SIGRTMAX
The following can be used to close the process cat with the appropriate PID:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
cclay 24925 0.0 0.1 14048 2048 pts/10 Ss 16:46 0:00 -bash
cclay 24933 0.4 0.1 14048 2044 pts/107 Ss 16:46 0:00 -bash
cclay 24941 1.0 0.0 6424 348 pts/10 S+ 16:46 0:00 cat -
cclay 24942 0.0 0.0 11332 1076 pts/107 R+ 16:46 0:00 ps u
lab46:~$ kill -1 24941
lab46:~$ kill hup 24941
lab46:~$ kill sigup 24941
If we kill a process by simply saying "kill PID", the kill process automatically uses signal "15" or "SIGTERM"
In the kill command, signal "9" or "SIGKILL" is the most powerful signal and will close anything. It should be used sparingly, as it can cause problems if not fully understood.
Running "ps aux", we can view every process running on the system.
Here is an example of viewing all of a single persons processes on the system.
lab46:~$ ps aux | grep $USER
root 24922 0.0 0.2 99900 4520 ? Ss 16:46 0:00 sshd: cclay [priv]
cclay 24924 0.0 0.1 99900 2152 ? S 16:46 0:00 sshd: cclay@pts/10
cclay 24925 0.0 0.1 14056 2068 pts/10 Ss 16:46 0:00 -bash
root 24930 0.0 0.2 99900 4524 ? Ss 16:46 0:00 sshd: cclay [priv]
cclay 24932 0.0 0.1 99900 2152 ? S 16:46 0:00 sshd: cclay@pts/107
cclay 24933 0.0 0.1 14056 2096 pts/107 Ss 16:46 0:00 -bash
cclay 24953 0.0 0.0 6424 344 pts/10 S+ 16:47 0:00 cat -
cclay 26488 0.0 0.0 11332 1064 pts/107 R+ 16:57 0:00 ps u aux
cclay 26489 0.0 0.0 15228 948 pts/107 S+ 16:57 0:00 grep cclay
lab46:~$
Another interesting command is **__top__**, which will show realtime system information such as process and resource usage.
Example(Only the first 12 lines of output):
top - 16:58:09 up 12 days, 17:52, 49 users, load average: 1.39, 0.57, 0.33
Tasks: 408 total, 4 running, 399 sleeping, 5 stopped, 0 zombie
%Cpu(s): 2.1 us, 7.9 sy, 0.0 ni, 75.6 id, 0.0 wa, 0.0 hi, 0.3 si, 14.1 st
KiB Mem: 1535680 total, 1499612 used, 36068 free, 398056 buffers
KiB Swap: 131068 total, 37104 used, 93964 free. 524684 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31518 thakes3 20 0 120988 2484 1364 S 23.3 0.2 3:07.53 irssi
28580 root 20 0 0 0 0 S 4.3 0.0 2:08.08 kworker/0:1
7 root 20 0 0 0 0 R 3.6 0.0 33:35.86 rcu_sched
26493 jjacobs7 20 0 16044 1700 1052 S 2.6 0.1 0:00.52 top
26549 wedge 20 0 9908 1740 1348 S 2.3 0.1 0:00.07 bash
You can customize the top process by toggling keys such as "z" for color, or 1 for specific cpu core usage.
Example:{{:opus:fall2014:cclay:topex.png?500|}}
In unix we use shells. The default shell that we use is Bash.
(View the latest bash vulnerability here: [[http://en.wikipedia.org/wiki/Shellshock_(software_bug)]]
AT&T UNIX VS. BSD UNIX SHELLS
^Shells ^ Meaning ^
| AT&T- sh | bourne shell |
| AT&T- bash | bourne again shell |
| BSD- csh | C shell |
When running a process in a terminal, we can save it rather than killing it by using CTRL Z
Example:
lab46:~$ cat -
^Z
[1]+ Stopped cat -
lab46:~$
We can view these stopped or active process with the command **jobs**. We can use the command **bg** to run processes in the background and the command **fg** to run processes in the foreground.
Example of freezing, viewing, and re-running processes:
lab46:~$ sleep 3600
(sleep 180;echo hi)
^Z
[2]+ Stopped sleep 3600
lab46:~$ jobs
[1]- Stopped cat -
[2]+ Stopped sleep 3600
lab46:~$ bg 1
[1]- cat - &
lab46:~$ jobs
[1]+ Stopped cat -
[2]- Stopped sleep 3600
lab46:~$
bg 2
[2]- sleep 3600 &
lab46:~$ jobs
[1]+ Stopped cat -
[2]- Running sleep 3600 &
lab46:~$ fg 2
sleep 3600
__October 9, 2014__
We can view hidden files by issuing ls -a , which will show all files.
Lots of these hidden files such as .bashrc are configuration files.
A useful command we can use is alias.
Here is an example so that when we type bob, we get bob printed on our terminal.
lab46:~$ alias bob='echo bob'
lab46:~$ bob
bob
An alias will stay until you end your session. You can take aliases away with the unalias command. If you want permanent aliases, you can add them in .bashrc in our home directory.
The environment variable $PATH shows all of the paths that the system will look for commands. If we type jim, it will look for a command called jim in all of the following places:
lab46:~$ $PATH
-bash: /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games: No such file or directory
Started going over scripting!
__Shell Scripts__
we can save shell scripts in files ending in .sh or .bash
Here is an example of a simple script called myscript.sh 3 line script:
df
who
ls
Now lets run this script (not all output included)!
lab46:~/src/unix_linux/classwork/10_9_14$ bash myscript.sh
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 3997376 1784952 1986328 48% /
udev 10240 0 10240 0% /dev
tmpfs 153568 160 153408 1% /run
We are not able to run this script with ./ because we don't have the proper permissions to execute it, which is why we used bash. If we changed the permissions with chmod, we will be able to then execute the script normally.
Example (not all output included):
lab46:~/src/unix_linux/classwork/10_9_14$ chmod 700 myscript.sh
lab46:~/src/unix_linux/classwork/10_9_14$ ./myscript.sh
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 3997376 1784968 1986312 48% /
udev 10240 0 10240 0% /dev
tmpfs 153568 160 153408 1% /run
To see our environment, which is bash: #!/bin/bash
Now lets make another cool script called nametest.sh to create a mini program that asks for your name and spits it back out :D Here is the script:
echo -n "What is your name? "
read name
echo "Hello, $name, nice to meet you! "
exit 0
Now lets run it!
lab46:~/src/unix_linux/classwork/10_9_14$ ./nametest.sh
What is your name? Casper
Hello, Casper, nice to meet you!
lab46:~/src/unix_linux/classwork/10_9_14$
Bash is a higher level among programming languages. We don't have to worry about things such as data types.
{{:opus:fall2014:cclay:proglang.png?600|}}
We can use tab completion (AKA push tab after typing in an in-complete name) to complete unique names such as file names without typing it all out. If have two files with similar names, tab completion will complete the name up until the point at which it is still unique. If you have two files with similar names such as hhhhhs and hhhs, you can push tab twice to get information on the files, and see why it wont complete the entire name.
Example:
lab46:~/src/unix_linux/classwork/10_9_14$ ls
hasdkhfakjhfafqhfiufubflks hasdkhfakjhfafqhfiufubflksajfasjfdsafjffaoiehfiquhfahf myscript.sh nametest.sh
lab46:~/src/unix_linux/classwork/10_9_14$ hash
hasdkhfakjhfafqhfiufubflks/
hasdkhfakjhfafqhfiufubflksajfasjfdsafjffaoiehfiquhfahf/
myscript.sh
nametest.sh
lab46:~/src/unix_linux/classwork/10_9_14$ hash
We can use the **rm** command to remove files and directories. Using **rm -f** deletes a director and everything in it.
Now lets test checking conditions inside our scripts and using loops. It is important to remember that things must be properly spaced such as 5 = 0. If it was 5=0, the script will try to assign 5 to 0 instead of checking the condition.
Here is a script called numbertest.sh (we used -eq for = and -gt for > to avoid string vs number comparison problems):
{{:opus:fall2014:cclay:numtest.png?500|}}
Now lets run this amazing game we just made!
lab46:~/src/unix_linux/classwork/10_9_14$ ./numbertest.sh
Pick a number: 1
too low
Do you want to play a again?y
Pick a number: 7
too high
Do you want to play a again?y
Pick a number: 5
correct
Do you want to play a again?n
lab46:~/src/unix_linux/classwork/10_9_14$
__October 21, 2014__
Starting class today talking about random numbers to improve our number guessing game! Going to have to leave early today for a Dentist appointment :(
We can print a random number with the command echo $RANDOM
lab46:~/public_html/Lab09$ echo $RANDOM
2944
lab46:~/public_html/Lab09$ echo $RANDOM
15863
lab46:~/public_html/Lab09$ echo $RANDOM
9721
% modulus
/ division
34%5 = 4
34/5 = 6
__October 23, 2014__
We can use bc to easily do math in scripts.
lab46:~/src/unix_linux/classwork/10_23_14$ echo "2+2" | bc
4
Went over loops.
Now need to make a grade calculator for the course for our next project. Started working on that in class.
__October 23, 2014__
We can use floating points in bash with the -l argument of bc. If we tried to compute 6/7, we would get 0.
Instead, we can try the following command:
lab46:~$ echo "(6/7)*100" | bc -l
85.71428571428571428500
To get only an integer, we can do the following with what we already know:
lab46:~$ echo "(6/7)*100" | bc -l | cut -d'.' -f1
85
To get the first two decimals, the following can be done:
lab46:~$ echo "(6/7)*100" | bc -l | cut -c1-5
85.71
To finally add the percent sign, we can do this:
lab46:~$ echo "$(echo "(6/7)*100" | bc -l | cut -c1-5)%"
85.71%
For the grade calculator project, we can calculate the attendance by splitting up the dates. Currently, they are all on one line. Lets split them up onto their own lines using translate, and white space as a delimiter.
lab46:~$ status unix | grep $USER | tr " " "\n"
[cclay]
238
240
245
247
252
254
259
261
266
268
273
275
280
282
294
296
__October 30, 2014__
Knowledge Assessment! Wasn't so bad!
__November 4, 2014__
Today we venture off into regular expressions!
*Remember - Wildcards are for files, regular expressions are for text *
Regular Expressions are used to describe patters, patterns in text that we are looking for.
^ Regular Expressions ^ Meaning ^ Extended Expressions (newer)? |
| . | matches any single character | No |
| * | match 0 or more of the previous | No |
| [] | match any one of enclosed characters | No |
| [%%^%%] | do not match any of enclosed chars | No |
| \< | match start of word | No |
| \> | match end of word | No |
| %%^%% | match start of line | No |
| $ | match end of line | No |
| %%|%% | or | Yes |
| () | grouping -> \( \) (sed) | Yes |
| + | match 1 or more of the previous | Yes |
Now, lets try some stuff out.
To find the number of words that are exactly 4 characters long, the following can be done:
lab46:/usr/share/dict$ cat words | grep '^....$' | wc
3346 3346 16730
Therefore, there are 3346 words that are exactly 4 letters long.
To find the amount of words that are at least 4 letters long and end with a lower case g, the following can be used:
lab46:/usr/share/dict$ cat words | grep '...g$' | wc
6999 6999 70411
This results in 6999 words.
You can also use this command to do the same thing. The reason the previous command worked was because the beginning of the line did not matter.
lab46:/usr/share/dict$ cat words | grep '^....*g$' | wc
6999 6999 70411
This also results in 6999 words.
We can find the words with at-least 3 characters and are all lowercase with the following:
lab46:/usr/share/dict$ cat words | grep '^[a-z][a-z][a-z][a-z]*$' | wc
62797 62797 584774
We get back 62797 words.
To find any words that have at least 3 or more vowels, we can use the following:
lab46:/usr/share/dict$ cat words | grep '^.*[aeiouy].*[aeiouy].*[aeiouy].*[aeiouy]*' | wc
64422 64422 682102
This results in 64422 words.
We can enable extended regular expressions mode by using **egrep** instead of **grep**.
**fgrep** is used to find only literal strings. This can be faster if you are trying to only
find literal strings.
To find any words that end in 'ed' or 'ing', we can use the following:
lab46:/usr/share/dict$ cat words | egrep '*(ed)$|(ing)$' | wc
13412 13412 130200
This results in 13412 words.
Next class, we will blow our minds once again by learning about sed :D
__November 6, 2014__
The grade calculator is now an official project. Today we are continuing regular expressions and will start sed.
If we want to find all of the even weeks in the opus grades, we can use the following:
lab46:~$ status unix | grep 'opus' | egrep 'week([02468]|[0-9][02468])'
1:opus:week2entry
1:opus:week4entry
1:opus:week6entry
1:opus:week8entry
Here is a cool script using sed to re-arrange the opus grades. I don't completely understand it, but here is the code!
lab46:~$ status unix | grep 'opus' | sed 's/^.*\([01]\):\([a-z][a-z]*\):\(week\)\([0-9]\)\(.*\)$/\5 for \3 \4 \2 [\1]/g'
entry for week 1 opus [1]
entry for week 2 opus [1]
entry for week 3 opus [1]
entry for week 4 opus [1]
entry for week 5 opus [1]
entry for week 6 opus [1]
entry for week 7 opus [1]
entry for week 8 opus [1]
entry for week 9 opus [1]
__November 11, 2014__
./myscript a b c
$# - number of arguments
$* - all your arguments "a b c"
$0 - script name
$1 - "a"
$2 - "b"
$3 - "c"
Determine where a script is located.
lab46:~$ whereis status
status: /usr/local/bin/status /usr/local/bin/status.logic
commands **at** and **cron** / **crontab** are good for automation of things at certain times.
__November 13, 2014__
Going over more automation stuff.
We can use the command **at** to specify a time to run a command at.
Example:
at 16:16
ls > out
ctrl-d
Cron jobs are useful for automating tasks.
Crontab -e will open crontab configuration to create new tasks.
An example of a task is:
*/4 * * * * /usr/bin/who
This task will run the who command ever 4 minutes of every minute, hour, day, week, year
__November 18, 2014__
Today we got a new project. It is the timeline project. Basically, we need to calculate the amount of time spent on the system overall and per month for a specified user. Something interesting that we will do in this project is create a histogram.
Two useful commands: **last** and **lastlog**. Last will show all of the logins and information about those logins of all of the users in the last year. Lastlog will show the most recent last login of each user.
I think I can either search against the home directory to see if a username exists, or simply say there is no data/ the user does not exist by searching through login data. It would be nice to distinguish between the two.
__November 20, 2014__
Today was a work on stuff day. New project to do, the timeonline project. Last night I finished the grade calculator project. It is quite nice. I have a nice gui, along with a debug mode.
__December 2, 2014__
Had issues editing my opus, they are fixed now though (at least when I am at the lair).
Time for the end of course experience :D Still need to do the time online project.