UNIX-y tasks

Things you should do before too much time passes:

• If I had a dime for everytime someone said they could use a refresher in basic programming / problem solving / computer science concepts… and then this bumps into me: UNSW Computing 1 - The Art of Programming 12 week free course starts December 3rd.

• Talked out the following commands:
  • head
  • tail
  • paste
  • join
  • diff
  • patch
  • comm
• Among others. Using them in the filtering of data.
• OPUS Part 3 Keyword is due before it is no longer November 16th.

• Super Puzzle Box 2 Turbo, due November 14th.
• OPUS Part 3 Keyword Selection select a keyword and get started.

• Change of pace! The X Window System (or X, X11, X11R6/7)
• client/server model designed to operate in a networked environment
• X clients talk to an X server– they do NOT have to be on the same machine
  • The thin clients in the LAIR take advantage of this capability
• DISPLAY environment variable
• your X location string (host + X screen)
• redirected X client output (xeyes, xlogo) to other screens
• played on the LAIRwall

• Continued to play with RegEx class listing example. I assigned its completion for Friday (e-mail me you final command-line to produce the filtered output).

• More fun with Regular Expressions, started an in-class exploration doing some hands-on learning with RegEx on the CCC Spring course schedule.

• Regular Expressions are upon us!

• Some more process fun:
  • The init process
  • Hierarchy of processes / connectivity to terminals (PTYs)
  • The existence of zombies
  • How parents should always prefer their children to die once they complete their usefulness, else they'll become a zombie
• Played with grep(1), sed(1), and cut(1) to tweak some ps(1) output
  • Used a Regular Expression, something we will talk more about this coming week.

• Project 0x03- Puzzle Box 2 is now available, and is due on the 24th. Different than the first, I would not wait until the last minute to complete this one.
• We talked about processes (programs in action)
  • ps(1) - process status
  • top(1) - show currently active processes
• We talked about signals (the kill(1) command), and killed many cats in various odd yet specific ways.
• We wrote a small C program that:
  • took as input 4 integers from STDIN
  • calculated an displayed their average
  • Used STDIN redirection to feed the program its data from an external file

• Ventured into C programming for a bit
• Covered some of the recognized programming paradigms:
  • structured (most languages we are familiar with)
  • functional (LISP)
  • logical (Prolog, Haskell)
  • object-oriented (Eiffel, Smalltalk)
• Wrote and more-closely analyzed “Hello, World”
• Implemented an equivalent command-line C program as we did in bash on Wednesday.

• More scripting:
  • “Self-aware” script through the use of file access
  • “Intruder” detection with an elif and command-line arguments
  • More extensive command-line arguments, including usage with a list-based for loop.

• Due to the impending break, I declared the day a work day, so everyone could get caught up on work and ask questions.
• gimmeh is mangling some of the “Unconventional Naming” responses. I am aware of the problem- just run with whatever it decides to store. Any responses that come up blank just e-mail me what you put.

• OPUS Part 2 Keyword is due October 16th.
• Starting this month, I will cease accepting assignments submitted past their due date (unless there are extenuating circumstances). Many people have been getting assignments done on time, others are procrastinating until the last day, and have trouble as they are rushing and don't have much time left. Please don't wait until the last minute, procrastination isn't the way.
• Please look through this wiki page I put together on the "6 tries to guess a number" script we started writing in class on Friday. There are some additional variations from what we did. Please play with the code and figure out what is going on. And be sure to ask questions on anything that isn't making sense.
• Some insightful commentary on the UNIX Command Line: To Understand the Command Line. Well worth giving a read.

• Project #2: Unconventional Naming by October 5th.
• Don't forget! The full Part 1 of the OPUS is due once it becomes October.
• We explored more with shell scripting, including:
  • Using echo to output information to the user (STDOUT)
  • Using read to input information from the user into a variable (STDIN)
  • The $RANDOM environment variable, which gives us obscenely ranged random whole numbers
  • if statements, enabling us to make boolean choices (exactly 1 per if block)
    • the closing fi statement to end an “if block” (1, at the very end of all the if/elif/else action)
    • We can have an else statement to contain the countering action (0 or 1 in total)
    • if there is more than one condition we wish to check for, else if (elif) statements can be strung along (0 or more in total).
  • The [ and ] commands, giving us some more capable evaluative functionality, including our 6 relational operators:
    • -eq is equal
    • -ne is NOT equal
    • -gt is greater than
    • -ge is greater than or equal
    • -lt is less than
    • -le is less than or equal
  • The use of the exit command to force termination of the script
    • The good habit of placing a “exit 0” at the end of your script
  • We ended up writing one of those "6 tries to guess a number" games, a perfect utilization of I/O, variables, and decisions using an if block.
  • Customization of the range of random values $RANDOM will give us with a little bc, here string, and command expansion magic:
    • value=`bc -q <<< "($RANDOM % 10)+1"` (for a value in the range of 1-10, leave off the +1 and it is 0-9, by the mathematical properties of the modulus division (%)).
  • Finally, a quick for loop was thrown in doing a quick count from 0-5 (6 total iterations) giving the user their 6 choices.

• OPUS Part 1 to be completed by October 1st.
• OPUS Part 2 Keyword Selection by October 1st.
• Useful commands to know: head(1), tail(1)
• This is a great example of when consumers become the producers.
• For those that enabled workdue notifications in gimmeh, you may have started receiving daily e-mails (depending on your settings). Remember, it is a double opt-in system- you have to enable workdue-status, AND email-workdue, otherwise it'll skip you by.
• We played a lot more with wildcards, posing some puzzles to craft patterns for (in /usr/bin)
  • All the files 4 chars in length: ls -d ???? | wc -l
  • All the files 2 or more chars in length: ls -d ??* | wc -l
  • All the files that end with a lowercase vowel: ls -d *[aeiou] | wc -l
  • All the files that do not start with an uppercase vowel and whose 3rd letter is a 'd': ls -d [^AEIOU]?d* | wc -l
  • All the files that do NOT start NOR end with a lowercase 'm', but DOES contain elsewhere in its name a lowercase 'm': ls -d [^m]*m*[^m] | wc -l
• If you are ever in need of an “emergency” ls, echo * will work in a pinch, but won't be pretty.
• We discovered that the default behavior of ls(1) when it encounters a directory is to expand is list that directory's contents. This might seem to “break” our wildcard until we see that it is in fact successfully matching the directory's name, then just brings whatever contents along for the ride. We can fix this by issuing the -d argument to ls, which will cause it to treat directories like regular files (ie just list them, and not descend into them).
• When you see mention of command(#), it is referring to a particular section of the manual pages for a command (file(1), cp(1), printf(1), printf(3)).
  • To look up a specific section of the manual, insert it as an argument before the command; for example:
    • printf(3): man 3 printf
    • cp(1): man 1 cp
  • By default, without a section number, the first available manual page in the lowest numbered section will be called up.
• You can search through the available manual pages with apropos(1)

• Last full day of summer in the Northern Hemisphere.
• Project #1: The Puzzle Box by September 28th.
• Control Characters were covered (CTRL-a through CTRL-z).
  • TIP: Do NOT try and issue Control Characters while not on the command line! Programs like VI and SCREEN capture control sequences and can apply different functionality.
• Some vi power tips given
  • recording mode explained and demonstrated
  • CTRL-a on a number in vim will do something amazing
  • The opposite is CTRL-x (apparently)
• How to unfreeze your terminal (CTRL-q, it is what started us on control characters).
• Wildcards were introduced:
  • * - match 0 or more of any character
  • ? - match 1 of any character
  • [ ] - character class; match 1 of any of the enclosed characters
  • [^ ] - inverted character class; do NOT match 1 of any of the enclosed characters
  • Examples:
    • ls *.c – match all files ending with “.c”
    • ls g?? – match all 3 letter files starting with a lowercase 'g'
    • ls ??* – match all files that are 2 or more characters in length
    • ls *[rstlne] – match all files ending with lowercase r, s, t, l, n, e
    • ls *[0-9] – match all files ending a number (0-9)
  • Common mistakes made with the character class:
    • you don't need to separate values with commas. Putting a comma in a character class will cause the comma to be a character searched for to match that particular pattern.
    • you don't need a character class if you know the specific character you are looking to match. [e] is more typing than e.
  • Character classes support some ranges:
    • Single digit numbers: [0-9]
    • Lowercase alphabet: [a-z]
    • Uppercase alphabet: [A-Z]
    • Any letter, regardless of case: [A-Za-z]
    • Partial ranges: [a-f]
    • Multiple partial ranges: [b-gm-q]

• Project #0: Archive Handling by September 26th.
• Be sure to evaluate your received OPUS Part 1 Keyword by September 24th. Then get started on the demonstration before the month is up.
• X11, the traditional and standard UNIX Graphics environment, turned 25 on Saturday.
• I introduced the gimmeh tool. Please run it and customize your settings, and tend to any course-related content.
• Opus Part 1 keywords have been swapped. You have until the end of the month to wrap up the demonstrations of the received keyword.
• We reviewed pipes, which lead into a great introduction to shell scripting.
  • We saw one-liners like: last | less
  • and: last | grep $USER | wc -l
  • which evolved into: echo “You logged in `last | grep $USER | wc -l` times this month.”
  • plus the amazing concept connection to put that line in a file: echo 'echo “You logged in `last | grep $USER | wc -l` times this month.”' > logincnt.sh
• The most frequent mistake is typing the wrong quotes. Double check those things!
• grep returns matching lines (by default) based on a provided search pattern (grep $USER will return all lines that contain your username, for example). It stands for globally search for regular expression and print (GREP). We will be doing a lot more with grep in the coming weeks.
• A script is just a text file with valid command-line combinations set with the execute bit and then run like any other command (./logincnt.sh, or invoke through a shell via: bash ./logincnt.sh).
• Proper shell scripts should have a shabang line to start them off- the one most valid for our explorations this semester will be: #!/bin/bash
  • The interpreter specified via absolute path in the shabang line should conform to the syntax present within the script. We'll be learning the bash shell (bourne again shell) and syntax, so if you're doing stuff for class, that'll probably be the one you want.
  • On the other hand, if you are exploring something like awk, perl, python, or even some other shell, and using its syntax, putting the appropriate shabang line will yield you the most success.
  • a script is generally called such as it is typically high level and can be executed as-is.
  • a programming language is generally called such as its syntax needs to be converted into a more machine-readable form (compiler, assembler) to be run. The distinction is not always that cut and dry, but could be a good rule of thumb.
• Mail forwarding is accomplished by sending a copy of any incoming messages to all addresses contained within the ~/.forward file, located in the base of your home directory. This file must not be world writable; if it is, this functionality will be disabled.

• Remember that your keyword for the Opus is due to be completed this weekend.
• We customized our vim sessions by editing/creating (with vim) our ~/.vimrc file.
• We learned how to create files with touch(1)
• Some explanations of STDIN, STDOUT, and STDERR were in order (remember, everything is a file).
• I/O Redirection was then explored (<, >, >>, 1>, 1>>, 2>, 2>>).
• I also mentioned the here string << and we used it with the STDIN redirector to force command-line level input as file-based STDIN redirection: <<< "desired input"
  • For example, with bc: bc <<< “2+2”
• Pipes were then built (using the | symbol) for some awesome incantations.
• Variables and the variable expansion operator ($)
• Quotes were covered (half- “”, full- '', and command expansion `` (backticks)).
• File permissions were covered, and accomplished using the chmod command.
  • We looked at what happens when you have read-only and write-only files.
• Remember that your OPUS Part 1 Keyword is due by September 16th. We'll be doing a swap shortly thereafter.

• We finished covering the basic vi commands (prefix + [modifier] + command)
  • for navigation
    • by character: h, j, k, l (and cursor keys)
    • by word: w/W, b/B
    • by line: ^, $, #G
  • for insertion: i, I, a, A, o, O
  • for getting back to command mode: ESC
  • for modification:
    • copy/yank: prefix + y + navigation command
    • cut/delete: prefix + d + navigation command, x/X
    • paste: prefix + p
    • change: prefix + c + navagation command
    • toggle (alter case): prefix + ~ (tilde)
    • substitute: prefix + s
    • join lines: prefix + J
    • undo: u
  • for searching:
    • initiate a search: /thing you are looking for
    • jump to next match: n
    • jump to previous match: N
• We also looked at a few of the extended vi command mode commands (hit “:” to enter extended command mode):
  • move: addressmaddress
  • copy: addresscoaddress
  • search/replace: addresss/pattern/replacement/g
  • save (current file): w
  • save (as file): w filename
  • quit (if saved): q
  • quit (without saving): q!
  • save and quit: wq
• An address can be one of the following (you can mix and match them as is appropriate):
  • an absolute line number: 7 (7th line in the file)
  • a relative line number: . (current line), -2 (two lines prior)
  • an absolute range: 1,3 (lines 1-3)
  • a relative range: .,+5 (current line through the following 5 lines)
  • for search/replace: % (entire file)
• Be sure to take some time to play in vi/vim. We covered some of the basic commands last week (insert mode, ESC for command-mode, character (arrow keys/h, j, k, l), word (w, b), and line (^, $) navigation).
• In addition to playing in vi, be sure to keep playing on the command-line with the tools we've learned so far.
• There will likely be a project appearing some time this week or next.

• By this day, be subscribed to the class mailing list, and have chosen a keyword. I will count the successful completion of these two things as a project.
• Please make some time to play- experiment with the commands we've learned; explore the filesystem.
• To prepare for what is soon to come, please glance at this and start getting ready to change the way you think about editing files.
• If you haven't set up your data directory yet, please type this: ln -sf /usr/local/etc/data/$USER ~/data
  • If after doing this, you see a red “data” symlink in your home directory, you typed that wrong.
• Please be mindful of others- at times it has been approaching too noisy with conversations during class. I encourage communication, but not at the expense of other's learning- learn the fine art of soft speaking or whispering.
• UNIX Historical Video: AT&T Archives: The UNIX Operating System

• My assertion from the first day of class is that Joe O's preference for Internet Explorer, despite the fact that he knows better, is an example of akrasia (the state of acting against one's better judgement).
• For your consideration, Arthur C. Clarke's Three Laws:
  1. When a distinguished but elderly scientist states that something is possible, she/he is almost certainly right. When they state that something is impossible, they are very probably wrong.
  2. The only way of discovering the limits of the possible is to venture a little way past them into the impossible.
  3. Any sufficiently advanced technology is indistinguishable from magic.
• Please to be learning and committing to memory the three tenets of the UNIX philosophy (then sample some other views of UNIX Philosophy and UNIX Koans):
  1. Small is beautiful.
  2. Do one thing, and do that one thing well.
  3. Everything is a file. (well, 99% of the time)
• Please do yourself a favor and read UNIX for the Beginning Mage.
• Although interpretations vary, I prefer to believe there are three types of files (please familiarize yourself with them and how they differ):
  1. regular
  2. directory
  3. special
• Also, begin acclimating yourself to the notion of the standard system data streams:
  • 0. Standard Input (STDIN)
  • 1. Standard Output (STDOUT)
  • 2. Standard Error (STDERR)
• Become comfortable with navigating the filesystem, for we will soon be exploring some sights, and the more familiar you are with cd and ls, the better.
• If you enjoy contemplating incomprehensible objects and seeing their adoption into culture, check out the blivet.
• You have an e-mail account on Lab46. Your address is lab46username@lab46.corning-cc.edu. You can check your mail with the alpine command.

• I had a request for documentation on setting up the screen/irssi session for class chat. Here it is.
• Clone and start using your personal Lab46 Mercurial Repository
• Select your Part 1 Opus keyword BEFORE September 7th.
• Start working on said keyword, for you need to have the definition part done by mid-September.
• Before leaving, detach from any screen sessions (CTRL-a d), log out of any terminals (exit command), close any web browsers or other open applications on your desktop, and right click → exit to log off. You do NOT want to leave while still being logged in!
• familiarize yourself with some commands we have learned so far: ls, hg, man, wtf, pom, cd

• Edit/customize at least the Title and Introduction sections of your Opus BEFORE September 7th.
• Subscribe a frequently checked (or pushed) e-mail address to the class mailing list
• When you come to class, remember to open a Lab46 terminal, which will be used for attendance/participation purposes.
• Make sure you can access the UNIX course homepage from home (you may want to bookmark it).
• Make sure you can log into Lab46 from home with your username/password (using PuTTY if on Windows, or some other SSH client).
• Get on the class chat, and familiarize yourself with basic screen usage (attaching/detaching).
• * familiarize yourself with some commands we have learned so far: who, man, screen, irssi, exit/logout