Lab46 Wiki

To continue to build on our knowledge of Regular Expressions, Scripting, and utilizing them to solve problems.

Referencing the following manual pages will likely be of considerable value:

• grep(1)
• sed(1)
• awk(1)

This lab is involved. Information obtained in early steps are built upon with increasingly complex functionality.

If at any point something doesn't make sense, or you aren't getting the output you think you should be getting- ask.

It is your responsibility to understand what is going on, please be proactive by asking questions (mailing list, IRC, etc.).

As we've been exploring Regular Expressions, Shell Scripting, and even the various tools on the system, you've been told that these are important building blocks to aid you in more effective problem solving.

Now, we've amassed a considerable amount of knowledge, we can begin to see the fruits of our labor, and will put our skills to use solving a problem that might previously have been considered “difficult” or near “impossible”.

As students at CCC, a routine activity that takes place each semester is the selection of classes for the following semester. To put a schedule together, courses must be looked up and a selection of compatible times are selected, sometimes choosing from a selection of offerings, and ultimately a CRN (Course Reference Number) must be identified in order to communicate to the system.

On the main CCC web site is a link entitled Class Schedule, which allows us to select a semester and perform searches for available courses.

As it turns out, this functionality generates data in HTML format, yet it contains all the useful information we might possibly need for course selections.

This is one of those perfect examples that can be solved with our UNIX skills… the data we find available to us is in a form not immediately readable for our needs.. so what do we do when the universe doesn't align to our needs? We realign the universe!

In preparation for this exercise, I have taken the liberty of downloading a class listing for the Fall 2013 semester (Fall 2014 is not yet available at the time of this lab's release- but that's okay; if you get the logic worked out, all we have to do is substitute the data set!). This list contains all the courses offered at the primary college locations and Internet courses, and excludes ACE courses, and courses taught at non-primary college locations (high schools, etc.).

This file can be found in the courselist/ subdirectory of the UNIX Public Directory. It is called: fall2013-20130417.html.gz

We'll want to copy this file to your home directory.

Now that we have a copy of the data, we can move on to studying it.

The first step we must take when tackling a problem like this is to get an understanding of the data we are working with. Regular Expressions are cool and all, but they aren't useful unless we know what it is we are describing.

Our first task is to locate any common patterns in our data that we might be able to use to our advantage with Regular Expressions.

As it stands, each course has an information string as follows (I'll use UNIX as an example):

After the initial HTML data, we get actually course data we are interested in… there's a pattern we can take advantage of here: the course information is separated into fields, and each field is separated by a hyphen “-”. In its default state, the data is arranged as follows:

1. Course Title
2. Course Reference Number (CRN)
3. Course Prefix/Number
4. Course Section

Check out some other courses and verify that this pattern holds true. The actual data will vary, but the pattern/presentation of the data should be identical. Because of that, we can describe it with a Regular Expression pattern, and perform manipulations on it.

Although there's additional information (room, time, instructor, credits), let's start off by isolating all the individual course information strings (that exhibit that pattern above).

Using the UNIX class again as an example, the actual line in question is as follows:

<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92629">UNIX/Linux Fundamentals - 92629 - CSCS 1730 - 001</a></th>

If we go and look at another class, say ARTS 1030, we see the following:

<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92630">Drawing I - 92630 - ARTS 1030 - 001</a></th>

and GOVT 1010, we see the following:

<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=93193">American Federal Government - 93193 - GOVT 1010 - 001</a></th>

In context, these lines are surrounded by other lines of information, which we aren't immediately interested in. For example, looking at HUMA 1020:

</tbody></table>
<br>
<br>
</td>
</tr>
<tr>
<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92249">Basic Humanities II - 92249 - HUMA 1020 - 001</a></th>                                                                        
</tr>
<tr>
<td class="dddefault">
<span class="fieldlabeltext">Associated Term: </span>Fall 2013  
<br>
<span class="fieldlabeltext">Registration Dates: </span>Mar 25, 2013 to Dec 15, 2013

Each class should be in a similar situation. The line containing the course information is surrounded by lines that contain other information (whether useful or useless, there is other data there than what we are presently interested in locating).

Return to the command prompt. Time to start prototyping our solution.

We'll want to come up with a command-line that isolates those course information lines for us. A prototype for that command-line will look something like this (substitute your working RegEx pattern in place of the string “REGEX” in the example below):

lab46:~$ cat fall2013-20110417.html | grep 'REGEX' | less

When you put in the same pattern you came up with while searching in vi, your screen should be filled with data that looks like this (and much much more):

<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92473">Accounting Practices - 92473 - ACCT 1000 - 001</a></th>
<a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_syllabus?term_in=201410&amp;crn_in=92473">Learning Outcomes</a>
<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92467">Accounting Practices - 92467 - ACCT 1000 - 002</a></th>
<a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_syllabus?term_in=201410&amp;crn_in=92467">Learning Outcomes</a>
<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92938">Accounting Practices - 92938 - ACCT 1000 - 003</a></th>
<a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_syllabus?term_in=201410&amp;crn_in=92938">Learning Outcomes</a>
<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92789">Accounting Practices - 92789 - ACCT 1000 - 005</a></th>
<a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_syllabus?term_in=201410&amp;crn_in=92789">Learning Outcomes</a>
<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92792">Accounting Practices - 92792 - ACCT 1000 - 006</a></th>
<a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_syllabus?term_in=201410&amp;crn_in=92792">Learning Outcomes</a>
<th class="ddtitle" scope="colgroup"><a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_detail_sched?term_in=201410&amp;crn_in=92980">Accounting Practices - 92980 - ACCT 1000 - 007</a></th>
<a href="https://bssprod.corning-cc.edu/PROD/bwckschd.p_disp_syllabus?term_in=201410&amp;crn_in=92980">Learning Outcomes</a>
:

Because we piped our output to less(1), it stops after the first screenful of information. Pressing the down/up arrow keys or the space bar will navigate us through this data.

What we're interested in at this point is that the data that is being produced all seems to match those lines in the file that contain the course information string.

When you're satisfied with the information your pattern and resultant grep search has produced, our next step is to refine the information– to make it more readable.

To do this, we will make use of the sed(1) utility, which is a steam editor; it allows us to take the output and perform edits on it, much like we could in a text editor, only we specify on the command-line the actual work we wish to perform.

If you recall from our explorations of vi, it has a search and replace capability that proved to be rather powerful. sed(1) also possesses this ability, and we can unlock it as follows:

cat FILE | grep 'REGEX' | sed 's/PATTERN/REPLACEMENT/g' | less

Of specific interest to us are the s, PATTERN, REPLACEMENT and g options to sed(1). They have the following functionality:

• s: Invoke the sed(1) search and replace command. By default the forward slash / is the field separator.
• PATTERN: The first field following the search command is the pattern we are looking for. In this case, we want to come up with a new pattern that will match a portion of the text we wish to get rid of.
• REPLACEMENT: This field is what we wish to replace any matched text with.
• g: Not necessarily needed in all cases, the g indicates we wish to perform this search and replace globally for all occurrences on a line. I'd recommend getting in the habit of using it, and then recognizing when you don't want to use it.

So, looking at the data leading up to the course information we're interested in, can we come up with a pattern to describe it? I think so.

To test your pattern, you'll want to do the following:

lab46:~$ cat fall2013-20110417.html | grep 'REGEX' | sed 's/PATTERN//g' | less

Where PATTERN is a new Regular Expression pattern that successfully matches the beginning of the lines we're interested in (actually all that grep is producing at this point), and replacing it with nothing (the two consecutive slashes indicate we're not interested in replacing the matched data with anything).

If successful your output should appear as follows (note this is showing fall 2011 data, so the CRNs and number of offerings may be different):

Accounting Practices - 81559 - ACCT 1000 - 001</a></th>
Accounting Practices - 82350 - ACCT 1000 - 003</a></th>
Financial Accounting - 82355 - ACCT 1030 - 001</a></th>
Financial Accounting - 81558 - ACCT 1030 - 002</a></th>
Financial Accounting - 81107 - ACCT 1030 - 003</a></th>
Financial Accounting - 81108 - ACCT 1030 - 004</a></th>
Financial Accounting - 81173 - ACCT 1030 - 005</a></th>
Financial Accounting - 82115 - ACCT 1030 - 006</a></th>
Managerial Accounting - 82078 - ACCT 1040 - 003</a></th>
Accounting Procedures - 81123 - ACCT 1050 - 001</a></th>
Federal Income Tax - 81783 - ACCT 1100 - 001</a></th>
Federal Income Tax - 82358 - ACCT 1100 - 002</a></th>
Intermediate Accounting I - 81124 - ACCT 2030 - 001</a></th>
Intermediate Accounting I - 82359 - ACCT 2030 - 002</a></th>
Computerized Accounting - 82361 - ACCT 2100 - 001</a></th>
Cultural Anthropology - 81139 - ANTH 2120 - 001</a></th>
Elem Mod Stand Arabic Con&amp;StrI - 82296 - ARAB 1010 - 001</a></th>
Elem Mod Arabic Con&amp;StrI - lab - 82297 - ARAB 1010 - 071</a></th>
Introduction Art Appreciation - 81505 - ARTS 1004 - 002</a></th>
Drawing I - 81771 - ARTS 1030 - 001</a></th>
Drawing I - 82176 - ARTS 1030 - 002</a></th>
Drawing I - 82112 - ARTS 1030 - 003</a></th>
Drawing I - 81503 - ARTS 1030 - 004</a></th>
Ceramics I - 81151 - ARTS 1210 - 001</a></th>
Ceramics I - 82110 - ARTS 1210 - 002</a></th>
Ceramics I - 81504 - ARTS 1210 - 003</a></th>
Ceramics I - 82134 - ARTS 1210 - 004</a></th>
Ceramics I - 81176 - ARTS 1210 - 005</a></th>
Basic Black &amp; White Photo - 81873 - ARTS 1220 - 001</a></th>
Basic Black &amp; White Photo - 81874 - ARTS 1220 - 002</a></th>
Basic Black &amp; White Photo - 81875 - ARTS 1220 - 003</a></th>
History/Appreciation of Art I - 81180 - ARTS 1310 - 001</a></th>
:

Our sed should have successfully stripped off the leading HTML text that we're uninterested in. Once that happens, suddenly our data becomes that much more readable.

Note that there's still HTML data trailing our information. That can be addressed in another sed call:

lab46:~$ cat fall2013-20110417.html | grep 'REGEX' | sed 's/PATTERN//g' | sed 's/<\/a>.*$//g' | less
Accounting Practices - 81559 - ACCT 1000 - 001
Accounting Practices - 82350 - ACCT 1000 - 003
Financial Accounting - 82355 - ACCT 1030 - 001
Financial Accounting - 81558 - ACCT 1030 - 002
Financial Accounting - 81107 - ACCT 1030 - 003
Financial Accounting - 81108 - ACCT 1030 - 004
Financial Accounting - 81173 - ACCT 1030 - 005
Financial Accounting - 82115 - ACCT 1030 - 006
Managerial Accounting - 82078 - ACCT 1040 - 003
Accounting Procedures - 81123 - ACCT 1050 - 001
Federal Income Tax - 81783 - ACCT 1100 - 001
Federal Income Tax - 82358 - ACCT 1100 - 002
Intermediate Accounting I - 81124 - ACCT 2030 - 001
Intermediate Accounting I - 82359 - ACCT 2030 - 002
Computerized Accounting - 82361 - ACCT 2100 - 001
Cultural Anthropology - 81139 - ANTH 2120 - 001
Elem Mod Stand Arabic Con&amp;StrI - 82296 - ARAB 1010 - 001
Elem Mod Arabic Con&amp;StrI - lab - 82297 - ARAB 1010 - 071
Introduction Art Appreciation - 81505 - ARTS 1004 - 002
Drawing I - 81771 - ARTS 1030 - 001
Drawing I - 82176 - ARTS 1030 - 002
Drawing I - 82112 - ARTS 1030 - 003
Drawing I - 81503 - ARTS 1030 - 004
Ceramics I - 81151 - ARTS 1210 - 001
Ceramics I - 82110 - ARTS 1210 - 002
Ceramics I - 81504 - ARTS 1210 - 003
Ceramics I - 82134 - ARTS 1210 - 004
Ceramics I - 81176 - ARTS 1210 - 005
Basic Black &amp; White Photo - 81873 - ARTS 1220 - 001
Basic Black &amp; White Photo - 81874 - ARTS 1220 - 002
Basic Black &amp; White Photo - 81875 - ARTS 1220 - 003
History/Appreciation of Art I - 81180 - ARTS 1310 - 001

In the provided expression, the following happens:

• The pattern <\/A>.*$ explicitly matches the closing “a” tag, and then matches whatever follows until the end of the line.
• We replace that matched pattern with NOTHING.

The result should be as appears in the sample above… no HTML data, just real readable course information.

Hopefully you're starting to see the value in what the Regular Expressions have enabled for us– we were able to take raw data in some arbitrary format, and through analyzing it, adequately whittle away at it until it becomes a format readable to us.

Once in that format, we can then perform some more valuable tasks on that data.

In the courselist/ subdirectory of the UNIX Public Directory are some additional files of value:

• fall2010-20100315.html.gz
• fall2010-20101113.html.gz
• fall2011-20110417.html.gz
• fall2013-20130417.html.gz
• spring2010-20091022.html.gz
• spring2010-20101113.html.gz
• spring2011-20101105.html.hz
• spring2011-20101113.html.gz
• winter2011-20101113.html.gz

Each of these files contains a snapshot of semester course information, referenced by semester, and snapshot date. Please make a copy of these additional files, uncompress them, and let's create a script to perform some data analysis.

There are a ton of questions we could ask of this data:

• How many remedial (courses below the 1000 level) are offered a given semester?
• Do any quantity of particular course(s) increase/decrease over time?
• Is there a noticeable change in certain course offerings between a fall and a spring?

Then there are some questions that, with our current skill level, may cause us a bit of trouble:

• What is the range of CRN numbers for a given semester? (Lowest through highest)
• Which course prefix has the MOST offerings a given semester?
• Which course prefix has the LEAST offerings a given semester?
• Which course prefix offered the MOST remedial course offerings?

While we may be able to derive answers to these questions… in some respects the data is not conveniently arranged for our analysis purposes. At the moment we have our data in the following format:

And to answer some of these questions, especially when grep's are concerned, we'd ideally want the data arranged more like:

So once again our data may not be exactly the way we want it. Do we give up? HECK NO, we conform the universe to our demands…

I consider where we are at now to be amongst some of the most powerful of concepts we learn in this class. What we are going to do now hopefully should take the cake and illustrate the true potential of the capabilities we are able to wield provided a good working knowledge of Regular Expressions and related tools.

To do our next trick, we need to study our data once again:

As you can see, the information as it is currently formatted takes the following structure, as compared to the desired structure:

So how could we do this? To accomplish this task, we need to explore another RegEx capability and exercise our options in the sed REPLACEMENT field.

For my examples, I'll name your patterns REGEX1, REGEX2, REGEX3, and REGEX4.

In order to rearrange our data, we need to effectively describe the data (as you did above) in order to reference it in groups. The RegEx symbols \( and \) denote Regular Expression groups, which we can use to isolate specific patterns for later reference.

Check this out:

lab46:~$ cat fall2013-20110417.html | grep 'REGEX' | sed 's/PATTERN//g' | sed 's/<\/a>.*$//g' > output
lab46:~$ 

Notice what we just did here… we took our information in its current form of filtering and output it to a file (called output), effectively taking a snapshot of our progress.

That should make sense, we're just using I/O Redirection to send the output of that pipelined command-line to a file instead of to STDOUT.

Feel free to make use of similar output junctures during the solution of a problem like this- and who knows, you might need to do particular processing with certain arrangements of data. So if you output your data at certain key points, you could be making your work a lot easier.

Moving on:

lab46:~$ cat output | sed 's/^\(REGEX1\) - \(REGEX2\) - \(REGEX3\) - \(REGEX4\)$/\2:\3-\4:\1/g' | less
81559:ACCT 1000-001:Accounting Practices
82350:ACCT 1000-003:Accounting Practices
82355:ACCT 1030-001:Financial Accounting
81558:ACCT 1030-002:Financial Accounting
81107:ACCT 1030-003:Financial Accounting
81108:ACCT 1030-004:Financial Accounting
81173:ACCT 1030-005:Financial Accounting
82115:ACCT 1030-006:Financial Accounting
82078:ACCT 1040-003:Managerial Accounting
81123:ACCT 1050-001:Accounting Procedures
81783:ACCT 1100-001:Federal Income Tax
82358:ACCT 1100-002:Federal Income Tax
81124:ACCT 2030-001:Intermediate Accounting I
82359:ACCT 2030-002:Intermediate Accounting I
82361:ACCT 2100-001:Computerized Accounting
81139:ANTH 2120-001:Cultural Anthropology
82296:ARAB 1010-001:Elem Mod Stand Arabic Con&StrI
82297:ARAB 1010-071:Elem Mod Arabic Con&StrI - lab
81505:ARTS 1004-002:Introduction Art Appreciation
81771:ARTS 1030-001:Drawing I
82176:ARTS 1030-002:Drawing I
82112:ARTS 1030-003:Drawing I
81503:ARTS 1030-004:Drawing I
81151:ARTS 1210-001:Ceramics I
82110:ARTS 1210-002:Ceramics I
81504:ARTS 1210-003:Ceramics I
82134:ARTS 1210-004:Ceramics I
81176:ARTS 1210-005:Ceramics I
81873:ARTS 1220-001:Basic Black & White Photo
81874:ARTS 1220-002:Basic Black & White Photo
81875:ARTS 1220-003:Basic Black & White Photo
81180:ARTS 1310-001:History/Appreciation of Art I

Once you get it— WOW! the data changed, just the way we wanted. Now we can do further analysis and write shell scripts that better assist us in our tasks. Activities like this is what separates someone who can effectively command technology as a tool to assist you to someone who resorts to manual data entry, racking up hours of time manually preparing the data to attempt to answer the same questions we've asked and gotten answers to. And our processing takes a fraction of the time it would take compared to trying to do all this data filtering and rearranging by hand.

That is the power of Regular Expressions. We can effectively delegate the manual labor to the computer, which is very good at manual (and menial) tasks, and is great at following instructions.

Plus, the less we are involved at the grunt-work level, the less chance there are of errors being introduced. The computer, when it follows correct instructions, will process the data effectively, versus the unpredictability of a human manually working on the data, accidentally inserting typos or other glitches that would threaten the validity of the end data.

To cap off our experience, let's do one last foray into rearranging our data.

This assignment has activities which you should tend to- document/summarize knowledge learned on your Opus.

As always, the class mailing list and class IRC channel are available for assistance, but not answers.