Project: Archive and Data Manipulation (adm0)

To begin putting your skills to work accomplishing tasks and solving problems using command-line tools.

To successfully accomplish/perform this project, the listed resources/experiences need to be consulted/achieved:

• ability to read the manual pages and use the information therein
• ability to copy, move, and list files
• ability to navigate around the filesystem

It would be especially useful to review the manual pages or any documentation on the following resources:

• cp(1)
• mv(1)
• ls(1)
• mkdir(1)
• tar(1)
• xz(1)
• gzip(1)
• bzip2(1)
• zip(1)
• tac(1)
• rev(1)
• cat(1)
• file(1)
• uudecode(1)
• md5sum(1)

When we talk about archives, there are commonly two separate actions taking place. Sometimes they are intertwined, others they represent discrete steps.

They are:

• archiving / extracting
• compression / decompression

Archives are merely a manifestation of a common computing concept: a container.

Containers encapsulate things; in this case- files. And the fact that UNIX tries to make everything a file really enhances the viability of this ability.

Compression, on the other hand, is an action performed on a single file. Utilizing various algorithms, we accomplish a sort of “more in less”… we can take the data present and cram it into a smaller box (file)… where the aim is to take up less storage on the filesystem (also makes copying easier).

There are many compression algorithms in existence. There are commonly two categories of compression algorithm:

• lossless - no data is lost as a part of the compression process
• lossy - unnecessary data is discarded as part of the compression process

Wikipedia has categories identifying various algorithms implemented for both lossless and lossy compression algorithms.

Where confusion may set in is when a tool combines the actions of archival AND compression. But if you think about it, even in such cases, we always end up with one file, and that file is compressed (unless we have a concatenation of separately compressed files into a single file.

Archives are useful in that they let us pack items together. If something needs 100 files, making a copy of that, or copying it/install it onto another system would be made more complex if we had to deal with each of those files individually. Archives simplify the problem in that they can provide us all those files, all contained within a single file (lessening opportunities for error). So, archives make our lives easier.

As part of this activity is to test your ability to navigate around the filesystem and manipulate files on your own, there is no grabit configured for this project.

In the UNIX Public Directory you will find an adm0/ subdirectory.

There you will find directory files by the names of all the users in the class this semester. Locate yours and go into it. There you should find four archive files.

You'll want to make a copy of these files to some project-specific working directory in your home directory (~/src/SEMESTER/unix/adm0/, perhaps?)

On your development system, I want you to do the following:

1. Figure out the format of the archive, and read up on the available tools for manipulating it
2. Install any needed tools to accomplish the task of accessing the information contained within
3. Extract the contents of the archive and study it (contents will extract to the current working directory, so you WILL want to be in a custom project directory)
4. Analyze the files extracted from the archive. Each file will ultimately be contextually readable plain text (in English), but some may be encoded or compressed or otherwise manipulated and will need further processing to get to the final readable state.
   • Once in their readable states, name the files a, b, c, d, e, f, g, h, in order of their file sizes (in bytes), from least to greatest.
5. Place these single-lettered files in a new tar archive called result.tar (files should be added to the archive in the current directory, do not embed any directory information in the archive).
6. Compress it (max compression) with gzip; it should now be called result.tar.gz
   • you are going to submit this archive
7. In addition to the created archive, you will also submit a text file named adm0steps which will contain step-by-step command-lines used to copy, extract, manipulate, rename, create a new archive and compress result.tar.gz (document from the point of having the copied files in place on your development system).
   • you do NOT need to include and repository, verify, or submit commands, JUST those steps for accomplishing the core task of the files in the project to stated specifications.
   • The file should JUST contain the exact commands you used, in order from start to finish. If you'd like to add any additional commentary, prefix it with a # sign.
     • Commands should be left justified, one command-line per line (lines can wrap).

One of the tests I will perform for output compliance of your code will involve comparing your program's output against a range of input values, to see if they all output in conformance with project specifications.

I will make use of a checksum to verify exactness.

You will need to run this from your adm0 project directory, where your individual a-h files are located.

You can check your project by typing in the following at the prompt (on lab46):

lab46:~/src/SEMESTER/unix/adm0$ filechk unix adm0

If all aligns, you will see this:

==========unix/adm0 whole file comparison=========================================
 For the file: a
     you want: cca000c9cb8a5c134bed61154a7907ba
     you have: cca000c9cb8a5c134bed61154a7907ba MATCH

 For the file: b
     you want: c8136ca761229bad59497021a8f425af
     you have: c8136ca761229bad59497021a8f425af MATCH

 For the file: c
     you want: d6db0da4b084fff4b255ae7a4e95ed62
     you have: d6db0da4b084fff4b255ae7a4e95ed62 MATCH

 For the file: d
     you want: dadd5272203fa77b80f26cf355e6e833
     you have: dadd5272203fa77b80f26cf355e6e833 MATCH

 For the file: e
     you want: af095aeaaf55a8a3b351a921baebc9e7
     you have: af095aeaaf55a8a3b351a921baebc9e7 MATCH

 For the file: f
     you want: 84d0fd81532fac6c743c8054f76f0270
     you have: 84d0fd81532fac6c743c8054f76f0270 MATCH

 For the file: g
     you want: c36a56a9ab8190e4d007bd16e377639a
     you have: c36a56a9ab8190e4d007bd16e377639a MATCH

 For the file: h
     you want: 226c53b09f112cf7323cd5263302ea95
     you have: 226c53b09f112cf7323cd5263302ea95 MATCH

If something is off, your checksum will not match the adm0 checksum, and verification will instead say “MISMATCH”, like follows (note that a mismatched checksum can be anything, and likely not what is seen in this example):

==========unix/adm0 whole file comparison=========================================
 For the file: a
     you want: cca000c9cb8a5c134bed61154a7907ba
     you have: cca000c9cb8a5c134bed61154a7907ba MATCH

 For the file: b
     you want: d8136ca761229bad59497021a8f425af
     you have: c8136ca761229bad59497021a8f425af MISMATCH

 For the file: c
     you want: d6db0da4b084fff4b255ae7a4e95ed62
     you have: d6db0da4b084fff4b255ae7a4e95ed62 MATCH

 For the file: d
     you want: dadd5272203fa77b80f26cf355e6e833
     you have: dadd5272203fa77b80f26cf355e6e833 MATCH

 For the file: e
     you want: af095aeaaf55a8a3b351a921baebc9e7
     you have: af095aeaaf55a8a3b351a921baebc9e7 MATCH

 For the file: f
     you want: 84d0fd81532fac6c743c8054f76f0270
     you have: 84d0fd81532fac6c743c8054f76f0270 MATCH

 For the file: g
     you want: d36a56a9ab8190e4d007bd16e377639a
     you have: c36a56a9ab8190e4d007bd16e377639a MISMATCH

 For the file: h
     you want: 226c53b09f112cf7323cd5263302ea95
     you have: 226c53b09f112cf7323cd5263302ea95 MATCH

On the system hosting the needed resources, egress from your home directory and navigate to the respective class public directory, locate the subdirectory for this project and navigate there.

Assess the layout of files. What type of files are here? Are they named in a manner so as to indicate a specific course of action?

Once you locate your files, proceed to copy them into your home directory, into a custom project subdirectory you've made, ideally in your repository.

Ingress to that destination, ensure said files have been included into your repository. Do note: your repository by default may be configured to ignore many archive files. To override this, add them specifically by name.

Transition to your development system, navigate into your repository. Obtain the files, and verify they are present.

Focusing on one archive at a time:

• what type of archive is it?
• what tools might be needed to extract and/or decompress it?
• reference the manual page(s) for any tools in question, determine any options that need to be applied.
• attempt to extract files from the archive. Did it work? How can you tell what new additions there are? (Do note: many of these tools have options to enable *verbose* operation, which might prove particularly helpful in this endeavour).
• one at a time, investigate any new files:
  • is it viewable?
• is it readable?

* if not, is there something that looks out of sorts that you could manipulate to correct it?

• is it contextually readable, english text?

* take note of the file size

• as you accumulate processed files, proceed to order them by file size

To successfully complete this project, the following criteria must be met:

• Submit a copy of your archives to me using the submit tool.

To submit this program to me using the submit tool, run the following command at your lab46 prompt:

$ submit unix adm0 result.tar.gz adm0steps
Submitting unix project "adm0":
    -> result.tar.gz(OK)
    -> adm0steps(OK)

SUCCESSFULLY SUBMITTED

You should get some sort of confirmation indicating successful submission if all went according to plan. If not, check for typos and or locational mismatches.

I'll be evaluating the project based on the following criteria:

26:adm0:final tally of results (26/26)
*:adm0:archive submitted [2/2]
*:adm0:archive has correct name of result.tar.gz [2/2]
*:adm0:archive is max compressed with gzip [2/2]
*:adm0:archive is a tar archive [2/2]
*:adm0:archive extracts into current directory [2/2]
*:adm0:archive contains 8 english readable files [2/2]
*:adm0:archived files are named a-h [2/2]
*:adm0:archived files named in order of size [2/2]
*:adm0:instructions submitted in text file [2/2]
*:adm0:instructions in file named adm0steps [2/2]
*:adm0:adm0steps contains list of instructions for accomplishing task [2/2]
*:adm0:adm0steps instructions are accurate and correct [2/2]
*:adm0:adm0steps any extra information after hash mark [2/2]