Lab46 Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======UNIX/Linux Fundamentals Journal======
====MONTH Day, YEAR====

This is a sample format for a dated entry. Please substitute the actual date for "Month Day, Year", and duplicate the level 4 heading to make additional entries.

As an aid, feel free to use the following questions to help you generate content for your entries:

  * What action or concept of significance, as related to the course, did you experience on this date?
  * Why was this significant?
  * What concepts are you dealing with that may not make perfect sense?
  * What challenges are you facing with respect to the course?

====January 27, 2015==== 

This is the first entry in my Unix/Linux Fundamentals Opus. Since the first Unix/Linux class I have been reading Unix for the Beginning Mage, and have found it to be very easy to read as well as a very interesting introduction to the world of Unix.

In addition I have been able to log in to lab46 using several different methods including the terminal on my Mac, using putty on the Corning CC windows machines, and using the secure shell with the Google chrome web browser. This is nice to know for the future, because it presents options for getting my Unix/Linux fundamentals work done. I can already see that I am starting to think differently about file systems and directories and I can tell that this is only the beginning of the change in my thought process. From this point I will continue to practice my Unix spells while I finish reading Unix for the Beginning Mage.

====February 2, 2015====

This opus entry will consist of my adventures relating to the arc0 project:

One of the main challenges that I came across while working on the Archive Handling project was using tar. After I had used the unxz spell to decompress the archive.tar.xz file, I was having difficulty using the tar spell to extract the files from the archive. Looking back reflectively I believe the reason for my difficulty was I attempting to use **tar -x archive.tar** which did not have the intended results of extracting the files from the archive that I wanted it to have, I think in order to have succeeded in using -x function letter to extract the files from the archive I would have had to specify the files I wanted to extract from the archive. 

However, attempting to do this incorrectly helped me to discover another tool that eventually led me to extract the files from the archive which was **tar - - help**. In terms of quantity tar - - help was much more extensive then the tar(1) manual page, and also listed right away two variations of the tar spell that I found very useful. These were ** tar -tvf  and tar -xf**.

** tar -tvf** Lists all files of an archive verbosely and **tar -xf** Extracts all of the files from an archive. Even though tar -tvf did not get used until after I had already extracted everything from archive.tar, I still saw the potential for a useful tool if I wanted to know what was in the archive without extracting everything from it.

Going forward with the Archive Handling project, I still need to do some further research into the **gzip** tool to determine the correct way of compressing myarchive1.tar.gz on the second highest level of compression.

====February 9, 2015====

My opus entry for this week will be about the first puzzle box!

Spells that will be discussed include **file, uuencode and uudecode**. The file utility was very useful in completing the puzzle box project, mainly because it showed clearly what kind of file was actually in front of you. This is because you cannot use the naming conventions of many of the files that were encountered in the puzzle box project to determine what they contain. So actually being able to determine the true contents of the files was essential to progressing through the puzzle.

Output from **file making.waves**: "making.waves: uuencoded or xxencoded, ASCII text"

I believe that understanding the output from the command **file making.waves** was the deceptively simple part of this puzzle box. Because once you understand what the uuencode spell does this is all the information you need to finish progressing through the puzzle box. By reading that the ASCII text has been encoded you can take the necessary steps to figure out how to decode the text.   

There was a definite moment of clarity for me when I was working on this project and that occurred after I had been reading the manual page for uuencode for quite some time, without much to show for it. At this point I was under the impression that I would be able to find a solution to the step I needed to take by using the uuencode command, and failed to realize uuencode had a counterpart (uudecode). Even though by going through the manual pages reflectively I can now see that the uuencode manual page provided the information I was trying to find during that time ("uudecode will emit the decoded file to standard out"). But upon reading the Wikipedia page for Uuencoding I discovered the uudecode spell that was what needed to be used, which made my understanding of what needed to be done much clearer. This also made the output of the file command for making.waves much clearer too by presenting a solution to decode the uuencoded ASCII text to just ASCII text. Discovering additional unix tools to solve the puzzle box was a fun part of this project, and I am excited to see what comes next.

Things that are important for me going forward include reading and understanding the manual pages, immersing myself further into the world of unix/linux, and learning to use vi!

====February 10, 2015====
Today in class we are starting to learn shell scripting.

Some notes:

#!/bin/bash - ssssshhhhh bang please run my script in bash, computer.
# pound sign for comments

====February 15, 2015====
Kinda continuing my notes from tuesdays class to show how awesome I think shell scripting will be, and to summarize the content of the first scripts I was exposed to. But then I will go into detail about the adventures relating to the **NEXT PUZZLE BOX**.

So lets start with some important things I learned about bash (bourne again shell) scripts:

  * Can be saved as a file ending in .bash .sh or no dot at all... depending on organization and what helps you remember the files content.
  * using chmod to change the file permissions to execute for the user. (chmod 700 will give the user read, write, and execute permissions, and no permissions to the group and world.
  * Bash scripts can be ran by calling the spell *bash file_name* or once a file has executable permissions you can use ./file_name to run your bash script.
  * Using for loops and if then fi statements are both things I would like to learn more about so I am able to utilize them effectively in my future shell scripts.
**not a finished entry** but fast forward to 2/23 to continue reading :)

====February 23, 2015====
Okay, so I never actually finished what I wanted to say back on 2/15. Flash forward to today where I mostly want to talk about what happened in the next puzzle box.

But first a future question that I would like to document to ask later. When using the submit tool on lab46 do the appropriate files that are being asked to submit need to be modified to include personal and assignment information?

Regarding the "next puzzle box" the single most important piece of information that I was able to take away was reading carefully! When finding the password there was one specific character I was overlooking repeatedly, which was very frustrating I must say. But I did pick up some additional tools to utilize in the future which were the spells sort and bc.

At this point in time it's hard to walk through each specific steps I took. I was able to take advantage of the oct2ascii program that was given, even though the encoded password was in hexadecimal. Which led me to figure a spell using echo and bc to convert the hexadecimal to octal so I could use that output directed into a file. Then I used the file in the oct2ascii program where I discovered a sweet mustache.

those steps that were described above can be shown with the following spells:  
  * **(echo "obase=8;ibase=16"; cat hex_file) | bc > oct_file **
  * followed with **cat file | ./oct2ascii > password_file**

Another tool I was able to get some more practice with in this puzzle box was vim! I used a technique very similar to an example shown in class which one character was substituted for another in the entire file. This was useful for me when working with the file that contained the password in hexadecimal, because in order to use the first spell that converted the hexadecimal to octal the file (hex_file) could only have one entry per line. So using vim, I substituted every space character for a new line (**:%s/ /\r/g**). I was able to find the \r referencing a new line in vim on a stack exchange forum.

I should also add the hex characters containing a-f needed to be changed respectively to A-F for the bc spell to work without errors translating those 7 lines. 


====March 3, 2015====
So, I have not completed the UNIX data recovery project yet. Therefore I do not have much to write about that project yet, other than it has me pretty much stuck on how to proceed. But because of the importance of meme recovery. I will spend an appropriate amount of time discovering how to reconstruct the memory dump into something that benefits all of mankind.

But last UNIX/Linux Fundamentals class we learned lots of cool stuff that I should document, so I can refer back when needed in the future.

A big topic of last class was numeric based loops, and list based loops.

The numeric based loop that we used looked like:
<code>for((i=0;i<14;i++));do </code>

Where i=0 was the initial value, i<14 was the terminating condition, and i++ was what the numeric based loop stepped by. This script is shown below:
<code>
#!/bin/bash
#
#funwithfor1.sh
#
for((i=0;i<14;i++))do
	value=$(($i*$i))
	echo "${value}" > ${0}-${i}
done
exit 0
</code>
The output of this script produced a new file with each step, containing the value that the variables i*i produced. For example the file **fun_script.sh-8** contained the number 64.

The list based loop example that we did in class was contained in the script that looked like:
<code>
#!/bin/bash
#
#
#
i=1
echo "$0 was run with $# arguments. They are:"
for item in $*;do
	echo "\$$i is: $item"
	let i=$i+1
done
exit 0
</code>
This loop counted the number of arguments we used then echoed back what the arguments were. An example output is shown below:
<code>
./fun_with_list_based_for_loop was run with 6 arguments. They are:
$1 is: a
$2 is: b
$3 is: c
$4 is: d
$5 is: e
$6 is: f
</code>

Wildcards were another important area that we touched on last class. A wildcard in UNIX is a way to use patterns that are expressed via the command line.

The wildcards that we discussed were:
    * ** * ** 0 or more characters
    * **?** 1 of any single character
    * **[]** character class - one of any enclosed characters
    * **[^]** inverted character class - one of not matching any of enclosed

For example if you wanted to list all of the files in a directory with exactly 3 characters, a spell you could use would be ** ls ???**.

Now I really need to focus on the unix data recovery project. I believe I am on the verge of making progress if I can use the right dd commands, to get the different parts of the sacred meme. I will update more on the data recovery project when finished.

====March 9, 2015====
Hi opus, I see that we meet again this week...

The discussion topic for this opus entry will be the UNIX data recovery project. The data recovery project has been the trickiest project for me in UNIX/Linux class up to this point in the semester. Before we discussed using the dd spell (very powerful) when we met last Tuesday, I mostly found myself staring at the table of contents inside the memory dump we were given. 

Understanding how to use dd to grab the sections of memory that I needed to reconstruct the meme was the most challenging obstacle to overcome. I believe that I found this portion of the project so difficult was because I could not figure out how to put together each portion of dd in order to successfully accomplish the task I needed dd for. The different parts of dd that I needed were if= meaning the input file, of= meaning the output file, bs= meaning the number of bytes to use for the dd block size, count= meaning the length of dd input blocks that I was trying to copy, and skip= meaning the place where I wanted to copy from.

Count= and skip= corresponded respectively to the lengths and offsets that could be found inside of the table of contents that was readable in the memory dump.

For example the dd spell I needed to use to pull the first portion of the meme from the memory dump was:
<code> dd if=memdump.ram of=output_file bs=1 count=37096 skip=81056 </code>

This first portion of the lost meme turned out to be XZ compressed data which was very nice to see because it was an indication that I was starting to head down the right path.

The spells to grab the portions of the memory dump that contained the second and third part of the lost meme were identical with the lengths and offsets changed. Once I had all three portions of the meme pulled from the memory dump I was able to concatenate the three portions together, and I had RLE image data that I could work with, and manipulate to the correct meme specifications that my employer needed!

The spells that I used to work with the RLE image data were:
  * rletopnm which converted the RLE image data to netpbm image data
  * then pbmscale to scale the resolution of the meme to 414 x 418
  * then pnmtopng which converted the meme to a PNG file (portable network graphics)

Now the next portion of the data recovery project is to show the fabulous and historic meme to the world! I need to do this by putting the meme onto my lab46 webspace. I have played around with this task somewhat, but I have realized that I need to ask further questions in order feel comfortable about my file permissions when the content is being put out to the world.

I liked the premise of the unix data recovery project because it was a very real scenario, and turned lots of abstraction into something that became clearer after every step. This was refreshing to look back on and see that I could understand that the steps I used to accomplish the task at that was in front of me... 

**briefly switching from udr0 to talking about hello.c from 3/3/15**

At this point in my life, I have not taken C/C++ so I do not know how well I am going to be able to talk about the "Hello, World!" program that we wrote in C. But I did enjoy the brief exposure to the C programming language, and would be very interested in learning more for the future. 

by examining the file hello.c in vi that we came up with to print "Hello, World!" to standard output, and reading further into some of the elements included in the for loop that was used... I think I can almost understand what is happening.

To my understanding the way that we wrote the hello world program the for loop steps by one, until the length of the string "Hello, World!" is reached terminating the program, while the loop is happening the fputc portion of the code is printing one character at a time of "Hello, World!" to the standard output. I hope I have internalized what is happening correctly, and can use some of this knowledge to my benefit in the future. 
 

====March 17, 2015====
I have not yet finished udr1 so I haven't got to writing my opus entry yet. But I will be contributing very soon.

====March 18, 2015====
Well I mostly finished udr1 now, I just need to do some additional urev sessions and get some practice with UNIX permissions.

udr1 was bending my brain for a good check of time last week. I believe my initial problem was I was pulling 1 less byte than I needed from the data.file to construct the gizmo. Upon realizing I needed to include another character (the null operator). I began working on a script to de-reverse the gizmo. 

The reversal script was fairly straight forward in terms of what it was doing, however thinking about the addresses I had to use in the dd command got somewhat complicated. It was less complicated when I was planning on using data.file-normal, but then I decided to try using data.file-turbo to speed up the process, and in doing it also complicated the the dd command, and the for loop that my script contained. My script is shown below:

<code>
#!/bin/bash
# need to get gizmo
# but for real need to do a de-reversal of gizmo
rm -f gizmo

for((i=102;i>=0;i=i-1));do
dd if=gizmo.dat of=tmp bs=504 skip=$i count=1
cat tmp >> gizmo
done 
exit 0
</code>

So starting with a gizmo.dat file of size of 51912 bytes that needed to be reversed, and a grouping of 103 bytes (obtained from the factor command used on 51912, and taking the largest factor of 103), it can be determined that the block size needed for dd was 504 bytes at a time. This was found by dividing 51912/103=504. So in 103 steps down (i=i-1) of my for loop pulling 504 bytes at a time from gizmo.dat and appending each new chunk of 504 bytes to the real gizmo file, I was able to de-reverse gizmo in turbo mode.

Assuming the gizmo.dat was extracted from the correct file (data.file-normal) here is the script I would have used to extract gizmo from that file instead. (this never actually was used in obtaining an executable gizmo, but I hope it would be able to)

<code>
#!/bin/bash
# need to get gizmo
# but for real need to do a de-reversal of gizmo

for((i=51911;i>=0;i--));do
dd if=gizmo.dat of=tmp bs=1 skip=$i count=1
cat tmp >> gizmo
done 
exit 0
</code>

This script takes one byte off of the end of the file at a time, and appends it to a gizmo file one byte at a time. I preferred using getgizmo-turbo.bash which only took a matter of seconds to run the script (103 dd commands) 

I thought it was pretty cool to see what the script needed to do, and then write the script. It felt pretty cool the first time I ran it, well until I realized I had extracted the wrong data from data.file (I was fairly close and all being off by 1 byte at first)

**The udr1 sure was one whale of a project**

udr1 also introduced me to bvi as a tool for working with binary data rather than regular old text. I found the addresses needed for udr1 using bvi, and using bc to do some conversions to decimal to figuring out starting and stopping locations and what not. An example of a bc conversion would be starting bc and changing input base to hex (ibase=16) then using it to return a decimal number equal to your input.

Some other things that we talked about in lecture last week were regular expressions, and 4 octal number in a set of UNIX permissions. I will document this information here to reference later.

regular expression notes:
  *regular expressions - used inside files
  *. match any single character
  *[] match any one of enclosed
  *[^] do not match any one of enclosed
  *\< match start of word
  *\> match end of word
  *^ match start of line
  *$ match end of line
  * (*) 0 or more of the previous **star character in parentheses so there is no confusion**

above are basic regex - can be used in grep, sed, vi
regular that are used in basic unix tools

extended regex
  *\( \) pattern group
  *| or
  *() grouping 

Also the additional octal digits that can be used in permissions are:
  * 4 to set UID
  * 2 to set GID
  * 1 for a sticky bit
  * 0 for nothing

Now I should be getting started on udr2, which looks fairly intimidating at this point. I want to get an early start on it so I do not get over my head around Tuesday next week. 
 
====March 23, 2015====
Hey opus. I do not have a ton to contribute right now... but hopefully I start making some progress on udr2 soon, and can talk about the adventures it brings.

I probably should have started udr2 before right now, but all the same it sounds like it is going to be a fun and interesting project.

====April 7, 2015====
I haven't been purposefully neglecting my opus, but I have not had anything new to write about in a while. 

My plan right now is to finish udr2 today, and tomorrow. Then hopefully I'll have some good content to contribute.

====April 13, 2015====
**Notes from last class:**

How to make a ssh tunnel:

  * Used to gain better access to resources.
  * ssh -g -L 8080:www:80 user@lab46.corning-cc.edu
  * Local forwarding traffic
 
 
8080:www:80
 
8080: local port

www: destination

80 destination port

http://localhost:8080/opus/spring2015/user/start

Process Management:
  *process- program in action
  *process ID: PID

Then we practiced killing processes.

And learned about the different signals to send to kill a process.

Progress on udr2 has been incredibly slow for me, but I am still planning on pulling it all together. One of the coolest things I have discovered while working on this project is using the date command.

an example of how to use the date command that is relative to the second unix data recovery project is converting from seconds since 1/1/1970 to a date that is readable to us.

<code>date --date='@1428956124' </code>
Returns: **Mon Apr 13 16:15:24 EDT 2015**

This can be very useful for converting the timestamps in udr2 to something that makes more sense to us.

====April 20, 2015====

So one of the absolute coolest things that I have ever had the pleasure to witness occurred in UDR2.

Here is the command that I am referring to:

<code>date --date "@`echo \"ibase=16;5135D9EF\" | bc`"</code>

and this command returns the following date:

**Tue Mar  5 06:41:35 EST 2013**

====April 21, 2015====

With no UNIX class time today, I wanna get a good start on the EoCE. Hopefully I can stay more ahead of all of the tasks in this project, then UDR2.

Some of the tools I used in UDR2 with examples were as follows:

  *bc- converting hex to decimal. This was used for time-stamps, and hex addresses that needed to be decimal addresses for dd.
  *date- used for converting UNIX epoch time to a readable time-stamp when the EEG session was happeneing.
  *bgrep- used as a binary search tool that is used similarly to grep

====April 28, 2015====
Lab46 Wiki

User Tools

Site Tools

Page Tools
