<WRAP round gray box><WRAP muchbigger><html><center></html>STATUS updates<html></center></html></WRAP>
\\ 
</WRAP>
======TODO======

  * the formular plugin is giving me errors, need to figure this out (email assignment form)
  * can I install writer2latex on wildebeest herd without needing gcj??
  * <del>look into how to adequately set up subversion <nowiki>svn+ssh://</nowiki> authentication</del> using <nowiki>http://</nowiki> access now
  * set up symlink and cron job for userhomedirbackup.sh on nfs2; update wiki page for nfs
  * update grade not-z scripts to handle/be aware of winter terms
  * update system page for (new)www
======URLs======

Some links of interest:

  * [[http://www.llvm.org/|LLVM]]
  * [[http://www.reocities.com/harpin_floh/glglobe_page.html|GLglobe]]
  * [[http://www.heavens-above.com/|Heavens Above]]
  * http://wiki.debian.org/kristian_jerpetjoen
  * http://www.webupd8.org/2010/11/alternative-to-200-lines-kernel-patch.html
  * http://myproxylists.com/nix-brute-force
  * [[docs/plan9|My Plan9 Documentation page]]
  * [[system/sokraits_halfadder.offbyone.lan|The Sokraits/Halfadder System Documentation page]]
  * http://www.egghelp.org/commands/channels.htm
  * http://stackoverflow.com/questions/470617/get-current-date-and-time-in-php
  * http://www.rjsystems.nl/en/2100.php

======Other Days======
======January 31st, 2011======
=====wildebeest herd back=====
With some recent developments figuring out how to do XDMCP once again on the USB-enabled pods, I've re-awakened the full wildebeest herd, and aptitude update; aptitude update'd the whole lot... updated fluxbox menus, and cleaned package cache.

Just need to configure the flakes themselves to divvy out the XDMCP host connections.

=====XDMCP connections on flakes=====
After a few days, we've finally figured out how to get decently reliable XDMCP client connections going on the Ubuntu USB-terminal flake stations.

I'll be posting more particulars as I configure everything in the coming days.

=====LAIRstations 1-3 online=====
In my quest to have 4 dual-head LAIRstation machines for student use, I now have 3 of them up and operational... all running Ubuntu 10.10, autofs+LDAP, along with automatic Nvidia twinview going on boot.

I'll be setting up LAIRstation 4 in the coming days, especially as the great re-LAIR-rangement continues and dust settles.
======January 25th, 2011======
=====svn web html rendering=====
It would appear that, by default, when web viewing an SVN repository, viewing an HTML will retrieve and display the HTML code, instead of passing it to the browser to render.

I've looked a bit into this, and there appear to be solutions... I tried one or two promising looking ones, but did not get instant gratification, so I removed the options.

A link that appears to talk about what I want (for websvn, which may ultimately be the way to go):

  * http://old.nabble.com/-WebSVN--mixed-anon-auth-authentication-and-login-logoff-td28035968.html
======January 23rd, 2010======
=====roundcube webmail=====
I encountered a spiffy-looking webmail client called roundcube, which was easily installed on www.

It is now up and running (just point your browser at: http://lab46.corning-cc.edu/mail/).

I encountered one problem that has been fixed--- upon successful log in, you'd immediately be logged out (you'd see your inbox, then wham! login screen).

Turns out this is a common issue:

  * http://www.roundcubeforum.net/5-release-support/17-pending-issues/6172-another-immediate-log-out-after-log-problem.html
  * http://www.roundcubeforum.net/3-news-announcements/12-general-discussion/6196-login-page-doesnt-take-me-anywhere.html

As it turns out, the solution was to disable session encryption for the suhosin php functionality.

I ended up having to add the following to the sites-enabled/000-default file:

<code>
Alias /mail/program/js/tiny_mce/ /usr/share/tinymce/www/
Alias /mail /var/lib/roundcube
</code>

=====updates/restarts=====
In preparation for the new semester, I performed updates on the following machines:

  * lab46
  * irc
  * www
  * auth1, auth2, auth3
  * sokraits, halfadder

I rebooted www (actually shutdown then xm create), and gave it more memory-- it now has 384MB of memory.. it was hitting swap before and making some web lookups rather slow.

=====grade not-z scripts: calcweek.sh=====
I need to put in extra checks to ensure grade not-z scripts do not run prior to the start of the semester... assignments, journals, etc.

attendance is probably the only thing smart enough at the moment to know better, yet doesn't offer anything that the other functionalities can immediately lock onto to ensure that we're in a valid semester and week or not.

Actually, as it turns out, the problem was with **calcweek.sh**, whose logic couldn't handle the new year (it would compute the absolute day from the start of the year, and do a comparison against the absolute day of the semester year)... yielding some impressive negative numbers.

I have put in a simple check for the year change, and if detected, merely defaults to a "week 0" status which will cause all the other scripts to resume cooperating.

=====New User password e-mails=====
I automated the process of disclosing Lab46 account information to new users. The following script(s) were used:

<code bash 1>
for entry in `cat readylist`; do
    username="`echo $entry | cut -d':' -f1`"
    first="`echo $entry | cut -d':' -f2 | cut -d'*' -f1`"
    password="`echo $entry | cut -d':' -f3`"
    echo "$first," >> $username.letter
    echo >> $username.letter
    echo "Enclosed please find your Lab46 account information. This will be" >> $username.letter
    echo "used to log onto the Lab46 system for use with your classes this" >> $username.letter
    echo "semester. If you have any questions please let me know at: " >> $username.letter
    echo >> $username.letter; echo "                    haas@corning-cc.edu" >> $username.letter
    echo >> $username.letter; echo "-------------------------------------------" >> $username.letter
    echo "    Lab46 username: $username" >> $username.letter
    echo "    Initial Password: $password" >> $username.letter
    echo "    Hostname: lab46.corning-cc.edu" >> $username.letter
    echo "-------------------------------------------" >> $username.letter
    echo >> $username.letter
    echo "-Matthew" >> $username.letter
    echo "--" >> $username.letter
    echo "  Matthew Haas" >> $username.letter
    echo "  Instructor" >> $username.letter
    echo "  Computer & Information Science" >> $username.letter
    echo "  http://lab46.corning-cc.edu/haas/home/" >> $username.letter
    echo >> $username.letter
    echo "    \"Writing should be like breathing;" >> $username.letter
    echo "      it is one of those important things we do.\" --me" >> $username.letter
done
</code>

The above script generated the letters (from my generated username file).

On Lab46 I then fired them off as e-mails with the following:

<code bash 1>
for letter in `/bin/ls -1A *.letter`; do
    username="`echo $letter | cut -d'.' -f1`"
    mail ${username}@corning-cc.edu wedge@lab46.corning-cc.edu -s "Lab46 Account Information" < $letter
done
</code>
=====studentlistcreate.sh=====
There was apparently a long-standing bug in the student pages generation script, which caused some duplicates to appear.

It was one of those classic "similar substring" problems, but this time in the behavior of one of the tools used.

Original logic:

<code bash>
 16 # Gather some metrics
 17 for user in `/bin/ls -1 /home`; do
 18     uchk="`groups $user | egrep -o '(lair|offbyone|faculty|dslab)' | wc -l`"
 19     if [ $uchk -eq 0 ]; then
 20         fullname="`finger $user | grep -o 'Name.*$' | sed -e 's/Name: \(.*\)    $/\1/'`"
 21         jchk="`groups $user | egrep -o "$classgrp" | wc -l`"
</code>

And fixed it by adding a **head -1** to the pipeline after the call to **finger**, as follows:

<code bash>
 16 # Gather some metrics
 17 for user in `/bin/ls -1 /home`; do
 18     uchk="`groups $user | egrep -o '(lair|offbyone|faculty|dslab)' | wc -l`"
 19     if [ $uchk -eq 0 ]; then
 20         fullname="`finger $user | head -1 | grep -o 'Name.*$' | sed -e 's/Name: \(.*\)    $/\1/'`"
 21         jchk="`groups $user | egrep -o "$classgrp" | wc -l`"
</code>

Problem solved.

=====journalinit.sh=====
My journalinit.sh script apparently had some bug in it (I think a remnant of the [A-Z] LOCALE problems) that allowed the fullnames to be mixed in with the usernames, creating an inotify watch list that was about 3 times larger than it should have been.

A simple fix (restoring the regex to a sane [^A-Z] value) restored proper functionality.

Also I removed the in-script semester-detection logic and replaced it with a call to semester.sh

=====gengrps=====
Using my notes taken on the wiki from the spring and fall of 2010 semesters, I finally created a script to handle LDAP group membership for class-related purposes.

The script, gengrps, currently has the following contents:

<code bash 1>
#!/bin/bash                                                                               
#
# gengrps - script to assist with the generation of LDIF files for
#           populating class LDAP groups
#
# If run by itself, all LDIF definitions will be displayed to STDOUT.
# Ideal method of production execution:
#
#       gengrps | ldapmodify -x -W -D "cn=admin,dc=lair,dc=bits"
#
ATTNDPATH="/home/wedge/local/attendance/etc/list"
semester="`/usr/local/bin/semester.sh`"
LDAPINFO="dc=lair,dc=bits"

for group in `/bin/ls -1A ${ATTNDPATH}/class.${semester}.* | cut -d'.' -f3`; do
    grouplst="$grouplst $group"
done

for grp in $grouplst; do
    if [ "$grp" = "hpc1" ]; then
        echo "dn: cn=hpc,ou=groups,${LDAPINFO}"
    elif [ "$grp" = "hpc2" ]; then
        echo "dn: cn=hpc,ou=groups,${LDAPINFO}"
    else
        echo "dn: cn=${grp},ou=groups,${LDAPINFO}"
    fi  

    echo "changetype: modify"
    echo "add: memberUid"

    for student in `grep -v '^[A-Z]' ${ATTNDPATH}/class.${semester}.${grp}.*`; do
        echo "memberUid: ${student}"
    done
    echo
done

for student in `cat ${ATTNDPATH}/class.${semester}.* | grep '^[^A-Z]' | sort -u`; do
    echo "dn: cn=${student},ou=groups,${LDAPINFO}"
    echo "changetype: modify"
    echo "add: memberUid"
    echo "memberUid: wedge"
    echo
done

exit 0
</code>

Seems to work quite well, and when used in a pipeline with ldapmodify, operates without the need to directly create any temporary files.

The next step (at some point) will be to create a script that employs similar logic, but is used to remove group membership, such as at the end of the semester (a process I still performed manually this time around).
======January 20th, 2010======
=====student data initialization=====
I took a closer look at the various scripts and facilities that need to be in place for proper grade not-z operation.

====journals====
This began with getting the journal wiki pages set up.

I basically created a spring2011/ directory in the appropriate place in the directory tree, and put a start.txt and placeholder.txt there. I edited the ACLs and basically replaced the spring2010 entry with spring2011 (so existing spring2010 journals will still exist but nobody will have any permissions other than read at this point).

As the semester kicks off, I need to fire up journalinit.sh in a screen session on www, which will monitor for new journal creations. I don't want to do this now, as registration lists may still endure some changes. I should look into making it resilient enough to handle that sort of dynamic environment.

====data directories====
I also initialized the student data directories, which will be used by the grade not-z for collecting course metrics on the students.

I backed up the existing one (to data.fall2010) just in case I'd have a need for it, then cleared the contents of data and reinitialized it as follows (as root):

<code bash>
for user in `cat /home/wedge/local/attendance/etc/list/class.spring2011.* | grep -v '^[A-Z]' | sort | uniq`; do
    mkdir /home/wedge/local/data/$user
    chown -R wedge:$user /home/wedge/local/data/$user
    chmod 1770 /home/wedge/local/data/$user
done
</code>

=====student svn repositories=====
SVN appears to be fully functional to my expectations at this point... after some further tests, both project and user repositories are accessible with appropriate permissions.

The final bit was the inclusion of my ability to check out each users' repository's submit directory (read-only), which I plan to use for various assignments.

The code to do that is as follows:

<code>
[/submit]
wedge = r
</code>

Note the lack of any repository reference... this is global for all repositories (that succumb to this ACL, which is just the user repositories).
======January 19th, 2011=====
=====www svn=====
Continuing my configuration adventures with web-based svn, I am now reading through the SVN book.

I now have functioning web-based SVN access working, with access control!

I rolled this out for the backgammon, cpu, and mgough's bigworld project. I'm going to do mostly similar for the private user repositories (I think I'm going to end up placing them under /svn/* on the URL namespace).

====acl for user repositories====
I needed to generate the ACL information on a per-user basis (as far as I can tell)... I wrote the following script to assist:

<code bash>
for entry in `/bin/ls -1A`; do
    valid="`id $entry 2> /dev/null | wc -l`"
    if [ "$valid" -gt 0 ]; then
        echo "[$entry:/]" >> /var/repos/acl/users.acl
        echo "$entry = rw" >> /var/repos/acl/users.acl
        echo >> /var/repos/acl/users.acl
        echo "[$entry:/submit]" >> /var/repos/acl/users.acl
        echo "wedge = r" >> /var/repos/acl/users.acl
        echo >> /var/repos/acl/users.acl
    fi
    echo "processed $entry ..."
done
</code>

In a similar vein, I discovered there were a lot of lingering repositories of no-longer-existing users... I decided to prune them while I was in the neighborhood. Thanks to the help of the following script:

<code bash>
for entry in `/bin/ls -1A`; do
    valid="`id $entry 2> /dev/null | wc -l`"
    if [ "$valid" -eq 0 ]; then
        rm -rvf $entry
    fi
done
</code>

I also updated the **newuserbatch** script on lab46db to reflect this new information to facilitate things on new user creation.

=====http with ssl=====
I attempted to set up https, which I was making progress on, but initial reactions appeared as if port 443 was not in fact open via the campus connection.

Some informative links:

  * http://www.debian-administration.org/articles/31
  * http://www.debian-administration.org/article/349/Setting_up_an_SSL_server_with_Apache2

Unrelated to SSL, more related to path-based authentications in SVN:

  * http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html
  * http://svnbook.red-bean.com/nightly/en/svn.serverconfig.httpd.html
======January 18th, 2010======
=====Class preparations=====
Inching progressively closer to deploying class syllabi... updated pertinent class informations, created class attendance list files.

At current values, I have 52 total students, and 39 total unique students. It would also appear out of those 39 students, 16 are brand new to Lab46, as they do not yet have accounts.

The following script I wrote to parse through the attendance list information and generate a script-friendly input file for my lab46 user addition facilities:

<code bash>
for user in `cat *spring2011* | sed 's/ /*/g' | tr '\n' ':' | sed 's/:\([A-Z]\)/\n\1/g' | sed 's/:$//g' | sort | uniq`; do
    username="`echo $user | cut -d':' -f2`"
    id $username > /dev/null 2> /dev/null || echo $user | sed 's/^\(.*\):\(.*\)/\2:\1/g' | sed 's/\*/ /g'
done
</code>

====newuserbatch====
With the upgrade to LDAP and the merging of repos into www, I had to update the newuserbatch script on lab46db.

I have successfully tested it with the list of new students for the semester.

In the process, I worked out a few bugs that had existed in the script for quite some time.

=====webdav+svn=====
Slowly making progress on webdav+svn ... I think yesterday I got reading successfully working, and tonight I got writing.

The problem is, I still need to get access control implemented.

I have it authenticating against pam, but I need a way to restrict access per repository to certain users or groups.

Some links that have information:

  * http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=394097
  * http://www.debian-administration.org/articles/208
  * http://www.linuxforu.com/previews/subversion-16-security-improvements-illustrated/
  * http://www.linuxjournal.com/article/7655?page=0,3
  * http://www.howtoforge.com/debian_subversion_websvn
  * http://www.gentoo-wiki.info/HOWTO_Apache2_with_subversion_SVN_and_DAV
======January 17th, 2010======
=====www script migrations=====
I continued the migration of functionality over to the new www... copied scripts and data from /usr/local over, and made pertinent changes (all plain /var/www references changed to /var/www/lab46web).

Also installed lair-mail and lair-backup on the new www, removed oldwww from backup processing.

=====lair-backup possible bug=====
I noticed on some systems the existence of an **/etc/cron.d/backup.orig** file, which is likely a remainder of the package update process... I should put in some script logic to specifically remove this file, as the last thing we want is for the backups to happen //twice//.

=====syllabi started=====
I started preparing my spring 2011 syllabi... beginning with unix.
======January 16th, 2010======
=====new www deployed=====
After two unfruitful weeks of wrestling with cherokee, on a whim of trying to get started on my semester syllabi preparations, I quickly threw it all out the window, installed apache2, and converted over the existing configuration, deploying a new squeeze-based VM for our public web needs.

www also has the functionality of repos built-in, as I hope to also make web-based subversion access a reality, so repos has also been retired. I copied over the ssh keys from repos, so existing subversion checkouts should proceed transparently, and once I get the web-based svn stuff working, it will be a new aspect of awesome.

In the meantime, both wikis have been updated to the latest dokuwiki (2010-11-07)... a quick and effective upgrade process, with seemingly no significant snags.

I'll have to re-implement some of my manual fixes (the calendar hack, the SSO-hack), but it seems as though we are good to go on the wikis. I was even successful in better partitioning the various bits of web data (the main wiki now resides in a /lab46web directory, and with the help of Apache Alias statements, appropriately redirects to other pertinent directories for content rendering).

=====lab46web wiki updated=====
There was a security update to 2010-11-07, which revolved around some xmlrpc functionality I wasn't using, but it was a quick replacement of two files, so that is done. Both wikis are now at the latest and greatest stable release of dokuwiki.

=====Search for broken links=====
I encountered a couple broken links, and I thought to do a more in-depth check to see what else was lingering that might lead to any functionality problems. After a quick google, the following command-line served the purpose nicely:

<cli prompt="# ">
machine:~# ( find / -type l | xargs file ) | grep 'broken symbolic link'
</cli>

It will display each and every file that is identified by **file**(**1**) as a broken link. Amazingly, there were no further ones in my wiki directories.
======January 2nd, 2011======
=====ldapdelete: remove a user or entry in LDAP=====
Rigging up replication between the various auth servers, I ended up obsoleting my first replicating LDAP object (authrepl), and had need to remove it.

Here is the logic to do so:

<cli>
auth1:~# ldapdelete -v "cn=authrepl,dc=lair,dc=bits" -D "cn=admin,dc=lair,dc=bits" -W
Enter LDAP Password: <LDAP_admin_password>
deleting entry "cn=authrepl,dc=lair,dc=bits"
auth1:~# 
</cli>

Should come in handy.

=====lair-backup updated=====
I pushed through some additional tweaks, and also installed it on forklift and auth3.

=====lair-std updated=====
I finally got around to fixing the network detection logic... pushed updates out to everyone.

=====lair-ldap updated=====
With auth3 now online, I updated lair-ldap to include it in the configuration files, and pushed out updates to everyone.

=====auth3.lair.lan=====
I built a new VM, auth3, living on cobras, which serves as a LAIR LDAP consumer on the lair.lan domain. This will make for 3 LDAP servers in the LAIR... two on offbyone.lan (the producer and one consumer), and then this consumer on lair.lan.

This offers LDAP clients LAIR-wide more resiliency in the event of an outage.

======January 1st, 2011======
=====local centos mirror=====
With my continuing work on getting a VM up for bigworld, I decided to investigate setting up a centos mirror...

  * http://www.howtoforge.com/creating_a_local_yum_repository_centos
  * http://nerdicism.com/2009/04/how-to-setup-up-a-private-centos-mirror/
  * http://mirror.r-i-t.edu/centos/5.5/isos/x86_64/

=====lair-backup updated=====
A flurry of additional updates to lair-backup took place today, finally arriving at a configuration that is both stable and resilient. The monthly backups took place this morning, and I was able to sneak in the new lair-backup logic and get a majority of the systems to use it. Next month we should be all set.

=====lair-mail deployed and updated=====
I finished my lair-mail package, and rolled it out to all the various systems. After some initial missing files, I got everything worked out, so nullmailer is now appropriately configured for each system and should forward all mail to mail.offbyone.lan.

=====dokuwiki-status.sh=====
Once again, a bug is present in my dokuwiki status rollover script! When I came to check on my status page, it recorded the previous month link as "200912"... so I know whereabouts to look for the error.

It appears there was a double decrement (I did the "right thing" at the start of the script, then also later on..)... I commented out the second decrement, all should now be fine.

=====journalcacheclear.sh bug=====
Happy New Year! The new year, aside from incrementing the year, brought further stress testing to my various scripts.

One problem that was uncovered was with journalcacheclear.sh on www... turns out it does an explicit check for existing files in a given directory, and now that we're in 2011, the semester logic has kicked over (for one), and as such, the class attendance files do not yet exist.

Therefore, it was throwing errors.

I fixed it by adding in another if statement to check for the existence of the files before plowing ahead. This adequately silenced it (since it runs every 4 minutes or so).

<html><center></html>
^  [[status/status_201012|Previous Month]]  |  ^  [[:status|Current Month]]  |  ^  [[status/status_201102|Next Month]]  |
<html></center></html>