Lab46 Wiki

www.offbyone.lan is the LAIR web server primarily serving the Lab46 and related web and wiki pages.

• Created www on halfadder as a squeeze VM (20110103)
• Deployed this new www as the production www (20110116)
• Created wwwnew as a replacement to www (20140828)

• Lots to do

The following packages have been installed on www:

• lair-std lair-vm lair-nfs lair-autofs lair-ldap
• nginx-extras uwsgi-plugin-ldap uwsgi uwsgi-plugin-python
• php5-ldap php5-mcrypt
• subversion websvn
• php5-cgi php-pear php5-gd php5-mysql
• mercurial
• git

www is a Xen virtual machine. Pertinent configuration information follows:

www.offbyone.lan created on halfadder on 08/28/2014:

sokraits:~# xen-create-image --hostname=wwwnew --mac 00:16:3E:5D:88:23 --role=udev

The Xen config file for this VM is as follows:

##########################################################################
# LAIR Xen VM configuration file (www.offbyone.lan)
##########################################################################
 
#################################################
#  Kernel + memory size
#
bootloader  = '/usr/lib/xen-default/bin/pygrub'
vcpus       = '2'
memory      = '768'
 
#################################################
#  Disk device(s).
#
root        = '/dev/xvda1 ro'
disk        = [ 'file:/xen/images/www.disk,xvda1,w',
                'file:/xen/images/wwwvar.disk,xvda2,w',
                'file:/xen/images/wwwrepos.disk,xvda3,w',
                'file:/xen/images/www.swap,xvda4,w'  ]
 
#################################################
#  Hostname
#
name        = 'www'
 
#################################################
#  Networking
#
dhcp        = 'dhcp'
vif         = [ 'mac=00:16:3E:5D:88:CE' ]
#vif            = [ 'mac=00:16:3E:5D:88:CF' ]
 
#################################################
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

www uses lighttpd as its primary web server.

Residing in /etc/lighttpd/lighttpd.conf, this is the central config for the lighttpd web server.

The default likely works upon installation, but I went and further tweaked it, and removed comments.

lighttpd.conf

server.modules       = ( "mod_access", "mod_alias", "mod_accesslog", "mod_compress" )
server.document-root = "/var/www/"
server.upload-dirs   = ( "/var/cache/lighttpd/uploads" )
server.errorlog      = "/var/log/lighttpd/error.log"
index-file.names     = ( "index.php", "index.html" )
accesslog.filename   = "/var/log/lighttpd/access.log"
url.access-deny      = ( "~", ".inc" )
server.pid-file      = "/var/run/lighttpd.pid"
dir-listing.encoding = "utf-8"
server.dir-listing   = "disable"
server.username      = "www-data"
server.groupname     = "www-data"
compress.cache-dir   = "/var/cache/lighttpd/compress/"
compress.filetype    = ("text/plain", "text/html", "application/x-javascript", "text/css")
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
$HTTP["remoteip"] =~ "127.0.0.1" {
    alias.url += (
        "/doc/" => "/usr/share/doc/",
        "/images/" => "/usr/share/images/"
    )
    $HTTP["url"] =~ "^/doc/|^/images/" {
        dir-listing.activate = "enable"
    }
}

Next up, we wander into /etc/lighttpd/conf-available/, which has the various modules or subconfigurations that can be loaded.

We'll make a new file, and call it 12-mailman.conf:

12-mailman.conf

server.modules += ("mod_redirect", "mod_alias", "mod_cgi", "mod_accesslog")
 
alias.url += (
    "/mailman/images/" => "/var/lib/mailman/icons/",
    "/mailman/pipermail" => "/var/lib/mailman/archives/public/",
    "/mailman" => "/var/lib/mailman/cgi-bin/"
)
 
$HTTP["url"] =~ "^/mailman" {
    server.name = "lab46.corning-cc.edu"
 
    cgi.assign = (
        "/admin" => "",
        "/admindb" => "",
        "/confirm" => "",
        "/create" => "",
        "/edithtml" => "",
        "/listinfo" => "",
        "/options" => "",
        "/private" => "",
        "/rmlist" => "",
        "/roster" => "",
        "/subscribe" => "")
    server.indexfiles = ("listinfo", "index.html")
}
 
$HTTP["url"] =~ "^/mailman/pipermail/" {
    dir-listing.activate = "enable"
    dir-listing.hide-dotfiles = "enable"
    server.follow-symlink = "enable"
}

Following this, we need to enable the mailman “module”, which we can do by issuing the following:

mail:/etc/lighttpd/conf-available$ sudo lighty-enable-mod mailman

These values were tweaked until proven working.

Be sure to restart lighttpd to ensure the settings take effect.

We're actually going to avoid the front-end mailman web interface as much as possible, but just for reference (with our current configuration), it WOULD be at:

http://mail.offbyone.lan/mailman/listinfo

but we'll all predominantly be accessing it via:

http://lab46.corning-cc.edu/mailman/listinfo

because that's how the world can access it.

In figuring this out, I used:

• http://blog.ryara.net/2009/03/27/mailman-with-lighttpd-and-postfix-on-gentoo/ - useful with adaptations
• http://wiki.list.org/display/DOC/Mailman+and+lighttpd (didn't actually end up using)
• http://wiki.list.org/pages/viewpage.action?pageId=7602370 - to convince www to point to us

Useful link:

• http://www.howtoforge.com/debian_subversion_websvn

Installed packages:

• libapache2-svn

Installing libapache2-svn resulted in the following:

www:~# aptitude install libapache2-svn
Setting up libapache2-svn (1.4.2dfsg1-3) ...
Enabling dav as a dependency
Module dav installed; run /etc/init.d/apache2 force-reload to enable.
Module dav_svn installed; run /etc/init.d/apache2 force-reload to enable.

www:~# /etc/init.d/apache2 force-reload
Forcing reload of web server (apache2)... waiting .
www:~# 

Because the world will be accessing the mailman web frontend, it needs to be accessible via 'lab46.corning-cc.edu', ie www's web server.

To accomplish this, we set up proxying via apache to point any mailman-bound requests to mail at the appropriate location, so unless one knew better, one would think the mailman content was hosted locally on www.

First up, we need to enable the apache proxy module. There are several… we are after http proxying, so just to be on the safe side:

www:~$ sudo a2enmod proxy
www:~$ sudo a2enmod proxy_http

Messages confirming the module's enablement will be output (and likely yell saying it is already loaded with at least one of the steps– but hey, we're going for completeness).

Next, we need to allow proxying attempts to get through, which need to be directed at the /etc/apache2/mods-available/proxy.conf file (on www):

proxy.conf

<IfModule mod_proxy.c>
    ProxyRequests Off
 
    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
        Allow from all
        ProxyFtpDirCharset UTF-8
    </Proxy>
    ProxyVia On
</IfModule>

Really the only change made here was the disabling of “Deny from all” and the explicit enabling of “Allow from all”.

Save, and exit.

Next file we're after is where actions on the existing web space are taking place… in time this is likely to evolve, but at the time of writing the following snippet was added at the end of /etc/apache2/httpd.conf on www.

httpd.conf

<IfModule mod_proxy.c>
    ProxyRequests Off
    <Location /mailman/>
        ProxyPass           http://mail.offbyone.lan/mailman/
        ProxyPassReverse    http://mail.offbyone.lan/mailman/
    </Location>
</IfModule>

This is where the real magic takes place– it tells us that any requests to the mailman directory off the documentroot should be directed to mail's web server, at the specified location (for convenience, I just made them match up, but this is only for aesthetics… it could literally be where-ever I'd like it to be).

And there you have it— save, exit, and restart apache.

I didn't confirm this, but everyone seems to say after enabling a module, to make it take effect, you need to do:

www:~$ sudo /etc/init.d/apache2 force-reload

versus just a 'restart' action. I'm pointing it out here because even if there's nothing extra to it than what would have happened during a restart, it still accomplishes the task we want, with some verbage giving us additional certainty that things are in fact being reloaded.

To make php work with lighttpd, we utilize it through the CGI interface. To do this, we install the 'php5-cgi' package.

In /etc/php5/cgi/php.ini, we want to enable the following option (it is in there and default set to zero, set it to one):

cgi.fix_pathinfo = 1

Overall, the settings in /etc/lighttpd/conf-available/10-fastcgi.conf are pretty much exactly as we want them.

I made a few changes:

            "PHP_FCGI_CHILDREN" => "12",
            "PHP_FCGI_MAX_REQUESTS" => "12288"

But aside from that, we could have even kept the defaults and likely wouldn't have had any issues.

Next, we want to enable the 'fastcgi' module in lighttpd:

www:~$ sudo lighty-enable-mod fastcgi
Available modules: auth cgi fastcgi proxy rrdtool simple-vhost ssi ssl status userdir 
Already enabled modules: 
Enabling fastcgi: ok
Run /etc/init.d/lighttpd force-reload to enable changes
www:~$ 

And of course, do as instructed:

www:~$ sudo /etc/init.d/lighttpd force-reload
Stopping web server: lighttpd.
Starting web server: lighttpd.
www:~$ 

web-based mercurial access is done via an apache config AND a cgi script (along with finer grained configuration done via the individual hgrc files in the pertinent repositories.

Mercurial web-based access can be accomplished via their hgweb.cgi script. I configured it as follows (/etc/apache2/conf.d/hgweb.conf):

<Directory /var/repos/hg>
    RewriteEngine On
    RewriteRule ^/hg/(.*) /hg/hgweb.cgi/$1
    AuthType Basic
    AuthName "Lab46/LAIR Mercurial Repository (Authorization Required)"
    AuthPAM_Enabled On
    AuthUserFile /dev/null
    AuthBasicAuthoritative Off
    Require valid-user
    AllowOverride None
    Options ExecCGI FollowSymLinks -MultiViews +SymLinksIfOwnerMatch Indexes
    Order allow,deny
    Allow from all
</Directory>

I had to edit /var/repos/hg/hgweb.cgi and provide an absolute path to the config file, hgweb.config, so the whole file appears as follows (I only had to change the one line:

#!/usr/bin/env python
#
# An example hgweb CGI script, edit as necessary
# See also http://mercurial.selenic.com/wiki/PublishingRepositories

# Path to repo or hgweb config to serve (see 'hg help hgweb')
config = "/var/repos/hg/hgweb.config"

# Uncomment and adjust if Mercurial is not installed system-wide
# (consult "installed modules" path from 'hg debuginstall'):
#import sys; sys.path.insert(0, "/path/to/python/lib")

# Uncomment to send python tracebacks to the browser if an error occurs:
#import cgitb; cgitb.enable()

from mercurial import demandimport; demandimport.enable()
from mercurial.hgweb import hgweb, wsgicgi
application = hgweb(config)
wsgicgi.launch(application)

The hgweb.config file in /var/repos/hg/ looks as follows:

[paths]
/ = /var/repos/hg/*

[web]
baseurl = /hg
descend = True

The following is the hgrc template I am using for user personal mercurial repositories:

[paths]
default = http://www/hg/user/USER

[web]
push_ssl = False
allow_push = USER wedge
allow_read = USER wedge
description = NAME's personal Mercurial Repository
allow_archive = gz, zip, bz2

[ui]
username = NAME <USER@lab46.corning-cc.edu>

[hooks]
pretxncommit.acl = python:hgext.acl.hook
pretxnchangegroup.acl = python:hgext.acl.hook

[acl]
sources = serve push pull

[acl.groups]
users = USER wedge

[acl.deny.branches]
frozen-branch = *

[acl.allow.branches]
* = @users
branch-for-tests =

[acl.allow]
** = @users

The “userdir” module to lighttpd enables this desired functionality.

Let's enable it, and restart the web server:

www:~$ sudo lighty-enable-mod userdir
Available modules: auth cgi fastcgi proxy rrdtool simple-vhost ssi ssl status userdir 
Already enabled modules: fastcgi 
Enabling userdir: ok
Run /etc/init.d/lighttpd force-reload to enable changes
www:~$ sudo /etc/init.d/lighttpd force-reload
Stopping web server: lighttpd.
Starting web server: lighttpd.
www:~$ 

Done!

Since the Lab46 web site is now implemented in dokuwiki, we need to ensure that lighttpd has all the configuration bits in place so that it can successfully serve dokuwiki data.

First up, I made a modular configuration just for dokuwiki stuff, in /etc/lighttpd/conf-available/12-dokuwiki.conf, which contains the following:

12-dokuwiki.conf

## This dokuwiki config provides the functionality for interoperating with
## the dokuwiki wikis we're hosting.
##
## Documentation: http://www.dokuwiki.org/install:lighttpd
##                http://www.dokuwiki.org/rewrite#lighttpd
 
##
## Lab46 wiki
##
 
## subdir of dokuwiki
var.dokudir  = "/var/www"
 
## make sure php files are always served through FastCGI and never as static files
#static-file.exclude-extensions = ( ".php" ) # already set in main config
 
## deny access completely to these
$HTTP["url"] =~ "/(\.|_)ht" { url.access-deny = ( "" ) }
$HTTP["url"] =~ "^" + var.dokudir + "/(bin|data|inc|conf)/"  { url.access-deny = ( "" ) }
 
## rewrites for dokuwiki
$HTTP["url"] =~ "^" + var.dokudir { index-file.names = ("doku.php") }
    url.rewrite = (
      "^" + var.dokudir + "/lib/.*$"              => "$0",
      "^" + var.dokudir + "/_media/(.*)?\?(.*)$"  => var.dokudir + "/lib/exe/fetch.php?media=$1&$2",
      "^" + var.dokudir + "/_media/(.*)$"         => var.dokudir + "/lib/exe/fetch.php?media=$1",
      "^" + var.dokudir + "/_detail/(.*)?\?(.*)$" => var.dokudir + "/lib/exe/detail.php?media=$1&$2",
      "^" + var.dokudir + "/_detail/(.*)?$"       => var.dokudir + "/lib/exe/detail.php?media=$1",
      "^" + var.dokudir + "/_export/([^/]+)/(.*)\?(.*)$" => var.dokudir + "/doku.php?do=export_$1&id=$2&$3",
      "^" + var.dokudir + "/_export/([^/]+)/(.*)" => var.dokudir + "/doku.php?do=export_$1&id=$2",
      "^" + var.dokudir + "/doku.php.*"           => "$0",
      "^" + var.dokudir + "/feed.php.*"           => "$0",
      "^" + var.dokudir + "/(.*)\?(.*)"           => var.dokudir + "/doku.php?id=$1&$2",
      "^" + var.dokudir + "/(.*)"                 => var.dokudir + "/doku.php?id=$1"
    )
 
##
## My (haas/) wiki
##
 
## subdir of dokuwikivar.dokudir2  = "/var/www/haas"
## deny access completely to these
#$HTTP["url"] =~ "/(\.|_)ht" { url.access-deny = ( "" ) }
$HTTP["url"] =~ "^" + var.dokudir2 + "/(bin|data|inc|conf)/"  { url.access-deny = ( "" ) }
 
## rewrites for dokuwiki
$HTTP["url"] =~ "^" + var.dokudir2 { index-file.names = ("doku.php") }
    url.rewrite += (
      "^" + var.dokudir2 + "/lib/.*$"              => "$0",
      "^" + var.dokudir2 + "/_media/(.*)?\?(.*)$"  => var.dokudir2 + "/lib/exe/fetch.php?media=$1&$2",
      "^" + var.dokudir2 + "/_media/(.*)$"         => var.dokudir2 + "/lib/exe/fetch.php?media=$1",
      "^" + var.dokudir2 + "/_detail/(.*)?\?(.*)$" => var.dokudir2 + "/lib/exe/detail.php?media=$1&$2",
      "^" + var.dokudir2 + "/_detail/(.*)?$"       => var.dokudir2 + "/lib/exe/detail.php?media=$1",
      "^" + var.dokudir2 + "/_export/([^/]+)/(.*)\?(.*)$" => var.dokudir2 + "/doku.php?do=export_$1&id=$2&$3",
      "^" + var.dokudir2 + "/_export/([^/]+)/(.*)" => var.dokudir2 + "/doku.php?do=export_$1&id=$2",
      "^" + var.dokudir2 + "/doku.php.*"           => "$0",
      "^" + var.dokudir2 + "/feed.php.*"           => "$0",
      "^" + var.dokudir2 + "/(.*)\?(.*)"           => var.dokudir2 + "/doku.php?id=$1&$2",
      "^" + var.dokudir2 + "/(.*)"                 => var.dokudir2 + "/doku.php?id=$1"
    )

Don't forget to include this as well— “lighty-enable-mod dokuwiki”, and force a server reload.

A certain level of care must be taken, I discovered, when dealing with variable declarations. lighttpd does NOT like any duplicate variable declarations. Anywhere.

The other neat trick that we have in place is that, with mailman running entirely on mail.offbyone.lan, using its own lighttpd web server to host the mailman web pages, we want to rig up that same transparency. lighttpd's mod_proxy fits the bill.

10-proxy.conf

## Let lighttpd act as a proxy server for special file types, hosts etc
##
## Documentation: /usr/share/doc/lighttpd-doc/proxy.txt
##                http://www.lighttpd.net/documentation/proxy.html
 
server.modules   += ( "mod_proxy" )
 
$HTTP["url"] =~ "^/mailman" {
    proxy.server = ( "" =>
        (
            (
                "host" => "10.80.2.17",
                "port" => 80
            )
        )
    )
}

Enable it, force a reload, and we should be good to go.

lighttpd has some status reporting capabilities available via its “status” module, mod_status. I opted to enable it, and lock it down to LAIR-only networks.

The resulting file is as follows:

10-status.conf

## mod_status generates the status overview of the webserver.
##
## Documentation: /usr/share/doc/lighttpd-doc/status.txt
##      http://trac.lighttpd.net/trac/wiki/Docs%3AModStatus
 
server.modules += ( "mod_status" )
 
## relative URL which is used to retrieve the status-page
$HTTP["remoteip"] == "10.80.0.0/16" {
    status.config-url      = "/server-config"
    status.status-url      = "/server-status"
    status.statistics-url  = "/server-statistics"
}

To enable, the usual game.

Then, if you're on one of the LAIR local networks, you can hit up those pages off the documentroot to see their particular content. Should be interesting over time.

One may no longer be necessary, and the other will be going away in the future, but for now we have the following in /etc/rc.local (execute bits set):

rc.local

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
 
# Try to ensure sane NFS environment
/sbin/modprobe nfs4
/sbin/sysctl fs.nfs.nfs_callback_tcpport=2049
/etc/init.d/nfs-common restart
 
# LAIRv1 Legacy Support
/sbin/ifconfig eth0:1 inet 192.168.10.18 netmask 255.255.255.0 up
 
exit 0

• php-cgi under lighttpd
• Setting up lighttpd for dokuwiki
• Tutorial for setting up PHP under lighttpd
• Setting up lighttpd for dokuwiki url rewriting

• http://www.debianhelp.co.uk/apachepam.htm
• http://www.howtoforge.com/debian_subversion_websvn
• http://httpd.apache.org/docs/1.3/urlmapping.html
• http://httpd.apache.org/docs/2.0/mod/core.html