Lab46 Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
=====Overview=====

www.offbyone.lan is the LAIR web server primarily serving the Lab46 and related web and wiki pages.


^  hostname  ^  RAM  ^  disk  ^  swap  ^  OS  ^  Kernel  ^
|  www.offbyone.lan  |  1024MB  |  4GB (/)  |  128MB  |  Debian 8.0 "Jessie" (AMD64)  |  3.14-2-amd64  |
|  :::  |  :::  |  4GB (/var/repos)  |  :::  |  :::  |  :::  |
|  :::  |  :::  |  4GB (/var/www)  |  :::  |  :::  |  :::  |

=====News=====

  * Created www on halfadder as a squeeze VM (20110103)
  * Deployed this new www as the production www (20110116)
  * Created wwwnew as a replacement to www (20140828)
=====TODO=====

  * Lots to do
=====Network Configuration=====

^  Machine  ^  Interface  ^  IP Address  ^  MAC Address  ^  Other Names  ^
|  www.offbyone.lan  |  eth0  |  10.80.2.18  |  00:16:3e:5d:88:ce  |  lab46.corning-cc.edu (for WWW)  |

^  Machine  ^  Interface  ^  IP Address  ^  MAC Address  ^  Other Names  ^
|  wwwnew  |  eth0  |  10.80.2.190  |  00:16:3e:0e:40:23  |    |

=====Packages=====

The following packages have been installed on www:

  * lair-std lair-vm lair-nfs lair-autofs lair-ldap
  * nginx-extras uwsgi-plugin-ldap uwsgi uwsgi-plugin-python
  * php5-ldap php5-mcrypt
  * subversion websvn
  * php5-cgi php-pear php5-gd php5-mysql
  * mercurial
  * git
=====Xen Configuration=====
www is a Xen virtual machine. Pertinent configuration information follows:

====Creation====
www.offbyone.lan created on halfadder on 08/28/2014:

<cli prompt="# ">
sokraits:~# xen-create-image --hostname=wwwnew --mac 00:16:3E:5D:88:23 --role=udev
</cli>

====Configuration====
The Xen config file for this VM is as follows:

<code python>
##########################################################################
# LAIR Xen VM configuration file (www.offbyone.lan)
##########################################################################

#################################################
#  Kernel + memory size
#
bootloader  = '/usr/lib/xen-default/bin/pygrub'
vcpus       = '2'
memory      = '768'

#################################################
#  Disk device(s).
#
root        = '/dev/xvda1 ro'
disk        = [ 'file:/xen/images/www.disk,xvda1,w',
                'file:/xen/images/wwwvar.disk,xvda2,w',
                'file:/xen/images/wwwrepos.disk,xvda3,w',
                'file:/xen/images/www.swap,xvda4,w'  ]

#################################################
#  Hostname
#
name        = 'www'

#################################################
#  Networking
#
dhcp        = 'dhcp'
vif         = [ 'mac=00:16:3E:5D:88:CE' ]
#vif            = [ 'mac=00:16:3E:5D:88:CF' ]

#################################################
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
</code>


=====lighttpd configuration=====
www uses lighttpd as its primary web server.

===main config===
Residing in /etc/lighttpd/lighttpd.conf, this is the central config for the lighttpd web server.

The default likely works upon installation, but I went and further tweaked it, and removed comments.

<file conf lighttpd.conf>
server.modules       = ( "mod_access", "mod_alias", "mod_accesslog", "mod_compress" )
server.document-root = "/var/www/"
server.upload-dirs   = ( "/var/cache/lighttpd/uploads" )
server.errorlog      = "/var/log/lighttpd/error.log"
index-file.names     = ( "index.php", "index.html" )
accesslog.filename   = "/var/log/lighttpd/access.log"
url.access-deny      = ( "~", ".inc" )
server.pid-file      = "/var/run/lighttpd.pid"
dir-listing.encoding = "utf-8"
server.dir-listing   = "disable"
server.username      = "www-data"
server.groupname     = "www-data"
compress.cache-dir   = "/var/cache/lighttpd/compress/"
compress.filetype    = ("text/plain", "text/html", "application/x-javascript", "text/css")
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
include_shell "/usr/share/lighttpd/create-mime.assign.pl"
include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
$HTTP["remoteip"] =~ "127.0.0.1" {
    alias.url += (
        "/doc/" => "/usr/share/doc/",
        "/images/" => "/usr/share/images/"
    )
    $HTTP["url"] =~ "^/doc/|^/images/" {
        dir-listing.activate = "enable"
    }
}
</file>

====CHANGE: lighttpd mailman config====
Next up, we wander into /etc/lighttpd/conf-available/, which has the various modules or subconfigurations that can be loaded.

We'll make a new file, and call it 12-mailman.conf:

<file conf 12-mailman.conf>
server.modules += ("mod_redirect", "mod_alias", "mod_cgi", "mod_accesslog")

alias.url += (
    "/mailman/images/" => "/var/lib/mailman/icons/",
    "/mailman/pipermail" => "/var/lib/mailman/archives/public/",
    "/mailman" => "/var/lib/mailman/cgi-bin/"
)

$HTTP["url"] =~ "^/mailman" {
    server.name = "lab46.corning-cc.edu"

    cgi.assign = (
        "/admin" => "",
        "/admindb" => "",
        "/confirm" => "",
        "/create" => "",
        "/edithtml" => "",
        "/listinfo" => "",
        "/options" => "",
        "/private" => "",
        "/rmlist" => "",
        "/roster" => "",
        "/subscribe" => "")
    server.indexfiles = ("listinfo", "index.html")
}

$HTTP["url"] =~ "^/mailman/pipermail/" {
    dir-listing.activate = "enable"
    dir-listing.hide-dotfiles = "enable"
    server.follow-symlink = "enable"
}
</file>

Following this, we need to enable the mailman "module", which we can do by issuing the following:

<cli>
mail:/etc/lighttpd/conf-available$ sudo lighty-enable-mod mailman
</cli>

These values were tweaked until proven working.

Be sure to restart lighttpd to ensure the settings take effect.


====accessing the web interface====
We're actually going to avoid the front-end mailman web interface as much as possible, but just for reference (with our current configuration), it WOULD be at:

[[http://mail.offbyone.lan/mailman/listinfo]]

but we'll all predominantly be accessing it via:

[[http://lab46.corning-cc.edu/mailman/listinfo]]

because that's how the world can access it.
====useful resources====
In figuring this out, I used:

  * [[http://blog.ryara.net/2009/03/27/mailman-with-lighttpd-and-postfix-on-gentoo/]] - useful with adaptations
  * [[http://wiki.list.org/display/DOC/Mailman+and+lighttpd]] (didn't actually end up using)
  * [[http://wiki.list.org/pages/viewpage.action?pageId=7602370]] - to convince www to point to us

====websvn====
Useful link:
  * http://www.howtoforge.com/debian_subversion_websvn

Installed packages:
  * libapache2-svn

Installing libapache2-svn resulted in the following:

<cli prompt="# ">
www:~# aptitude install libapache2-svn
Setting up libapache2-svn (1.4.2dfsg1-3) ...
Enabling dav as a dependency
Module dav installed; run /etc/init.d/apache2 force-reload to enable.
Module dav_svn installed; run /etc/init.d/apache2 force-reload to enable.

www:~# /etc/init.d/apache2 force-reload
Forcing reload of web server (apache2)... waiting .
www:~# 
</cli>

====UPDATE: Establishing proxying on www to mail====
Because the world will be accessing the mailman web frontend, it needs to be accessible via 'lab46.corning-cc.edu', ie www's web server.

To accomplish this, we set up proxying via apache to point any mailman-bound requests to mail at the appropriate location, so unless one knew better, one would think the mailman content was hosted locally on www.

First up, we need to enable the apache proxy module. There are several... we are after http proxying, so just to be on the safe side:

<cli>
www:~$ sudo a2enmod proxy
www:~$ sudo a2enmod proxy_http
</cli>

Messages confirming the module's enablement will be output (and likely yell saying it is already loaded with at least one of the steps-- but hey, we're going for completeness).

Next, we need to allow proxying attempts to get through, which need to be directed at the /etc/apache2/mods-available/proxy.conf file (on www):

<file apache proxy.conf>
<IfModule mod_proxy.c>
    ProxyRequests Off

    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
        Allow from all
        ProxyFtpDirCharset UTF-8
    </Proxy>
    ProxyVia On
</IfModule>
</file>

Really the only change made here was the disabling of "Deny from all" and the explicit enabling of "Allow from all".

Save, and exit.

Next file we're after is where actions on the existing web space are taking place... in time this is likely to evolve, but at the time of writing the following snippet was added at the end of /etc/apache2/httpd.conf on www.

<file apache httpd.conf>
<IfModule mod_proxy.c>
    ProxyRequests Off
    <Location /mailman/>
        ProxyPass           http://mail.offbyone.lan/mailman/
        ProxyPassReverse    http://mail.offbyone.lan/mailman/
    </Location>
</IfModule>
</file>

This is where the real magic takes place-- it tells us that any requests to the mailman directory off the documentroot should be directed to mail's web server, at the specified location (for convenience, I just made them match up, but this is only for aesthetics... it could literally be where-ever I'd like it to be).

And there you have it--- save, exit, and restart apache.

I didn't confirm this, but everyone seems to say after enabling a module, to make it take effect, you need to do:

<cli>
www:~$ sudo /etc/init.d/apache2 force-reload
</cli>

versus just a 'restart' action. I'm pointing it out here because even if there's nothing extra to it than what would have happened during a restart, it still accomplishes the task we want, with some verbage giving us additional certainty that things are in fact being reloaded.
=====PHP=====
To make php work with lighttpd, we utilize it through the CGI interface. To do this, we install the 'php5-cgi' package.

In /etc/php5/cgi/php.ini, we want to enable the following option (it is in there and default set to zero, set it to one):

<code conf>
cgi.fix_pathinfo = 1
</code>

Overall, the settings in /etc/lighttpd/conf-available/10-fastcgi.conf are pretty much exactly as we want them.

I made a few changes:

<code>
            "PHP_FCGI_CHILDREN" => "12",
            "PHP_FCGI_MAX_REQUESTS" => "12288"
</code>

But aside from that, we could have even kept the defaults and likely wouldn't have had any issues.

Next, we want to enable the 'fastcgi' module in lighttpd:

<cli>
www:~$ sudo lighty-enable-mod fastcgi
Available modules: auth cgi fastcgi proxy rrdtool simple-vhost ssi ssl status userdir 
Already enabled modules: 
Enabling fastcgi: ok
Run /etc/init.d/lighttpd force-reload to enable changes
www:~$ 
</cli>

And of course, do as instructed:

<cli>
www:~$ sudo /etc/init.d/lighttpd force-reload
Stopping web server: lighttpd.
Starting web server: lighttpd.
www:~$ 
</cli>
=====mercurial/hgweb=====

web-based mercurial access is done via an apache config AND a cgi script (along with finer grained configuration done via the individual hgrc files in the pertinent repositories.

====apache config====
Mercurial web-based access can be accomplished via their hgweb.cgi script. I configured it as follows (**/etc/apache2/conf.d/hgweb.conf**):


<code>
<Directory /var/repos/hg>
    RewriteEngine On
    RewriteRule ^/hg/(.*) /hg/hgweb.cgi/$1
    AuthType Basic
    AuthName "Lab46/LAIR Mercurial Repository (Authorization Required)"
    AuthPAM_Enabled On
    AuthUserFile /dev/null
    AuthBasicAuthoritative Off
    Require valid-user
    AllowOverride None
    Options ExecCGI FollowSymLinks -MultiViews +SymLinksIfOwnerMatch Indexes
    Order allow,deny
    Allow from all
</Directory>
</code>

====hgweb.cgi====
I had to edit **/var/repos/hg/hgweb.cgi** and provide an absolute path to the config file, **hgweb.config**, so the whole file appears as follows (I only had to change the one line:

<code>
#!/usr/bin/env python
#
# An example hgweb CGI script, edit as necessary
# See also http://mercurial.selenic.com/wiki/PublishingRepositories

# Path to repo or hgweb config to serve (see 'hg help hgweb')
config = "/var/repos/hg/hgweb.config"

# Uncomment and adjust if Mercurial is not installed system-wide
# (consult "installed modules" path from 'hg debuginstall'):
#import sys; sys.path.insert(0, "/path/to/python/lib")

# Uncomment to send python tracebacks to the browser if an error occurs:
#import cgitb; cgitb.enable()

from mercurial import demandimport; demandimport.enable()
from mercurial.hgweb import hgweb, wsgicgi
application = hgweb(config)
wsgicgi.launch(application)
</code>

====hgweb.config====
The **hgweb.config** file in **/var/repos/hg/** looks as follows:

<code>
[paths]
/ = /var/repos/hg/*

[web]
baseurl = /hg
descend = True
</code>

====hgrc.template====
The following is the hgrc template I am using for user personal mercurial repositories:

<code>
[paths]
default = http://www/hg/user/USER

[web]
push_ssl = False
allow_push = USER wedge
allow_read = USER wedge
description = NAME's personal Mercurial Repository
allow_archive = gz, zip, bz2

[ui]
username = NAME <USER@lab46.corning-cc.edu>

[hooks]
pretxncommit.acl = python:hgext.acl.hook
pretxnchangegroup.acl = python:hgext.acl.hook

[acl]
sources = serve push pull

[acl.groups]
users = USER wedge

[acl.deny.branches]
frozen-branch = *

[acl.allow.branches]
* = @users
branch-for-tests =

[acl.allow]
** = @users
</code>
=====user ~/public_html accessibility=====
The "userdir" module to lighttpd enables this desired functionality.

Let's enable it, and restart the web server:

<cli>
www:~$ sudo lighty-enable-mod userdir
Available modules: auth cgi fastcgi proxy rrdtool simple-vhost ssi ssl status userdir 
Already enabled modules: fastcgi 
Enabling userdir: ok
Run /etc/init.d/lighttpd force-reload to enable changes
www:~$ sudo /etc/init.d/lighttpd force-reload
Stopping web server: lighttpd.
Starting web server: lighttpd.
www:~$ 
</cli>

Done!
=====configuring lighttpd for dokuwiki=====
Since the Lab46 web site is now implemented in dokuwiki, we need to ensure that lighttpd has all the configuration bits in place so that it can successfully serve dokuwiki data.

First up, I made a modular configuration just for dokuwiki stuff, in /etc/lighttpd/conf-available/12-dokuwiki.conf, which contains the following:

<file conf 12-dokuwiki.conf>
## This dokuwiki config provides the functionality for interoperating with
## the dokuwiki wikis we're hosting.
##
## Documentation: http://www.dokuwiki.org/install:lighttpd
##                http://www.dokuwiki.org/rewrite#lighttpd

##
## Lab46 wiki
##

## subdir of dokuwiki
var.dokudir  = "/var/www"

## make sure php files are always served through FastCGI and never as static files
#static-file.exclude-extensions = ( ".php" ) # already set in main config

## deny access completely to these
$HTTP["url"] =~ "/(\.|_)ht" { url.access-deny = ( "" ) }
$HTTP["url"] =~ "^" + var.dokudir + "/(bin|data|inc|conf)/"  { url.access-deny = ( "" ) }

## rewrites for dokuwiki
$HTTP["url"] =~ "^" + var.dokudir { index-file.names = ("doku.php") }
    url.rewrite = (
      "^" + var.dokudir + "/lib/.*$"              => "$0",
      "^" + var.dokudir + "/_media/(.*)?\?(.*)$"  => var.dokudir + "/lib/exe/fetch.php?media=$1&$2",
      "^" + var.dokudir + "/_media/(.*)$"         => var.dokudir + "/lib/exe/fetch.php?media=$1",
      "^" + var.dokudir + "/_detail/(.*)?\?(.*)$" => var.dokudir + "/lib/exe/detail.php?media=$1&$2",
      "^" + var.dokudir + "/_detail/(.*)?$"       => var.dokudir + "/lib/exe/detail.php?media=$1",
      "^" + var.dokudir + "/_export/([^/]+)/(.*)\?(.*)$" => var.dokudir + "/doku.php?do=export_$1&id=$2&$3",
      "^" + var.dokudir + "/_export/([^/]+)/(.*)" => var.dokudir + "/doku.php?do=export_$1&id=$2",
      "^" + var.dokudir + "/doku.php.*"           => "$0",
      "^" + var.dokudir + "/feed.php.*"           => "$0",
      "^" + var.dokudir + "/(.*)\?(.*)"           => var.dokudir + "/doku.php?id=$1&$2",
      "^" + var.dokudir + "/(.*)"                 => var.dokudir + "/doku.php?id=$1"
    )

##
## My (haas/) wiki
##

## subdir of dokuwikivar.dokudir2  = "/var/www/haas"
## deny access completely to these
#$HTTP["url"] =~ "/(\.|_)ht" { url.access-deny = ( "" ) }
$HTTP["url"] =~ "^" + var.dokudir2 + "/(bin|data|inc|conf)/"  { url.access-deny = ( "" ) }

## rewrites for dokuwiki
$HTTP["url"] =~ "^" + var.dokudir2 { index-file.names = ("doku.php") }
    url.rewrite += (
      "^" + var.dokudir2 + "/lib/.*$"              => "$0",
      "^" + var.dokudir2 + "/_media/(.*)?\?(.*)$"  => var.dokudir2 + "/lib/exe/fetch.php?media=$1&$2",
      "^" + var.dokudir2 + "/_media/(.*)$"         => var.dokudir2 + "/lib/exe/fetch.php?media=$1",
      "^" + var.dokudir2 + "/_detail/(.*)?\?(.*)$" => var.dokudir2 + "/lib/exe/detail.php?media=$1&$2",
      "^" + var.dokudir2 + "/_detail/(.*)?$"       => var.dokudir2 + "/lib/exe/detail.php?media=$1",
      "^" + var.dokudir2 + "/_export/([^/]+)/(.*)\?(.*)$" => var.dokudir2 + "/doku.php?do=export_$1&id=$2&$3",
      "^" + var.dokudir2 + "/_export/([^/]+)/(.*)" => var.dokudir2 + "/doku.php?do=export_$1&id=$2",
      "^" + var.dokudir2 + "/doku.php.*"           => "$0",
      "^" + var.dokudir2 + "/feed.php.*"           => "$0",
      "^" + var.dokudir2 + "/(.*)\?(.*)"           => var.dokudir2 + "/doku.php?id=$1&$2",
      "^" + var.dokudir2 + "/(.*)"                 => var.dokudir2 + "/doku.php?id=$1"
    )
</file>

Don't forget to include this as well--- "lighty-enable-mod dokuwiki", and force a server reload.

A certain level of care must be taken, I discovered, when dealing with variable declarations. lighttpd does **NOT** like any duplicate variable declarations. Anywhere.
=====lighttpd proxy for transparent mailman access=====
The other neat trick that we have in place is that, with mailman running entirely on mail.offbyone.lan, using its own lighttpd web server to host the mailman web pages, we want to rig up that same transparency. lighttpd's mod_proxy fits the bill.

<file conf 10-proxy.conf>
## Let lighttpd act as a proxy server for special file types, hosts etc
##
## Documentation: /usr/share/doc/lighttpd-doc/proxy.txt
##                http://www.lighttpd.net/documentation/proxy.html

server.modules   += ( "mod_proxy" )

$HTTP["url"] =~ "^/mailman" {
    proxy.server = ( "" =>
        (
            (
                "host" => "10.80.2.17",
                "port" => 80
            )
        )
    )
}
</file>

Enable it, force a reload, and we should be good to go.

=====enabling lighttpd status output=====
lighttpd has some status reporting capabilities available via its "status" module, mod_status. I opted to enable it, and lock it down to LAIR-only networks.

The resulting file is as follows:

<file conf 10-status.conf>
## mod_status generates the status overview of the webserver.
##
## Documentation: /usr/share/doc/lighttpd-doc/status.txt
##      http://trac.lighttpd.net/trac/wiki/Docs%3AModStatus

server.modules += ( "mod_status" )

## relative URL which is used to retrieve the status-page
$HTTP["remoteip"] == "10.80.0.0/16" {
    status.config-url      = "/server-config"
    status.status-url      = "/server-status"
    status.statistics-url  = "/server-statistics"
}
</file>

To enable, the usual game.

Then, if you're on one of the LAIR local networks, you can hit up those pages off the documentroot to see their particular content. Should be interesting over time.

=====rc.local settings=====
One may no longer be necessary, and the other will be going away in //the future//, but for now we have the following in /etc/rc.local (execute bits set):

<file bash rc.local>
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#

# Try to ensure sane NFS environment
/sbin/modprobe nfs4
/sbin/sysctl fs.nfs.nfs_callback_tcpport=2049
/etc/init.d/nfs-common restart

# LAIRv1 Legacy Support
/sbin/ifconfig eth0:1 inet 192.168.10.18 netmask 255.255.255.0 up

exit 0
</file>
=====References=====

====lighttpd references====
  * [[http://redmine.lighttpd.net/wiki/1/TutorialLighttpdAndPHP|php-cgi under lighttpd]]
  * [[http://www.dokuwiki.org/install:lighttpd|Setting up lighttpd for dokuwiki]]
  * [[http://redmine.lighttpd.net/wiki/lighttpd/TutorialLighttpdAndPHP|Tutorial for setting up PHP under lighttpd]]
  * [[http://www.dokuwiki.org/rewrite#lighttpd|Setting up lighttpd for dokuwiki url rewriting]]

====apache references====
  * http://www.debianhelp.co.uk/apachepam.htm
  * http://www.howtoforge.com/debian_subversion_websvn
  * http://httpd.apache.org/docs/1.3/urlmapping.html
  * http://httpd.apache.org/docs/2.0/mod/core.html