Lab46 Wiki

Run the following command:

apt-get install lam4-dev lam-runtime lam-mpidoc gcc

Create a file anywhere with any name. Just remember where and what it is. This will be a list of the computers to be lambooted. For this guide, the file will be called 'hosts' and be stored in the current user's home directory.

Add every computer to be lambooted to the file. Here's an example:

system0
system1
system3
system4

You can also add them via their IP addresses:

192.168.1.100
192.168.1.101
192.168.1.102
192.168.1.103

run the following command:

lamboot -v ~/hosts

This is an example line, using the example file we set up. Make sure to point it to where you want it. The -v is for verbose mode, meaning it will print out full details of what it's doing.

I'm going to walk you through making, compiling, and running a simple MPI program that will have each system print out a name and number.

Make a new file. It will be ~/hello.c for this example. Copy the contents below into the file:

/*The Parallel Hello World Program*/
#include <stdio.h>
#include <mpi.h>

main(int argc, char **argv)
{
	int node;
	
	MPI_Init(&argc,&argv);
	MPI_Comm_rank(MPI_COMM_WORLD, &node);
	  
	printf("Hello World from Node %d\n",node);
	         
	MPI_Finalize();
}

While in the directory of the file run the following command, substituting in the names you're using:

mpicc -o hello hello.c

Run the following command, again making any proper substitutions:

mpirun -np 4 ./hello

In this example, we've told mpirun to use four of the currently lambooted systems. This number can be changed to anything in the range of the number of systems lambooted. For example, you can run his with a two even if there are more than four systems lambooted.

When you're all done, it's standard procedure to stop the lamboot. To do this, run the following command:

lamhalt -v

You don't need the -v, but it can be useful.

=Xen=

This guide was written after loosely following this other guide. [http://www.howtoforge.com/debian_etch_xen_3.1]

For this guide, there are two ways to install Xen. One is from source, and another is from custom archives made from the first time building from source. Both will be covered in this guide.

Download source code from website and extract it:

cd /usr/src
wget http://bits.xensource.com/oss-xen/release/3.1.0/src.tgz/xen-3.1.0-src.tgz
tar -xvzf xen-3.1.0-src.tgz

Run:

cd xen-3.1.0-src

Edit a line in Config.mk to match below (or appropriate to this effect):

XEN_ENABLE_PAE = n

Run:

time make -j 4 world
cd dist
./install.sh

Change Directory to kernel build folder (wherever that may be):

make menuconfig

* make sure memory support is set to 4GB (non-PAE) * set ID string (typically with name and date) * make any other customizations

Run:

time make -j 4

Copy the appropriate kernel, system map, and .config files to /boot and rename them appropriately.

Run:

make modules_install
mkinitramfs -o /boot/initrd.img-<kernel name> <kernel name>

Run the following commands as root, making the appropriate substitutions for localities:

apt-get install iproute bridge-utils python-twisted binutils zlib1g-dev python-dev transfig
		bzip2 screen ssh debootstrap libcurl3-dev libncurses5-dev x-dev build-essential gettext
cd /usr/src
scp user@location.of:/xensrc.tar .
tar -xvf xensrc.tar
./xen-3.1.0-src/dist/install.sh
rm xensrc.tar
cd /
scp user@location.of:/kernel.tar
tar -xvf kernel.tar
rm kernel.tar

Edit /boot/grub/menu.lst and add the new kernel to the grub menu above the other kernels (near the bottom):

title		Debian GNU/Linux, kernel kernelname
root		(hd0,0)
kernel		/boot/xen-3.1.0.gz
module		/boot/vmlinuz-kernelname root=/dev/hda1 ro console=tty0 max_loop=255
module		/boot/initrd.img-kernelname
savedefault

Run the following commands as root:

depmod kernelname
update-rc.d xend defaults 20 21
update-rc.d xendomains defaults 21 20
mv /lib/tls /lib/tls.disabled
reboot

These commands will make an archive containing the Xen source:

cd /usr/src
tar cvf xensrc.tar xen-3.1.0-src

These commands will make an archive containing the finished kernel and its modules:

cd /
tar cvf kernel.tar /boot/vmlinuz-kernelname /boot/config-kernelname /boot/System.map-kernelname /boot/initrd.img-kernelname /lib/modules/kernelname

Be sure to make all appropriate substitutions for kernelname.

NOTE TO THE READER: This section uses a specific setup as an example.

Run the following commands as root:

apt-get install xen-tools
mkdir /vserver

Edit the /etc/xen-tools/xen-tools.conf file. These are the lines changed for this example:

dir=/vserver
debootstrap=1
size=2Gb
dist=etch
image=full
gateway=137.238.7.254
netmask=255.255.255.0
passwd=1
kernel=/boot/vmlinuz-kernelname
initrd=/boot/initrd.img-kernelname
mirror=http://137.238.7.148:9999/debian/

You can now make the disk image for a xen virtual machine using the config file you just edited as a basis.

xen-create-image --hostname=____ --ip=____ --ide

You now have a xen virtual machine disk image, swap image, and cfg file for your virtual machine.

xm create -c /etc/xen/____.cfg

This will create a virtual machine from the specified cfg file and connect you to the console (-c).

Detach from the console by pressing CTRL+]

xm list

This will list all running virtual machines (and the host).

xm console <name>

This will re-attach yo to a virtual machine's console.

xm save <name>

This will save the current state of a virtual machine. Saved virtual machines are stored in /var/lib/xen/save

xm restore <name>

This will restore a virtual machine from a saved state. Keep in mind that this will not delete the saved state file.

xm shutdown <name>

This will shutdown the virtual machine properly

xm migrate --live <name> <newhost>

This will migrate a virtual machine from one Xen host to another. This is covered in more detail in the next section.

NOTE TO THE READER: This section uses a specific setup as an example.

In order to migrate a virtual machine, the image file for that machine has to be 'locally' accessible to both the current host and the new host. The simplest way to do this would be to have the image files all stored on an NFS server and mount an NFS share at boot time. This is not done using AutoFS, because those mounts are not made until login. It will also not be done using fstab, in order to ensure that the mount would take place long after a network connection is established. So we do the following: * Copy all image files to a folder on the NFS server. * Add that folder as an NFS share. * Edit /etc/rc.local and add an appropriate mount line to the bottom.

Edit the following line in /etc/xen/xend-config.sxp (or something to that effect) from this:

(xend-relocation-hosts-allow '^localhost$')

To something like this:

(xend-relocation-hosts-allow '^localhost$ ^137[.]238[.]7[.][0-9][0-9][0-9]$ ^shiznit[0-9][.]ds[.]geneseo[.]edu$')

=Various Tidbits=

This guide is specifically for a Dell Optiplex GX260 and it's integrated (see crappy) graphics.

Install Xorg (as root) with the following command:

apt-get install xorg

Have X handle most of the configuration on its own (thus helping you out) with the following command:

Xorg -configure

Still having problems? That's because Xorg rarely configures things properly. The following changes have to be made to the newly made /root/xorg.conf.new file. (Sorry about the poor readability)

...
Section "Input Device"
	...
	Option	"Device" "/dev/''psaux''"
	...
EndSection

Section "Monitor"
	...
	ModelName	"Monitor Model"
	''HorizSync 31.5-79.0''
	''VertRefresh 56.0-76.0''
	''Option "DPMS"''
	''Modeline "1280x1024" 109.62 1280 1336 1472 1720 1024 1024 1026 1062''
End Section
...
Section "Device"
	...
	Driver		"i810"
	''VideoRam	8192''
	...
	BusID       "PCI:0:2:0"
	''Option "UseFBDev" "true"''
	''Option "VBERestore" "true"''
EndSection

Section "Screen"
Add line:
		''Modes  	"1280x1024" "1024x768" "800x600"''
to each SubSection "Display"

Now, copy that configuration to the proper place (so that you don't have to always tell X what configuration to use) with the following command:

cp /root/xorg.conf.new /etc/X11/xorg.conf

* It might be wise to back up the original configuration beforehand.

Now, just run:

startx

If you see a grey screen with a black X in the middle, it works! Continue tweaking it to your heart's content. Once you get it working, try installing an X Window Manager!

Synergy is a fun utility that allows a computer (the synergy server) to share its mouse and keyboard with other computers (the clients). It is cross-platform and allows you to use one keyboard and mouse across multiple machines. It's fairly simple to set up. Included here are a few links to places that describe how to do so. * The Article that led me to looking at Synergy in the first place [http://lifehacker.com/software/top/hack-attack-control-multiple-computers-with-a-single-keyboard-and-mouse-254648.php] * The Synergy Source Forge Project Page. Contains Source, Unix Binaries, and Windows GUI [http://sourceforge.net/projects/synergy2/] * SynergyKM, a GUI tie-in for the Mac [http://software.landryhetu.com/synergy/] * Nice Descriptive Synergy Guide [http://www.linux.com/articles/54628]

Edit the file ~/.bashrc * Add the following line to the bottom of the file.

alias name=’command’

* Here are a few useful alias examples.

alias ls='ls -a --color=auto'
alias vi='vim'
alias ftp=’ftp -v’
alias lamboot='lamboot -v'
alias lamhalt='lamhalt -v'
alias grep='grep --color=auto'

For a Mac OS X installation, edit the ~/.profile file. It will probably not exist yet.

Edit (or create) the file ~/.vimrc * Add whatever you want as VIM defaults to the file. An example file is as follows:

syn on
set tabstop=4
set shiftwidth=4

Run the following commands:

apt-get install ntpdate
ntpdate ntp.geneseo.edu

There are two login messages, one before the login prompt, and one after. The one before is only seen on the local machine, but the one after is also seen during ssh sessions.

The message before the login prompt can be found and edited in the /etc/issue file.

The message after the login prompt can be found and edited in the /etc/motd.tail file. * The /etc/motd file is reset at boot time, using the contents of motd.tail

If there's ever a problem trying to work with a display, got onto that machine and run:

xhost +

That might do the trick.

Commands to try showing off: * xeyes * xloadimage * xsetroot -solid “{COLOR}” * any non-openGL xscreensaver (run on root screen with -root)

* vim * screen * ssh * ftp * less * bsdgames * ntpdate * pciutils

Restore current screen session:

screen -r

Detach from current screen session:

ctrl-a d

List all currently open screen sessions:

ctrl-a "

Create a new screen session:

ctrl-a c

The new universe in the DSLAB will use the internal domain name “dslab.lan”, and complements the LAIR's current domain name “offbyone.lan”.

juicebar (10.81.1.1) is the current DNS server, so perform all changes there for now.

Files of interest on juicebar include:

/var/named/

This is the location of the name server configuration files, as well as the actual DNS zone definitions.

/var/named/master    -  Zone definitions that the current server is responsible for.
/var/named/slave       -  Zone definitions that the current server is also hosting.
/var/named/standard -  Zone definitions for basic DNS services (loopback, etc.).

The only place where you'll ever have a need to go is into master. Any changes made to the slave configurations will probably be changed when the servers synchronize.

Inside /var/named/master are (at least) two files:

10.81.1 - reverse lookup
dslab.lan   - domain definition

dslab.lan defines all the names that are present in the dslab.lan domain, as well as what IP they should be mapped to.

10.81.1 does the opposite. It has the last octet of the IP address that it uses to look up the domain (reverse lookup).

Any changes/additions/modifications must be made, as appropriate, to BOTH files. Follow the existing format.

ADDITIONALLY: Any changes that are made, you MUST update the serial number. This is how the servers recognize changes. If you make changes but do not update the serial number, confusion may occur!

Standard format for the serial number is to use the date. For example, if the date is August 2, 2007, and this is your first change of the day, the serial number should be:

2007080201

If you make additional changes during the same day, obviously the date would not have changed, but you can increment the last 2 digits by one… so the second change would appear as:

2007080202

…and so on. This way, you can make up to 99 changes per day.

When you've completed changes to both files (good practice is to synchonize the serial numbers in both), be sure to restart the service:

juicebar:~# dhcpupdate

No changes need to be done to synchronize, although it may take some time for changes to be recognized by the other DNS server, or even other machines on the network, as DNS requests are commonly cached to speed up lookups.

In conjunction with DNS, DHCP helps to automate the network configuration and location of machines.

Currently, juicebar (10.81.1.4) is also running the DHCP server.

First off, the relevant file for DHCP is:

/etc/dhcpd.conf

Editing this file shows a bunch of information… perhaps slowly read over it to try and make any sense of it all.

The big focus will be the actual machine entries. Scroll/page down until you find the Dell cluster entries.. node132's is presented here as an example:

              host node132.dslab.lan {
                      hardware ethernet 00:08:74:D1:64:BD;
                      fixed-address node132.dslab.lan;
              }

Note the lack of any IP address– this is provided via DNS. However, do note the presence of the MAC address. This is how each machine will receive the same IP each time.

So.. to make additional entries, merely make a copy of an existing entry and fill it out appropriately; changing the host and fixed-address, and putting in the correct MAC address of the machine.

When finished, restart DHCP:

juicebar:~# dhcpupdate

If there are errors, DHCP may fail to start. You can check the system logs (/var/log/daemon.log) to see if there are any problems, and they'll usually indicate the line number of the error.

There exists a VPN server for DSLAB users to connect to. Following will be information to assist in establishing a working configuration, certificate/key, and hopefully a working connection. This information is provided from the point of view of the server– specific client deployments may vary (Linux is easy, MacOS X also somewhat, Windows… good luck).

I've set up a sample config file that can be modified for use. Please substitute all occurrences of the string “USER” with the actual user name associated with the client certificate/key.

You will want to make a COPY of this file, and place it on YOUR machine that will be attempting to VPN in… your specific config file does NOT need to reside on juicebar. The only reason it is here is so you can obtain a copy to use.

##############################################################################
#
#   DSLAB OpenVPN Client Configuration File (sample)
#
#   This configuration is to facilitate the joining of the DSLAB VPN.
#
#   Please replace all instances of USER with the actual user name (also the
#   name on the VPN certificate/key).
#
##############################################################################

##############################################################################
#   VPN Server Information
##############################################################################
remote          137.238.7.4             # IP of remote OpenVPN server
port            1194                    # Port on which to connect on server
proto           udp                     # Type of traffic {tcp-client|udp}

##############################################################################
#   Network Interfaces
##############################################################################
dev-type        tap                     # Type of interface to use {tap|tun}
dev             tap0                    # Interface name (tap0 or tun0)

##############################################################################
#   Credentials
##############################################################################
cd              /etc/openvpn            # establish proper working directory
key             dslab/client-USER.key   # Server key (private)
ca              dslab/ca.crt            # Certificate (public)
cert            dslab/client-USER.crt   # Server Cert (private)
tls-cipher      EDH-RSA-DES-CBC3-SHA    # set tls cipher type

##############################################################################
#   Client Settings
##############################################################################
comp-lzo                                # use fast LZO compression
keepalive       10      120             # send packets to keep sessions alive
nobind                                  # don't bind to local address & port
persist-key                             # don't re-read keys across restarts
persist-tun                             # on restart, don't reset tun device
pull                                    # Follow route suggestions of server
resolv-retry    infinite                # keep trying to connect if failure
route-delay     8                       # delay setting routes for 8 seconds
tls-client                              # enable TLS and assume client role

##############################################################################
#   System Options
##############################################################################
chroot          /etc/openvpn            # run in a chroot of VPN directory
user            nobody                  # after launching, drop privs
group           nobody                  # after launching, drop privs
daemon                                  # detach and run in background

##############################################################################
#   Verbosity/Logging Options
##############################################################################
#status         log/status.log          # status log file
log-append      log/backbone.log        # log file
verb            3                       # level of activity to log (0-11)
mute            20                      # log at most N consecutive messages

##############################################################################

On the client system, you'll need a place to put the keys/cert, config, and have log files stored… on Linux systems, a good place to put this would be in /etc/openvpn. A recommended directory layout would be as follows:

/etc/openvpn/
/etc/openvpn/client-USER.conf      # again, please replace 'USER' with your username
/etc/openvpn/dslab/
/etc/openvpn/dslab/ca.key
/etc/openvpn/dslab/client-USER.key
/etc/openvpn/dslab/client-USER.crt
/etc/openvpn/log/

Creating any of these directories that don't exist would be a very good thing for ultimate success.

We will need to create the necessary authenticating bits, which can be done on juicebar (as root). Follow the example below:

First up, go to the right place and get the variables loaded:

juicebar:~# cd /etc/openvpn/easy-rsa/
juicebar:/etc/openvpn/easy-rsa# . ./vars
NOTE: If you run ./clean-all, I will be doing a rm -rf on /etc/openvpn/easy-rsa/keys
juicebar:/etc/openvpn/easy-rsa#

NOTE: Make sure you do *NOT* run ./clean-all, as that will delete all the keys, and we'd have to start again from scratch.

NOTE: Really, do *NOT* run ./clean-all, it would be VERY BAD (well, we'd have to recreate certs/keys for EVERYONE, which would not be a fun time). So DON'T. Please.

Next, let's create a key (we'll use USER as the example… replace the desired client name (typically a username or short identifying word in place of USER– common practice is just to use your normal username, just substitute 'USER' in the example below with your actual username):

juicebar:/etc/openvpn/easy-rsa# ./build-key client-USER
Generating a 1024 bit RSA private key
...................................................................................................++++++
.....++++++
writing new private key to 'client-USER.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:
State or Province Name (full name) [NY]:
Locality Name (eg, city) [Upstate]:
Organization Name (eg, company) [BITS]:
Organizational Unit Name (eg, section) []:DSLAB
Common Name (eg, your name or your server's hostname) [client-USER]:
Email Address [haas@corning-cc.edu]:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
Using configuration from /etc/openvpn/easy-rsa/openssl.cnf
DEBUG[load_index]: unique_subject = "yes"
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :PRINTABLE:'NY'
localityName          :PRINTABLE:'Upstate'
organizationName      :PRINTABLE:'BITS'
organizationalUnitName:PRINTABLE:'DSLAB'
commonName            :PRINTABLE:'client-USER'
emailAddress          :IA5STRING:'haas@corning-cc.edu'
Certificate is to be certified until Apr 25 02:47:12 2019 GMT (3650 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
juicebar:/etc/openvpn/easy-rsa#

Finally, tar up the necessary files:

juicebar:/etc/openvpn/easy-rsa# cd keys
juicebar:/etc/openvpn/easy-rsa/keys# tar cvf archive/client-USER.tar client-USER.crt client-USER.key ca.crt
client-USER.crt
client-USER.key
ca.crt
juicebar:/etc/openvpn/easy-rsa/keys# gzip -9 archive/client-USER.tar
juicebar:/etc/openvpn/easy-rsa/keys#

Distribute it (archive/client-USER.tar.gz) to the appropriate client (along with a custom config file, named appropriately) so it can be used and great happiness with ensue.

juicebar:~# cd /etc/openvpn/ccd
juicebar:/etc/openvpn/ccd# cat client-template
push "route 10.80.1.0 255.255.255.0 10.81.1.1"
push "route 10.80.2.0 255.255.255.0 10.81.1.1"
push "route 10.80.3.0 255.255.255.0 10.81.1.1"
juicebar:/etc/openvpn/ccd# cp client-template client-USER

The OpenVPN probably doesn't need any restarting. If you SIGHUP it, it will fail to restart due to having dropped root privs, so you'd need to start it up again.

NOTE: As I just said, don't try to restart the OpenVPN daemon. It is set.

It is often helpful, while diagnosing a new client connection, to check the logs. Checking them on the server can be done as follows (as root):

juicebar:~# cd /etc/openvpn/log
juicebar:/etc/openvpn/log#

There are 2 files normally in this directory:

backbone.log
dslab.log

The file “backbone.log” is for the DSLAB-LAIR VPN connection. Any new VPN connections will NOT be using this, so looking there will not help you.

You will want to tail the “dslab.log” file, which can be done as follows:

juicebar:/etc/openvpn/log# tail -f dslab.log

Your terminal will then contain the end of the client connecting log file… have this running, and then attempt a client connection- you should see messages appear, which MIGHT prove of some value if the connection doesn't work as expected.

There are also log files created on the client's computer (under the “log/” subdirectory– which would be in the same place as the client config file, and next to the “dslab/” directory you'll have to make), and you can “tail -f” that/those files as well to ascertain that side of the connection.

The CoRAID SR420 is our new file storage array. Utilizing the ATA-over-Ethernet (AoE) protocol, it allows for shared storage amongst several machines.

Helping to provide that shared storage is the current use of OCFS2 (the Oracle Cluster FileSystem, version 2), which is a cluster filesystem capable of having multiple entities (peers) read/write to the same volume.

While it still needs more thorough testing, here is the low-down on its current configuration.

make 0 raid5 0.0 0.1 0.2 0.3

Creates logical blade 0 incorporating drives 0.0, 0.1, 0.2, 0.3

Don't forget to put it online:

online 0

That way, you can now list it.

Debian provides a package for AoE. Installing it, along with having a compiled kernel module in place, is all that is needed.

machine:~# apt-get install aoetools

Debian provides a package for OCFS2 as well:

machine:~# apt-get install ocfs2-tools

Be sure to have the kernel module compiled.

To check to see if there are any AoE devices on the network, perform the following:

machine:~# aoe-discover

Which will send out a packet broadcasting any AoE-compliant devices to identify themselves. Then:

machine:~# aoe-stat
           e0.0      2250.468GB   eth0 up

Which gives us the status of any detected AoE devices. In this case, we see (as e0.0) the 2.1TB volume on the CoRAID that was detected over the current machine's eth0 network interface.

The directory:

/dev/etherd/

Contains various AoE-related devices, and amongst them should be an “e0.0” device, which is the block device for the 2.1TB volume.

To mount the volume, simply do the following:

machine:~# mount /dev/etherd/e0.0 /mnt

Or whereever you wish to mount it. It may take a few seconds (this seems to be normal), but then it will mount. It can then be accessed. You can also concurrently mount it on other machines and go about as normal.

This does assume, of course, that the volume is formatted with a cluster filesystem (ie OCFS2) so that all chaos does not break loose.

For starters, there are really only 2 files that need to be created/configured.

/etc/ocfs2/cluster.conf
/etc/default/o2cb

We'll start with o2cb first:

node134:~# cat /etc/default/o2cb
#
# This is a configuration file for automatic startup of the O2CB
# driver.  It is generated by running 'dpkg-reconfigure ocfs2-tools'.
# Please use that method to modify this file.
#

# O2CB_ENABLED: 'true' means to load the driver on boot.
O2CB_ENABLED=true

# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
O2CB_BOOTCLUSTER=data

# O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead.
O2CB_HEARTBEAT_THRESHOLD=7

Of importance– setting O2CB_ENABLED to true (it is false by default) will cause the appropriate services to start up on boot.

O2CB_BOOTCLUSTER should also be set to the name of the OCFS2 “cluster”, which for now has been named “data”.

Next, we have cluster.conf:

node:
      ip_port = 7777
      ip_address = 192.168.6.132
      number = 0
      name =  node132
      cluster = data

node:
      ip_port = 7777
      ip_address = 192.168.6.133
      number = 1
      name =  node133
      cluster = data

...

node:
      ip_port = 7777
      ip_address = 192.168.6.147
      number = 15
      name =  node147
      cluster = data

cluster:
      node_count = 16
      name = data

Here, we set up all the machines that will be a part of accessing the shared volume. Each machine is given an entry and assigned a number.

Also note the cluster: directive at the very end indicating total node count as well as the name of the OCFS2 “cluster”.

IMPORTANT: Changes to cluster.conf MUST be propagated to ALL PARTICIPATING MEMBERS. I would imagine unmounting/shutting down OCFS2 services on each machine would be a very prudent thing to do. This is the complexity of the cluster filesystem– it MUST be aware of anyone who can touch it, lest weird things happen.

The script:

/etc/init.d/o2cb

Can be used to start, stop, check status of the OCFS2 cluster.

So, to bring it online on a particular machine, do the following:

/etc/init.d/o2cb start

There is a status argument that can be passed as well:

machine:~# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking cluster data: Online
Checking heartbeat: Not active

The output given is what appears to be an operational output (not sure about the heartbeat setting— something to look into).

And, of course, one can stop O2CB services:

machine:~# /etc/init.d/o2cb stop

OCFS2 has some utilities that can be used to administer certain functions, such as formatting, filesystem checking, and filesystem tuning.

Here is what I did to format the 2.1TB volume on the CoRAID with OCFS2:

machine:~# mkfs.ocfs2 -v -N 24 /dev/etherd/e0.0

Note that if you leave off the “-N 24” the filesystem will default to only 4 nodes. What happens here is when more than 4 machines try to mount, further mount attempts are refused once the limit is reached.

We will have to test and see what sort of performance impact more mounted OCFS2 nodes have on the whole situation.

The tunefs.ocfs2 tool can be used to change settings on an existing OCFS2 volume.

Here is what I did to extend the number of nodes from 4 to 24 (because I didn't originally do it, but my example above does, which would obsolete this step):

machine:~# tunefs.ocfs2 -N 24 -v /dev/etherd/e0.0

Why 24? Well, we've got the 16 video wall machines, and I figured I'd add in some breathing room in case I wanted to test things. Overall, though, I would imagine that the fewer machines mounting the volume, the better. Only further testing will yield what the optimal resulting configuration will end up being.

If you encounter kernel log messages of the following:

peth0: received packet with  own address as source address

It means there are bridge MAC addresses that are the same among more than one machine. No idea why this springs up. But, the way to fix it is to change the MAC address of the bridge when it comes up.

There is a modified /etc/xen/scripts/network-bridge script in place on the Dell cluster which has a user-specified value to insert into the MAC, thereby making it different.

Setting this value and rebooting the machine tends to clear up the issue.

D0    node149
D1    node152
D2    node148
E0    node132
E1    node133
E2    node134
E3    node135
E4    node136
E5    node137
E6    node138
E7    node139
E8    node140
E9    node141
EA    node142
EB    node143
EC    node144
ED    node145
EE    node146
EF    node147