haas:system:sokraits_halfadder.offbyone.lan
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| haas:system:sokraits_halfadder.offbyone.lan [2014/08/07 15:49] – [Setting /etc/mdadm/mdadm.conf] wedge | haas:system:sokraits_halfadder.offbyone.lan [2014/10/04 14:41] (current) – [rsync to fileserver] wedge | ||
|---|---|---|---|
| Line 4: | Line 4: | ||
| ^ hostname | ^ hostname | ||
| - | | sokraits.lair.lan | + | | sokraits.lair.lan |
| | ::: | ::: | 500GB + 500GB RAID1 (/ | | ::: | ::: | 500GB + 500GB RAID1 (/ | ||
| ^ hostname | ^ hostname | ||
| - | | halfadder.lair.lan | + | | halfadder.lair.lan |
| | ::: | ::: | 500GB + 500GB RAID1 (/ | | ::: | ::: | 500GB + 500GB RAID1 (/ | ||
| Line 21: | Line 21: | ||
| * Re-re-reinstalled sokraits with Debian Wheezy -> Jessie, due to failed boot drive (20140806) | * Re-re-reinstalled sokraits with Debian Wheezy -> Jessie, due to failed boot drive (20140806) | ||
| * Re-installed halfadder with Debian Jessie (20140806) | * Re-installed halfadder with Debian Jessie (20140806) | ||
| + | * Re-re-re-resetup sokraits as a clone of halfadder and netbooting with entire system in initrd, due to (another) failed boot drive (20141004) | ||
| =====TODO==== | =====TODO==== | ||
| - | * rig up ramdisk /var and /tmp w/ periodic writes (since we have an SSD /). | + | * <del>rig up ramdisk /var and /tmp w/ periodic writes (since we have an SSD /).</ |
| * find 3.5" to 5.25" drive brackets and remount sokraits data drives in case. | * find 3.5" to 5.25" drive brackets and remount sokraits data drives in case. | ||
| * on next halfadder reboot, verify that OCFS2 /export gets mounted automatically (last time I had to run "/ | * on next halfadder reboot, verify that OCFS2 /export gets mounted automatically (last time I had to run "/ | ||
| Line 122: | Line 123: | ||
| =====apt configuration===== | =====apt configuration===== | ||
| + | ====use LAIR apt proxy==== | ||
| + | To reduce traffic caused by package transactions, | ||
| + | |||
| + | < | ||
| + | Acquire:: | ||
| + | </ | ||
| + | |||
| + | ====no recommends==== | ||
| I wanted a small installation footprint, so I disabled the installation of recommended packages by default. | I wanted a small installation footprint, so I disabled the installation of recommended packages by default. | ||
| Line 156: | Line 165: | ||
| ocfs2-tools | ocfs2-tools | ||
| smartmontools | smartmontools | ||
| + | firmware-realtek | ||
| + | qemu-system-x86-64 | ||
| =====GRUB Configuration===== | =====GRUB Configuration===== | ||
| Line 269: | Line 279: | ||
| ## | ## | ||
| arch = amd64 | arch = amd64 | ||
| - | # | ||
| mirror | mirror | ||
| mirror_squeeze | mirror_squeeze | ||
| - | # | + | mirror_wheezy |
| - | mirror_wheezy | + | mirror_jessie |
| - | # | + | |
| - | #mirror_jessie | + | ###################################################################### |
| - | mirror_jessie | + | ## |
| + | ## Proxy Settings for repositories | ||
| + | ## | ||
| + | apt_proxy | ||
| ###################################################################### | ###################################################################### | ||
| Line 365: | Line 377: | ||
| The purpose of the disk array is to provide RAID1 (mirror) to the Xen VM images. | The purpose of the disk array is to provide RAID1 (mirror) to the Xen VM images. | ||
| + | ====Re-initializing==== | ||
| + | As we've had functioning RAID volumes for years, I thought I would re-do the arrays so as to take advantage of any new version features (when I first created them, mdadm was at version 0.8- now it is 1.2). | ||
| + | |||
| + | So, I first stopped the array: | ||
| + | |||
| + | <cli> | ||
| + | sokraits:~# mdadm --stop /dev/md0 | ||
| + | </ | ||
| + | |||
| + | Then, I zeroed out the superblocks on both constituent drives: | ||
| + | |||
| + | <cli> | ||
| + | sokraits:~# mdadm --zero-superblock /dev/sdb | ||
| + | sokraits:~# mdadm --zero-superblock /dev/sdc | ||
| + | </ | ||
| + | |||
| + | Now we can proceed with creating the new array. | ||
| ====creating / | ====creating / | ||
| I opted to build the array straight to disk-- no messing with partition tables. | I opted to build the array straight to disk-- no messing with partition tables. | ||
| Line 370: | Line 399: | ||
| <cli prompt="# | <cli prompt="# | ||
| halfadder: | halfadder: | ||
| - | mdadm: partition table exists on /dev/sdb but will be lost or | ||
| - | | ||
| - | mdadm: Note: this array has metadata at the start and | ||
| - | may not be suitable as a boot device. | ||
| - | store '/ | ||
| - | your boot-loader understands md/v1.x metadata, or use | ||
| - | --metadata=0.90 | ||
| - | mdadm: partition table exists on /dev/sdc but will be lost or | ||
| - | | ||
| - | Continue creating array? y | ||
| - | mdadm: Defaulting to version 1.2 metadata | ||
| - | mdadm: array /dev/md0 started. | ||
| - | halfadder: | ||
| - | </ | ||
| - | |||
| - | <cli> | ||
| - | root@sokraits:/ | ||
| mdadm: /dev/sdb appears to be part of a raid array: | mdadm: /dev/sdb appears to be part of a raid array: | ||
| | | ||
| Line 403: | Line 415: | ||
| mdadm: Defaulting to version 1.2 metadata | mdadm: Defaulting to version 1.2 metadata | ||
| mdadm: array /dev/md0 started. | mdadm: array /dev/md0 started. | ||
| + | halfadder: | ||
| </ | </ | ||
| + | |||
| + | |||
| ====checking disk array status==== | ====checking disk array status==== | ||
| To check the status: | To check the status: | ||
| Line 483: | Line 498: | ||
| # definitions of existing MD arrays | # definitions of existing MD arrays | ||
| - | ARRAY /dev/md/0 metadata=1.2 UUID=a0e64ddc:8cf6de4c:6e01f619:e9474564 | + | ARRAY /dev/md/0 metadata=1.2 UUID=c846eb24:6b9783db:cd9b436c:8470fd46 |
| # This file was auto-generated on Wed, 06 Aug 2014 11:17:46 -0400 | # This file was auto-generated on Wed, 06 Aug 2014 11:17:46 -0400 | ||
| Line 493: | Line 508: | ||
| <cli prompt="# | <cli prompt="# | ||
| - | halfadder: | + | halfadder: |
| + | /dev/md0: | ||
| Version : 1.2 | Version : 1.2 | ||
| - | Creation Time : Wed Nov 17 16:43:40 2010 | + | Creation Time : Thu Aug 7 11:50:06 2014 |
| Raid Level : raid1 | Raid Level : raid1 | ||
| - | Array Size : 488385424 | + | Array Size : 488255488 |
| - | Used Dev Size : 488385424 | + | Used Dev Size : 488255488 |
| Raid Devices : 2 | Raid Devices : 2 | ||
| Total Devices : 2 | Total Devices : 2 | ||
| Persistence : Superblock is persistent | Persistence : Superblock is persistent | ||
| - | | + | Intent Bitmap : Internal |
| - | State : clean | + | |
| + | | ||
| + | State : active | ||
| | | ||
| Working Devices : 2 | Working Devices : 2 | ||
| Line 511: | Line 529: | ||
| Name : halfadder: | Name : halfadder: | ||
| - | UUID : a0e64ddc:8cf6de4c:6e01f619:e9474564 | + | UUID : c846eb24:6b9783db:cd9b436c:8470fd46 |
| - | | + | |
| Number | Number | ||
| Line 526: | Line 544: | ||
| <cli prompt="# | <cli prompt="# | ||
| - | BOTH:~# rm -f / | ||
| BOTH:~# update-initramfs -t -u -k all | BOTH:~# update-initramfs -t -u -k all | ||
| update-initramfs: | update-initramfs: | ||
| Line 537: | Line 554: | ||
| With the tools installed, we need to configure some files. | With the tools installed, we need to configure some files. | ||
| - | ====/ | + | ====/ |
| - | First up, we need to get the peers talking so we can form the volume and get OCFS2 established. Let's make the **/ | + | First up, we need to get the peers talking so we can form the volume and get OCFS2 established. Let's make the **/etc/drbd.d/ |
| < | < | ||
| Line 566: | Line 583: | ||
| protocol C; | protocol C; | ||
| } | } | ||
| + | </ | ||
| + | This is only an intermediate step. Further changes are needed before we can bring it up in dual-primary mode. | ||
| + | |||
| + | ====/ | ||
| + | And the resource configuration (doesn' | ||
| + | |||
| + | < | ||
| resource xen_data | resource xen_data | ||
| { | { | ||
| + | device | ||
| + | disk /dev/md0; | ||
| + | meta-disk | ||
| + | |||
| on sokraits | on sokraits | ||
| { | { | ||
| - | | + | address |
| - | disk /dev/md0; | + | |
| - | | + | |
| - | meta-disk | + | |
| } | } | ||
| on halfadder | on halfadder | ||
| { | { | ||
| - | | + | address |
| - | disk /dev/md0; | + | |
| - | | + | |
| - | meta-disk | + | |
| } | } | ||
| } | } | ||
| </ | </ | ||
| - | |||
| - | This is only an intermediate step. Further changes are needed before we can bring it up in dual-primary mode. | ||
| - | |||
| ====bootstrapping DRBD==== | ====bootstrapping DRBD==== | ||
| Getting DRBD initially up-and-running has always has a bit of voodoo behind it... trying a number of commands and eventually stumbling upon something that works. I may have finally gotten the procedure down: | Getting DRBD initially up-and-running has always has a bit of voodoo behind it... trying a number of commands and eventually stumbling upon something that works. I may have finally gotten the procedure down: | ||
| Line 626: | Line 645: | ||
| sokraits:~# drbdadm -- --overwrite-data-of-peer primary xen_data | sokraits:~# drbdadm -- --overwrite-data-of-peer primary xen_data | ||
| </ | </ | ||
| + | |||
| + | Of course, this isn't so willy nilly if one of the peers has the more up-to-date copy of the data. | ||
| Upon which we can now view /proc/drbd and see messages like: | Upon which we can now view /proc/drbd and see messages like: | ||
| Line 713: | Line 734: | ||
| node: | node: | ||
| ip_port = 7777 | ip_port = 7777 | ||
| - | ip_address = 10.80.1.46 | + | ip_address = 172.16.1.1 |
| number = 0 | number = 0 | ||
| name = sokraits | name = sokraits | ||
| Line 720: | Line 741: | ||
| node: | node: | ||
| ip_port = 7777 | ip_port = 7777 | ||
| - | ip_address = 10.80.1.47 | + | ip_address = 172.16.1.2 |
| number = 1 | number = 1 | ||
| name = halfadder | name = halfadder | ||
| Line 729: | Line 750: | ||
| name = datastore | name = datastore | ||
| </ | </ | ||
| - | + | ====/ | |
| - | ====/ | + | |
| Once the other prerequisites are taken care of, we can bring the OCFS2 cluster up in dual primary mode, as the following config file allows for. Duplicate this on both peers. | Once the other prerequisites are taken care of, we can bring the OCFS2 cluster up in dual primary mode, as the following config file allows for. Duplicate this on both peers. | ||
| < | < | ||
| - | global | + | global |
| { | { | ||
| usage-count no; | usage-count no; | ||
| } | } | ||
| + | | ||
| common | common | ||
| { | { | ||
| Line 746: | Line 766: | ||
| degr-wfc-timeout 60; | degr-wfc-timeout 60; | ||
| become-primary-on both; | become-primary-on both; | ||
| - | } | + | } |
| - | + | ||
| - | net | + | |
| - | { | + | |
| - | allow-two-primaries; | + | |
| - | after-sb-0pri discard-zero-changes; | + | |
| - | after-sb-1pri discard-secondary; | + | |
| - | after-sb-2pri disconnect; | + | |
| - | } | + | |
| disk | disk | ||
| - | { | + | { |
| on-io-error detach; | on-io-error detach; | ||
| - | } | + | } |
| + | |||
| + | net | ||
| + | { | ||
| + | allow-two-primaries yes; | ||
| + | } | ||
| syncer | syncer | ||
| - | { | + | { |
| - | rate 40M; | + | rate 80M; |
| - | } | + | } |
| protocol C; | protocol C; | ||
| } | } | ||
| + | </ | ||
| - | resource xen_data | + | This **/etc/drbd.d/global_common.conf** file needs to be identical and present on BOTH DRBD peers. |
| - | { | + | |
| - | on sokraits | + | |
| - | { | + | |
| - | device | + | |
| - | disk | + | |
| - | address | + | |
| - | meta-disk | + | |
| - | } | + | |
| - | | + | Recognizing the changes does not require a reboot! The following command (run on both DRBD peers), will update the config: |
| - | { | + | |
| - | device | + | |
| - | disk /dev/md0; | + | |
| - | address | + | |
| - | meta-disk | + | |
| - | } | + | |
| - | } | + | |
| - | </ | + | |
| - | + | ||
| - | This **/ | + | |
| + | <cli> | ||
| + | machine:~# drbdadm adjust xen_data | ||
| + | </ | ||
| ====Bringing OCFS2 online==== | ====Bringing OCFS2 online==== | ||
| Assuming **/ | Assuming **/ | ||
| Line 848: | Line 851: | ||
| </ | </ | ||
| + | ====Turn swapiness way down==== | ||
| + | The Linux default for swapiness is 60, which will result in the system paging stuff out to swap. Cranking it down to 10 seems a more prudent setting, especially on systems with SSDs, where we want them used as little as possible. | ||
| + | |||
| + | I added the following line to **/ | ||
| + | |||
| + | < | ||
| + | vm.swappiness = 10 | ||
| + | </ | ||
| + | |||
| + | ====/tmp in RAM==== | ||
| + | Seems Debian has a nice built-in support for mounting /tmp in tmpfs (RAM-backed filesystem). All you need to do is edit **/ | ||
| + | |||
| + | < | ||
| + | RAMTMP=yes | ||
| + | </ | ||
| + | |||
| + | And reboot! | ||
| ====integrating the array' | ====integrating the array' | ||
| The disk array is going to hold both Xen virtual machine images (and supporting files), but also serve as another backup destination for resources in the LAIR. | The disk array is going to hold both Xen virtual machine images (and supporting files), but also serve as another backup destination for resources in the LAIR. | ||
| Line 857: | Line 877: | ||
| * /backup - location of backup data (symlink to / | * /backup - location of backup data (symlink to / | ||
| - | ====/ | ||
| - | Until a proper place can be found, the following manual configurations are performed in **/ | ||
| - | |||
| - | < | ||
| - | ethtool -K peth0 tx off | ||
| - | </ | ||
| - | I didn't want to apply it to / | + | ====Historical: |
| + | **This | ||
| - | ====Configuring xen-tools to create Debian jessie VMs==== | ||
| There are two changes needed to successfully create jessie VMs, and both are symlinks: | There are two changes needed to successfully create jessie VMs, and both are symlinks: | ||
| Line 1094: | Line 1108: | ||
| Wait at least 5 minutes for data reporting to make it into graphable form. | Wait at least 5 minutes for data reporting to make it into graphable form. | ||
| + | |||
| + | =====Sync' | ||
| + | Since we've been successful running the systems out of a RAMdisk, care must be taken to preserve any changes in the event of a reboot or power failure. | ||
| + | |||
| + | ====rsync to disk==== | ||
| + | |||
| + | In this light, I first had the systems rsync' | ||
| + | |||
| + | < | ||
| + | 12 */8 * | ||
| + | </ | ||
| + | |||
| + | ====rsync to fileserver==== | ||
| + | |||
| + | This worked handily until sokraits lost its boot drive (again! In 2 months time!) so I decided to investigate netbooting using an NFSroot. | ||
| + | |||
| + | In the process, I may have finally made a breakthrough in my longtime desire to put the entire system IN the initial ramdisk (so it would be running in RAM from the get-go). Turns out, according to the manual page, you merely have to put the system IN the initrd file... obviously one needs adequate memory (2x at boot- enough for the initrd, and enough to decompress it). | ||
| + | |||
| + | My cron job changed as follows: | ||
| + | |||
| + | < | ||
| + | 24 */8 * | ||
| + | </ | ||
| + | |||
| + | I plan to rig up either some daily autogeneration of the initrd, or have a script on standby that can use to make it. This will then become the method of booting both sokraits and halfadder (and potentially freeing up a still-working SSD in the process! Which I can use in data2). | ||
| + | |||
| + | On the fileserver, I then obtain the latest copy of the hypervisor, kernel, and generate a new all-system initrd: | ||
| + | |||
| + | <cli> | ||
| + | data1:/ | ||
| + | data1:/ | ||
| + | data1:/ | ||
| + | data1:/ | ||
| + | data1:/ | ||
| + | </ | ||
| + | |||
| + | ====pxeboot file for sokraits/ | ||
| + | On the fileserver, in **/ | ||
| + | |||
| + | The file(s) contain: | ||
| + | |||
| + | < | ||
| + | default netboot | ||
| + | prompt 1 | ||
| + | timeout 2 | ||
| + | |||
| + | label netboot | ||
| + | kernel mboot.c32 | ||
| + | append netboot/ | ||
| + | |||
| + | label memtest | ||
| + | kernel distros/ | ||
| + | </ | ||
| =====References===== | =====References===== | ||
| Line 1128: | Line 1195: | ||
| ====DRBD+OCFS2==== | ====DRBD+OCFS2==== | ||
| * http:// | * http:// | ||
| + | |||
| + | ====Debian from RAM==== | ||
| + | * http:// | ||
| + | * debirf: | ||
| + | * http:// | ||
| + | * http:// | ||
| + | |||
| + | ====/tmp as noexec==== | ||
| + | * http:// | ||
| + | |||
| + | ====netboot system to nfsroot==== | ||
| + | * http:// | ||
| + | * this led me to the initrd man page which indicated we might be able to stick the entire system in the initrd and PXE boot that. So many things become simpler at that point. | ||
haas/system/sokraits_halfadder.offbyone.lan.1407426562.txt.gz · Last modified: 2014/08/07 15:49 by wedge
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
