STATUS updates
Some links of interest:
On Tuesday, mdadm started reporting a failure in one of the drives in data1 (sdd)… I ordered a replacement, and finally got around to replacing it on Friday.
The hardware side of the equation went as expected and without incident. I then proceeded to spend far too much time convincing a broken grub to reinstall (ended up netbooting finnix, chrooting in).
I re-added the drive to the md volume (after rebuilding the initrd and rebooting):
data1:~# mdadm --add /dev/md0 /dev/sdd
and left it rebuilding the RAID.
With the md raid volume rebuilt, I set about getting the two peers once again talking… as has been the case in the past, this has proven non-trivial… the apparent working recipe:
data1:~# drbdadm disconnect lair_data data1:~# drbdadm connect --discard-my-data lair_data
I had also issued an outdate to data1's data (which didn't seem to make a difference, or maybe it did, and “discard-my-data” doesn't imply ALL data, but merely some sort of small difference in unsynced data)… at any rate… they are once again talking.