Boom....
So, I was cheap, I left the house to get too warm during the day. I ended up frying one of my main drives on my media server. And then the main drive on my internet server. Fuck.
Everything is dead. Now to recover.
Evaluating the damage
The plex server's main drive or rather the lvm volume that had most of the storage is failing rapidly. It won't even boot anymore now.
Dent's main drive is in a similar state so.
No:
- Internet
- Dhcp
- smtp
- mailing lists
- web server
- usenet
- media
I have no memory of where my freeipa server was and I couldn't browse to it or anything. I'm screwed.
What I do have
I've got an 8tb drive. I also ended up buying a synology diskstation SAN, better one than the one I had. This one has BTRFS. I don't want this happening again. 'ya know?
Freeipa doesn't work. I couldn't figure out what was up with it. However I did have a snapshot with dhcp and dns working great on the services vm on the esx server. So I got that working.
I let a pvmove run for a few days trying to move the logical volumes elsewhere. This is not working at all.
Currently, I have what seems to be a totally dead dent and the main server drive is down in bucket. I have a router that is working fine to connect us to the internet. I have dhcp/dns running on a VM in the esx server. So I've got internet. I have a new VM with centos 7 1810 and freeipa on it. That is where I am.
Detailed steps as I take them.
My current plan was to delete srv and debianroot which were on the dead drive. It seems to hang when I do this.
What I'm going to do instead is:
- Figure out exactly what drive is dead on bucket
- Remove that drive and the 8tb drive from lvm
- turn off bucket
- ditto physically on the machine
- put the 8tb drive into the new server
- remove the old server configuration from the esx server
- shut down old h2gt2g
I started playing with ipa. I added ipa.deepthot.aa to the /etc/hosts file of my laptop. Then I went onto the ipa server and started looking around. Looks good. I am going to muck with it a bit, just 'cause I'm here
I'm setting the mail domain to deepthot.org and the shell to bash
OK... now... to bucket
using 'lshw -C disk' I can see that the disk is a Seagate 4TB model ST4000DM000-1F21
I'm going to remove that one and the 8TB from the bucket lvm array
Tried to remove them several different ways. Finally, just pulled both drives and used the 'vgreduce --removemissing --force' That destroyed my partitions, however the server was backed up on my USB drives fairy recently.
Bucket boots fine. Both remaining drives are healthy. It still sounds really loud.
Monday 9/2/19 Day 2
Dent seems totally dead. I've powered down Bucket to make it quiet. I'm going to have to bring it back up eventually. I don't know if the Torrent VM is there still. I'll have to track down the VPN at least.
First Project New h2gt2g
- Remove h2gt2g from heartogold
- Power down h2gt2g
- Unpack new h2gt2g
- Install drives
- Switch out SANs
- Play around a bit with the new SAN
The only thing configured properly on the esx server was the nfs share I set up yesterday to serve up iso files. I went and deleted it from both vms and then deleted the nfs partition itself from here:
So, new SAN, regular RAID which sucks. Yeah it's btrfs, but it's not using it right. So boooo... Sent it back, I'm going to buy two more drives for the old raid instead. Put the 8TB into the old raid, it's now scanning and stuff. Time for bed.


No comments:
Post a Comment