Monday, September 2, 2019

Deep Thought Disaster

Boom....

So, I was cheap, I left the house to get too warm during the day.  I ended up frying one of my main drives on my media server.  And then the main drive on my internet server.  Fuck.

Everything is dead.  Now to recover.

Evaluating the damage

The plex server's main drive or rather the lvm volume that had most of the storage is failing rapidly.  It won't even boot anymore now.

Dent's main drive is in a similar state so.

No:
  • Internet
  • Dhcp
  • smtp
  • mail
  • mailing lists
  • web server
  • usenet
  • media
The main problem of course is the internet.  Hell my house is automated, nothing works without the internet.

I have no memory of where my freeipa server was and I couldn't browse to it or anything.  I'm screwed.

What I do have

I've got an 8tb drive.   I also ended up buying a synology diskstation SAN, better one than the one I had.  This one has BTRFS.  I don't want this happening again.  'ya know?

Freeipa doesn't work.  I couldn't figure out what was up with it.  However I did have a snapshot with dhcp and dns working great on the services vm on the esx server.  So I got that working.

I let a pvmove run for a few days trying to move the logical volumes elsewhere.  This is not working at all.

Currently, I have what seems to be a totally dead dent and the main server drive is down in bucket.  I have a router that is working fine to connect us to the internet.  I have dhcp/dns running on a VM in the esx server.  So I've got internet.    I have a new VM with centos 7 1810 and freeipa on it.  That is where I am.

Detailed steps as I take them.

My current plan was to delete srv and debianroot which were on the dead drive.  It seems to hang when I do this.

What I'm going to do instead is:
  1. Figure out exactly what drive is dead on bucket
  2. Remove that drive and the 8tb drive from lvm
  3. turn off bucket
  4. ditto physically on the machine
  5. put the 8tb drive into the new server
  6. remove the old server configuration from the esx server
  7. shut down old h2gt2g
That will leave it in a state where everything is gone and I'm ready to get going.

I started playing with ipa.  I added ipa.deepthot.aa to the /etc/hosts file of my laptop.  Then I went onto the ipa server and started looking around.  Looks good.  I am going to muck with it a bit, just 'cause I'm here
I'm setting the mail domain to deepthot.org and the shell to bash

OK... now... to bucket
using 'lshw -C disk' I can see that the disk is a Seagate 4TB model ST4000DM000-1F21

I'm going to remove that one and the 8TB from the bucket lvm array
Tried to remove them several different ways.  Finally, just pulled both drives and used the 'vgreduce --removemissing --force'  That destroyed my partitions, however the server was backed up on my USB drives fairy recently.

Bucket boots fine.  Both remaining drives are healthy.  It still sounds really loud.

Monday 9/2/19 Day 2

Dent seems totally dead.  I've powered down Bucket to make it quiet.  I'm going to have to bring it back up eventually.  I don't know if the Torrent VM is there still.  I'll have to track down the VPN at least.

First Project New h2gt2g

  1. Remove h2gt2g from heartogold
  2. Power down h2gt2g 
  3. Unpack new h2gt2g
  4. Install drives
  5. Switch out SANs
  6. Play around a bit with the new SAN
When mucking about with heartogold I found a corrupted VM on it.  Not sure how it got in that state, however I ended up turning on ssh, sshing into heartogold and editing a configuration file as outlined in: https://communities.vmware.com/thread/582673

You should be able to permanently remove any inventory reference to this VM by removing the five lines that refer to this entry in /etc/inventory/hostd/vmInventory.xml e.g.:
  <ConfigEntry id="0002">
    <objID>3</objID>
    <secDomain/>
    <vmxCfgPath>/vmfs/volumes/590cca43-c78bd3ac-1234-000c2908a466/VMware vCenter Server Appliance/VMware vCenter Server Appliance.vmx</vmxCfgPath>
  </ConfigEntry>

Copy a back-up before doing this and use vi to edit the file.
Stop vpxa and hostd before edit:
#/etc/init.d/vpxa stop
#/etc/init.d/hostd stop
restart these after edit:
#/etc/init.d/vpxa start
#/etc/init.d/hostd start


Bob
The only thing configured properly on the esx server was the nfs share I set up yesterday to serve up iso files.  I went and deleted it from both vms and then deleted the nfs partition itself from here:


So, new SAN, regular RAID which sucks.  Yeah it's btrfs, but it's not using it right.  So boooo...  Sent it back, I'm going to buy two more drives for the old raid instead.  Put the 8TB into the old raid, it's now scanning and stuff.  Time for bed.

No comments:

Post a Comment