Extremely stuck here - Server problem

[Big Jim]

I am currently running Windows hyper-v 2012
3 main VMs
1 - 2012 R2 - handles DHCP and AD
2 - 2008 R2 - handles file sharing
3 - 2012 R2 - handles a single backup but soon to be expanded

main drive ran out of space so all 3 VMs ran in to stopped-critical
I cleared some space and rebooted the machine
according to powershell command get-vm they are all backup and running, however nothing can see the DHCP server, not even the hyper-v server that it is running on.

1 NIC has a static IP the rest have been auto-assigned 169.xx.xx.xx addresses.

Cant connect to hyper-v server via hyper-v manager because the domain server isn't available.

both DRAC and OMSA tell me that all 6 NICs are "up"
the switch that they are connected to is also flashing away as I would expect it to.

What do I do ?

 

[Big Jim]

ok so stopping and restarting the VM seems to have resolved the issue.

I have a 278Gb drive for the VMs (5 created in total but only 3 running)
I have cleared all but the last snapshots from my VMs.
I only have 49GB free, this seems like a very inefficient use of HDD Space.

Even at 30GB each I shouldn't be using more than 150GB, seems to be a lot of very large files in my "Hyper-v" folder that I'm not sure the purpose of.


anyway Its now almost 1am I am going home to bed.

 

[HCHTech]

anyway Its now almost 1am I am going home to bed.

Phew! Glad you got it sorted, but you need some more space, dude. The extra files might be the result of Hyper-V's attempted self-repair.

 

[YeOldeStonecat]

For single servers...I keep the Hyper-V host in workgroup mode...seen enough cases where if there's an issue (like this) and the DC is not avail, it'll give issues logging in (despite supposed "cached credentials". I see no advantage to having the HV-Host join a domain if it's just a single bare metal host. I realize there are advantages with multiple bare metal hosts and multiple H-V environments..clustering, live migration, etc. But in those situations there's typically multiple spread about DCs too!

I always put a H-V hosts NICs on static IPs..so they're avaible no matter what..esp when a DHCP source (such as on a guest server) is not avail. Same with outside of bandwidth management NICs...static IP 'em ..manually...not via reservation.

If snapshots were enabled...always keep an eye on that. There was a bug in earlier versions where it would keep creeping up on you and forcing regular maintenance to "merge". ....very time consuming.

 

[HCHTech]

.I keep the Hyper-V host in workgroup mode...

I've never had a situation where I had more than one host server at a location, but I also never tried to join the host to the domain, I was afraid it would create some kind of loop which would destroy the space-time continuum - and we all know that would be bad....very bad.

Kind of like accidentally remoting into the machine you are using to remote with - nothing good comes from that!

 

[Big Jim]

So before I go ahead and remove my hyper-V host from the domain, can I just check
Will I still be able to access it remotely using hyper-V Manager from a domain connected workstation ?

 

[YeOldeStonecat]

I don't believe so...we just RDP or RMM into the host.

 

[Moltuae]

I've never had a situation where I had more than one host server at a location, but I also never tried to join the host to the domain, I was afraid it would create some kind of loop which would destroy the space-time continuum - and we all know that would be bad....very bad.

Kind of like accidentally remoting into the machine you are using to remote with - nothing good comes from that!

I have several customer's systems with domain-joined Hyper-V hosts; some with as many as 10 hosts, but also a few with just 1 host. I've never had any issues. As strange as it may seem, joining a host to a domain that is controlled by a guest DC is actually a common and recommended practice.

Ease of manageability is one reason for joining a single-host system to the domain but the main reason I make single-host systems domain-joined is for scalability and serviceability. I configure all host systems the same to enable additional host servers to be added easily later, whether there's going to be 1 host or 10 initially. It also simplifies the replacement/upgrade of a host server .... Just add the new host server to the domain, let the GPOs configure it, live-migrate the VMs, then shut down and remove the old server.

 

[Big Jim]

looks like I have messed things up again.

I was trying to tidy up all of the files (as the drive was previously showing as full)

so I started to use the "move" command in hyper-v manager to move the all of the VM machine files to one folder.
I have moved the files for all 3 of my live running VMs as well as the ones that were off and I am back to square one again.

initially my file server went down and wouldn't restart, so I rebooted the host and when it came back up the remaining 2 running VMs were paused,
at this point hyper-v server would only show the VMs for about 5 -10 seconds then they all disappeared again, this worked initially for about 5 minutes, after that hyper-v didn't even see the host
I used DRAC to power cycle the machine again.
I cannot even RDP to the host, although I know it is running because I can ping it.

 

[Moltuae]

at this point hyper-v server would only show the VMs for about 5 -10 seconds then they all disappeared again, this worked initially for about 5 minutes, after that hyper-v didn't even see the host
I used DRAC to power cycle the machine again.
I cannot even RDP to the host, although I know it is running because I can ping it.

Weirdness such as that makes me think you're dealing with a bigger underlying problem, such as system file corruption, possibly caused by a failing drive. It sounds like numerous services are failing to start. If that's happening repeatedly I'd probably start examining the file systems and drives.

When you say "hyper-v server would only show the VMs for about 5 -10 seconds then they all disappeared again", is this in Hyper-V manager? Are there any error logs? And what happens if you create a new VM using the same settings and VHD(s)? (be sure to backup the VHDs first). Also, what errors do you get if you attempt to start or un-pause the VMs?

 

[Big Jim]

i think attempting to move the servers whilst they were running was what caused the issue.
I got up earlier this morning to go down to the shop and get cracking with this before we opened.

the DC was back up and running pretty quickly (took me about 40 minutes), but the fileserver was toast, I gave up in the end for 3 reasons, 1 I already had a spare VM with an up to date 2012 R2 install on it ready to go, and 2 the FS was running 2008 R2, so this was an upgrade I was planning anyway, this little problem has just brought it forward a few months. 3 I am fixing this problem using pure google-fu.
The vhd file with all my data on it was in tact and the new 2012 VM adpoted it no problem.
I just had to configure the shares etc again.

my knowledge of servers is very limited, I don't know where to check a lot of the things you are asking me about to be honest.
I am not 100% sure where error logs for everything would be held, since I am running hyperv, individual VMs and a hyper-v manager.

in terms of file corruption, what would be the best way to go about checking that ?
OMSA reports all drives are up and running ok, how do you normally defrag a system runnig vms ?

systems do not seem to be running very fast though tbh, any of the server management mmcs that I have tried to open take a good few minutes to appear after clicking on them, DHCP/AD/GPO etc. is that normal ?

server is an R720
Dual hex core Xeons
48GB Ram
all vms running off a single RAID1 array (2x278GB 3.5" SAS Drives)

the storage VM is running off a RAID 10 array (4x 278GB 35" SaS drives)

file performance over the network seems fine

 

[YeOldeStonecat]

Defrag went away in the middle XP years. No need for it on modern systems, and with modern disks. And you don't want to run it on a hyper-v host. The NTFS file system is good at managing space utilization/organization.

How many guests? And what resources are you giving them for CPU cores and RAM? How are their drives spread across that R1/R1?

 

[Big Jim]

currently 3 guests all now running 2012 R2
4th guest installed but not running

Will have to check exact resource spread tomorrow but i think they are all the same, which is 12 cores and 8gb ram each.
definitely the DC has that and I know the now dead File server had that.

I have all VMs on the RAID 1 array, I don't know what you mean about the spread though ?



re defrag, really ? we see significant speed increases using a 3rd party defrag tool on some customers machines, that in itself is slowly disappearing now though as we try and migrate people over to SSDs

 

[Moltuae]

my knowledge of servers is very limited, I don't know where to check a lot of the things you are asking me about to be honest.
I am not 100% sure where error logs for everything would be held, since I am running hyperv, individual VMs and a hyper-v manager.

You can find the error logs in the same place you'd find them on a Windows desktop OS, ie Event Viewer (eventvwr.msc) => Applications and Services Logs => Microsoft => Windows, then drill-down to the appropriate logs. Not all logging is enabled by default, so you might need to enable (right-click) some of them beforehand.

Additionally (from Server 2012 onwards), you should see a nice graphical summary of any issues in Server Manager and a more detailed break-down of various events and other information such as 'Best Practices' and 'Performance' when you select the individual server roles. If all of the servers are 'added' to each other (and this is another reason I would always domain-join a Hyper-V host), you can see a summary of the entire system's health at a glance from any one of the servers (plus you can manage them all from a single (physical or virtual) server and perform actions such as restarting services, etc).

To check whether there are potential file system/drive issues, again for the most part you can use the same diagnostic procedures that you would on a Windows desktop OS. Check the logs for any file system or drive-related errors or tell-tale signs of general problems such as repeatedly failing/restating services. Additionally you could check the server's Light's-Out Management for any reported drive health issues, or you could remove a suspect drive and run checkdsk on it (after taking a backup of course).