Go Back   Technibble Forums > Service Solutions > Servers

  Technibble Sponsor

Reply
 
Thread Tools Display Modes
  #1  
Old 09-20-2011, 09:01 AM
NETWizz NETWizz is offline
 
Join Date: Jan 2008
Posts: 1,421
NETWizz has a spectacular aura aboutNETWizz has a spectacular aura about
Default Crazy Consulting Work

Okay, so I put in a bid to troubleshoot an Active Directory problem for a small company (600 computers)... and got chosen.

Chief Issue: Can Join XP and Server 2003 Computers to Domain, but cannot join Vista, 7, Server 2008, Server 2008 R2 computers to domain.

Error: RPC Server unavailable.

Bid Amount: $8,000

******************************

Okay, so I get there and first look at an XP machine and a Windows 7 machine to try to find the differences.


1. I run IPCONFIG /ALL on both systems... Everything is configured the same from the same DHCP server (obviously the IP addresses are different)... No problems here.

2. Next, I take a look at the Operation's Masters (FSMO roles) with "netdom query fsmo" All the FSMO roles are running, and I can ping the servers.

3. Next, I took a look in Sites and Services and compared the AD site info with the sub-nets and Domain Controllers... no problem...

4. Took a quick peak at replication with REPLMON to ensure all the Domain Controllers are properly syncing their Global Catalogs. Overall, I confirmed Active Directory is not broken.


5. Next I thought the XP and Vista+ machines might be talking to a different Domain Controller or getting a different answer from DNS or that something third-party is installed... something strange..., so I decided to poke into DNS.

6. I pinged things like servers, and domain controllers from both XP and 7... Got the same responses... Great.

7. To find an RPC server & join a domain, it is going to need to query the SRV records from DNS, so I did an nslookup on the SRV records for things like LDAP and kerberos.

... Basically queries like this nslookup -type=SRV _ldap._tcp.dc._msdcs.addomain.com


Found the problem is DNS:

Windows XP would display the SRV records for the Domain Controllers of the Active Directory Domain.

Windows 7 would NOT display any SRV records..

i.e. I got something like:
*** dnsserver.addomain.com can't find _ldap._tcp.dc._msdcs.addomain.com: Non-existent do
main



I told them to take me to the DNS servers

8. First thing I noticed was a failed hard drive in the Array that they didn't notice. Okay, that is NOT the cause of this problem, so I let them know.

9. I took a look at the DNS console on that Domain Controller & DNS server... Everything was fine!

10. I said, "You have two DNS servers, let's just try to reboot this one."

11. I reboot it and it must have taken 20+ minutes to boot. I looked at their IT guy in disgust and said, "Does it always hang taking forever to start networking and apply computer policies?" He said, "Yeah"

12. Before logon, I am greeted with "Windows has detected an IP Address Conflict." This is a DNS server and a Domain Controller with an IP Conflict.

13. Their tech said, "It always does that; we just click OK."

14. I asked, "What else has the same IP address?" He didn't know!

15. Shutdown the DNS server, so I can track it down...

16. Logon to their Cisco Catalyst switch and then do the following:

switch>en

There was no password!

switch# ping 10.x.x.x. (the IP of their DNS Server)
switch# show arp

It basically listed a long list that scrolled...

So, I did...

switch# show arp | include 10.x.x.x (the IP address of their DNS server that I just pinged)

I got a response like

10.x.x.x 1234.5678.9abc (A Cisco formatted Mac Address)

Okay, so I ran:

switch# show mac-address-table address 1234.5678.9abc

I can't remember the next command, but it was a show cdp neighbors (some argument for the interface & for detail)


It told me that the DNS server was on Interface Gigabit 0/1, which was fiber, lol... So, I followed that Fiber to an LIU... I asked their tech where the other end of it is.

17. He took me to another server-room... I logged onto the switch. ping, show arp, show mac-address-table... blah blah blah.. Interface FastEthernet 0/24

18. Followed that cable and it went to their Cisco firewall!!!

19. Logged onto the Cisco Firewall and did a show-run. That IP address was set as a management IP address.

20. The firewall was also running a caching DNS server!!! That was why those nslookups on Windows 7 would work for things like nslookup google.com.... nslookup server.addomain.com... but no SRV records (those aren't cached)...



21. I changed the IP address and removed the caching DNS server functionality from the Cisco Firewall... Then I booted the Domain Controller/DNS server, which booted in like 3 minutes!

Problem Fixed!



Q: So, why was XP getting the SRV records from the other DNS server and Vista/7 getting it from t he Firewall?

A: Windows XP seems to do a Round-Robin using any of its configured DNS servers though it WILL pick on on its local subnet before using a remote DNS server... This is the same behavior on Vista/7. I.e. If you have DNS 1 and DNS 2 setup on all your systems, you don't want them to ALL hit DNS 1 every time... Microsoft knows this! That said, you don't want to go to DNS 2 if it is across a slow WAN with high latency... Again, MS knows this.

So why the difference?

A: Vista/7 WILL query a caching DNS server, first. Hence, they were seeing records from the Firewall NOT another, properly configured Domain Controller... Hence, there were no SRV records & Vista/7 couldn't find the RPC server.


Crazy Hugh?


This took about 4 hours and I got paid $8,000 for which I need to file a 1099-Misc with the IRS.
Reply With Quote
  #2  
Old 09-20-2011, 09:19 AM
seedubya's Avatar
seedubya seedubya is online now
 
Join Date: Jan 2008
Location: Carlow, Ireland
Posts: 3,374
seedubya has a spectacular aura aboutseedubya has a spectacular aura about
Default

You deserve it. That was brilliant troubleshooting. I would have been struggling after step 6 or so.

Who's Hugh, btw?

Last edited by seedubya; 09-20-2011 at 01:47 PM.
Reply With Quote
  #3  
Old 09-20-2011, 09:39 AM
controlfreak controlfreak is offline
 
Join Date: Mar 2010
Posts: 66
controlfreak is an unknown quantity at this point
Default

Congrats good troubleshooting! Sounded like questions from a MCTIP exam. You were obviously worth the money as there techs clearly didn't seem qualified to run that network.
Reply With Quote
  #4  
Old 09-20-2011, 02:46 PM
NETWizz NETWizz is offline
 
Join Date: Jan 2008
Posts: 1,421
NETWizz has a spectacular aura aboutNETWizz has a spectacular aura about
Default

I have done the MCSE 2003 and recently the MCITP as well as become a Microsoft Certified Trainer.

My favorite thing still is teaching on the side at my local Community College.

I am A+ & Net+, but I don't teach any of the CompTIA stuff. That would be fun to get into.
Reply With Quote
  #5  
Old 09-20-2011, 05:47 PM
dbdawn dbdawn is offline
 
Join Date: Jan 2010
Posts: 925
dbdawn will become famous soon enough
Default

NO wonder your nick is NETWizz

But ya, you would think the in-house tech wouldn't be ignoring the errors on that DNS server on first boot.

Last edited by dbdawn; 09-20-2011 at 05:49 PM.
Reply With Quote
  #6  
Old 09-20-2011, 06:13 PM
Elemdee's Avatar
Elemdee Elemdee is offline
 
Join Date: Mar 2010
Posts: 773
Elemdee is on a distinguished road
Default

Quote:
Originally Posted by NETWizz View Post
13. Their tech said, "It always does that; we just click OK."
Reply With Quote
  #7  
Old 09-20-2011, 09:29 PM
wimwauters's Avatar
wimwauters wimwauters is offline
 
Join Date: Aug 2009
Location: Rushden, Northants, UK
Posts: 745
wimwauters is on a distinguished road
Default

This just beggars belief!

600 computers and they don't haven't got a single IT professional employed!?

I bet their backup doesn't work either... have they ever even heard the concept of an IT sysadmin?
Reply With Quote
  #8  
Old 09-20-2011, 10:22 PM
NETWizz NETWizz is offline
 
Join Date: Jan 2008
Posts: 1,421
NETWizz has a spectacular aura aboutNETWizz has a spectacular aura about
Default

Quote:
Originally Posted by wimwauters View Post
This just beggars belief!

600 computers and they don't haven't got a single IT professional employed!?

I bet their backup doesn't work either... have they ever even heard the concept of an IT sysadmin?
I am not sure I know what you mean the concept of an IT sysadmin?
Reply With Quote
  #9  
Old 09-20-2011, 11:34 PM
codegreen codegreen is online now
 
Join Date: Mar 2011
Location: New Brunswick, Canada
Posts: 1,317
codegreen is a jewel in the roughcodegreen is a jewel in the roughcodegreen is a jewel in the rough
Default

Are you saying they have a full-time IT person, but they had to call in a contractor to troubleshoot their network? If so, why does their current admin still have a job?

In any case, nice work! You were certainly worth the money.
__________________
Stephen Gilbert

Code Green Technology:
Fredericton Computer Repair
Reply With Quote
  #10  
Old 09-21-2011, 04:07 AM
NETWizz NETWizz is offline
 
Join Date: Jan 2008
Posts: 1,421
NETWizz has a spectacular aura aboutNETWizz has a spectacular aura about
Default

Quote:
Originally Posted by codegreen View Post
Are you saying they have a full-time IT person, but they had to call in a contractor to troubleshoot their network? If so, why does their current admin still have a job?

In any case, nice work! You were certainly worth the money.
Yes, I am saying they have two (2) IT people though I think one just answers the phone and works helpdesk I don't know if they are full-time or not though. Yes, they had to call in a contractor to troubleshoot their Active Directory Infrastructure. It is actually surprisingly common.

What tends to happen (not necessarily in this place) is the person who gets the job often has no experience and gets moved from another department OR they get the job because the know some manager who hires his friends. Maybe that person is good with computers and a manager thinks that is all Enterprise Admins do not understanding the scope of the job.

I do at least 6 to 8 jobs a year like this one only most are for deploying things like Lynch Server or Migrating Exchange... this is in addition to my full-time job and adjuncting as an associate professor (part-time pee-on no benefits) i.e. teaching one (1) course via distance education each month.


I can't say authoritatively as I don't understand this company's IT culture, I don't know this IT guy's job title, I don't know his specific job requirements etc. Maybe he is hired for Helpdesk & paid $30k in which case he is in way over his head though not his fault.

Yes, he is still employed.

He seems to keep things running day-to-day in a re-active manner. I.e. If something says there is a problem he clicks "OK," but doesn't troubleshoot it. If someone needs a password reset, I am sure he can do that. If something breaks, they repair it. What they don't do is improve their infrastructure, upgrade their infrastructure, migrate to newer software... anything pro-active. If it is working, they don't touch it probably because they are scared/afraid of breaking something because they don't know they can fix it being they don't even understand how it works.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 03:07 AM.


Powered by vBulletin®
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Technibble.com is based out of MELBOURNE, AUSTRALIA.