My Client-On-The-Side... From-Hell

[NETWizz]

Okay, so I am called out to:

1. Troubleshoot a bad network port that has no connectivity.

2. Replace a Customer's Network Switch

3. Set-Up their Remote access Controller

I quoted:

1. $100 to troubleshoot and fix the cable (figured it just needed to be patched... MAYBE a bad keystone jack... Nope)

2. $400 to configure and replace a switch... cables provided (Figured I could talk them into Cisco, ProCurve, or Juniper... Nope)

3. $200 to configure the Dell Remote Access Controller...Um... this went bad.


************************************************** ******

1. The network cable:

• Got there and they had it patched but not working
• Of couurse, it wasn't labeled, so I had to use a Tone and Probe to find it... Damn.
• Put the cheap tester on it and found it had a reversal/miswire AND an open pair...
• Replaced the Keystone Jack even though it looked like it was done right... NOT FIXED.
• Looked at the patch panel and it was wired right!
• NOT FIXED... Removed from patch panel and connected an RJ-45 jack just for testing purposes! NOT FIXED
• I KNOW I did NOT wire either end wrong. WTF.. Damn... Damn...!!!?
• Threw my new toy from Fluke networks (the Cable IQ thing I bought for $1000)... It said the problem is 25 feet away. WTF!

This is where it was telling me the error was... I was thinking "maybe a rat got it???"




Okay, so I measured carefully and opened it up! Look at this. Again, WTF is wrong with other techs???



Someone made his or her OWN cable joiner. Imagine all the DB loss in there! Not only that, they made up their own wiring diagram because following the EIA/TIA standards is impossible if you are incompetent, unqualified, and/or scamming someone!

Absolutely just !P*(#$ amazing!


I tied the new cable to old and pulled it. Thank god it fed the 120'. I totally re-terminated it, and of course... it worked perfect! <== NOTE, I did NOT quote the extra charges nor revise my quote because I promised to fix the problem for $100 though I was planning on simply patching it to be honest. Never the less, this port took 1.5 hours for $100

*****************

2. Changing the Switch! Uh... Dammit. 35' cables and 25' cables wrapped up in a ball!

This is the patch panel from hell:



Oh, and they used those booted cables that are a real PITA to remove!


This is the cable that was removed... I just left it in a pile for them.

[NETWizz]

Okay... This is AFTER. I had to do some cable management because they don't have proper runners for tie-down to both sides of the MDF

• I moved their little organizer up to take out some slack somewhat neatly
• I also provided NEW cable without the stupid boots!

Unfortunately, the best thing I had on me are some wire ties...




Pretty Ugly... but it is a LOT easier to work on, somewhat cleaner, and in their budget.



**********************************

Setting up the Remote Access Controller:

When I tried to open the webpage after configuring it with the RACADM command, I got this: "login failed unable to connect to appliance due to network problem."

UM... NO. There is NO network problem... NO blocked ports... or anything on the LAN. WTF, Dell?

All I found on it was this, which suggests it is a firewall problem...:
I checked and found this: "http://en.community.dell.com/support-forums/servers/f/177/t/19164142.aspx"

I KNOW IT IS NOT THOUGH!

Okay, so Let's look at the Remote Access Controller:

Check DRAC settings:
racadm getsysinfo

Cold Reset the DRAC without having to unplug the server:
racadm racresetcfg

Then reconfigure it and verify:
racadm setniccfg -s 10.x.x.x 255.255.248.0 10.x.x.x
racadm config -g cfgLanNetworking -o cfgDNSServer1 10.x.x.x
racadm config -g cfgLanNetworking -o cfgDNSServer2 10.x.x.x
racadm getsysinfo

Same Problem! Doh!


It did NOT even reset the Remote Access Controller... it just did this:

C:\Windows\system32>racadm racresetcfg
The RAC configuration has initiated restoration to factory defaults.
Please wait up to a minute for this process to complete before accessing
the RAC again.


No problem, I will just update the DRAC Firmware, which says it resets it... Besides 1.60 has to be better than 1.45 anyway.

It prompts me to format drive H: and I am like, WTF??? Sure, why not? No data that I or anyone else needs is on that one... Okay... must be something the last guy forgot to do...

Fast forward 16 minutes of waiting for it to complete the World's Longest Firmware upgrade:


^^^^DAMN... @(^*%&* Another ********** day in the neighborhood!!!!

Well, maybe we will try to update it through the web interface then... No problem!




^^^^Well gosh.. Blasted... &*(@#*#. Apparently not even the logon box works now!


Let's take a look with the RACADM tool:



^^^^Gosh... <UNKNOWN> Firmware, build, version, and IP.... < sarcasm >Just Wonderful</sarcasm>

[Cadishead Computers]

Holy Cow!. What a mess that switch is in..

Not to imagine your thoughts on when the tester showed 25ft away, and opened the trunking up to see that horror. He must of thought out of sight, out of mind.. I mean how stupid can you be?. Don't answer that

Major kudos to you, and the cable IQ tester. That thing is amazing!

Would love to see the after pics too of the switch, and your cabling

How long had they had no connection on that machine?

Edit, sorry took longer to post.. Seen the after pics.

[NETWizz]

OKay... Well... No fear... Google is here!

Hum... Let's see:
Per this, http://en.community.dell.com/support.../19863815.aspx, I am not supposed to format the DRACs flash if asked... It is a known issue.


^^^^Why can't Dell maybe put this in the instructions if it is a "known issue"??? Should I have to read the forums before flashing firmware that is "Recommended"???

Guess I will un-format the Remote Access Controller... then try to re-flash it.


Okay... Maybe not on that re-flash part:



Okey... Dokey... Will try to do it from the command line with RACADM to force it... It can SEE the flash, so it SHOULD be able to write to it...

Downloaded this:


Extracted it... of coruse


Referred to Dell Documentation to RTFM... Roger that... Easy Easy:

Using RACADM Command line interface to Update Firmware: 
-------------------------------------------------------
1. Download the self extracting zip file to the server 
   with RAC installed. 
2. Extract the update files to the c:\temp directory. 
3. Open a command shell window on the server with the RAC
   hardware and update file.
4. In the command shell, enter the following racadm command.

   racadm fwupdate -p -u -d c:\temp

   -p: upload the update file from path specified using -d opt.
   -u: update RAC after loading new firmware.
   -d: path of firmware update file location. This path should
       not have the filename.

5. After the update completes, the RAC will reset.

Doh:

C:\Windows\system32>racadm fwupdate -p -u -d c:\rac
ERROR: A firmware update operation is already in progress.


Hum Found some Great Advice Here, but it's too late:
http://en.community.dell.com/support.../19254673.aspx

Seems "racadm vmdisconnect" is a good command to run though I didn't have any Virtual Media connected anyway!


One last thing to check...

[NETWizz]

The good news is I called my normal Dell Rep (we have a good repore) and he took care of acknowledging it is a "Known Problem" and even though I killed the flash firmware Dell is taking responsibility... I can do the labor... Perfect!

After taking out all those excess parts that don't do anything except prevent a metldown:


Lots of fans, shrouds, and that pesky old Dell RAC card:



New Card inserted:


New DRAC Card connected


All Fixed... Good As New:



I Know I have balls of Steal, too: rolleyes: because I updated the firmware on the new DRAC card!

............. No problems with this one.


Of course, I configured it, tested that the website worked, verified its IP addresses, setup email alerts, set it to talk to Active Directory for authentication... etc... <== No problems at all. It even took the firmware update without any issue!




It took a total of 2 visits and 4 hours to complete what I expected to complete in less than 2 hours in 1 visit.


What I learned:

• Don't QUOTE without seeing it!
• Assume each job will take longer.
• Plan time for when things go WRONG!
• Assume past techs did NOT do their job properly.
• Always check warranties before flashing things & don't assume a firmware flash will be successful just because the past 1000 you did were trouble-free!

[Cadishead Computers]

Again major kudos for getting all this sorted.

To be 100% honest, I would of been completely stuck at the rac controller part. The majority of your post went straight over my head.

Looks like I have some heavy reading to do. I don't do much server work, or networking because of this type of issue. I would much rather have the res side of things when issues like this crop up.

[YeOldeStonecat]

Originally Posted by NETWizz

Okay, so I measured carefully and opened it up! Look at this. Again, WTF is wrong with other techs???

Believe it or not....I'm come across a network, where I was going to lay all new cable throughout..and in several locations in the raceway...they combined shorter cable runs by HAND SPLICING THEM! YES.."HAND SPLICED...electrical tape on the twists which were wrapped by hand on two short cables to make one longer run!" Hidden in the raceway.

[NETWizz]

Originally Posted by YeOldeStonecat

Believe it or not....I'm come across a network, where I was going to lay all new cable throughout..and in several locations in the raceway...they combined shorter cable runs by HAND SPLICING THEM! YES.."HAND SPLICED...electrical tape on the twists which were wrapped by hand on two short cables to make one longer run!" Hidden in the raceway.

Wow! That's even worse than this. I wish people would just pull a new cable instead of try to splice things together...

[NETWizz]

Originally Posted by Cadishead Computers

Again major kudos for getting all this sorted.

To be 100% honest, I would of been completely stuck at the rac controller part. The majority of your post went straight over my head.

Looks like I have some heavy reading to do. I don't do much server work, or networking because of this type of issue. I would much rather have the res side of things when issues like this crop up.

Nah, you would do just fine. There are instructions on the Internet for how to configure the Remote Access controller. Generally speaking, you press CTRL+E when it tells you during the boot, but if you are already in Windows and don't want to reboot, you use the Command Line OR Dell Open Manage Server Administrator (a bit quirky). Overall, you just put in an IP address, password, and perhaps an SMTP server. It's about the same difficulty as setting up a network printer.

Even if you had trouble, you can always call Dell tech support. For servers, you get someone in the United States, Canada, the United Kingdom, or Australia. You WILL get a REAL tech who earns REAL wages in a REAL first-world country that speaks actual English! They treat you a LOT better when you buy Storage, SANs, Blades, Servers, Switches than Desktops, Laptops, and Printers.

^^^ Trust me. Support for equipment costing $7000 beats support for something that costs $700

[YeOldeStonecat]

Originally Posted by NETWizz

^^^ Trust me. Support for equipment costing $7000 beats support for something that costs $700

Yup..support for true business hardware is good. Including business PCs...say, Dell Optiplex. Good quick US based support, overnight shipment of warranty parts (or 4 hour onsite delivery if you have a higher warranty package)

Versus support for home grade stuff....spent tons of time in phone hold queues only to deal with overseas brutal language barrier support from someone reading from a script.