Intermittent Network Packet Loss - Tools?

[sethay]

I have run into this a few times since we opened up a few years ago. Small business networks that have intermittent network packet loss from computer to computer, sometimes also comes with internet packet loss.

Generally this has been caused by bad hardware. Two examples that come to mind: one was a bad NIC and one was a bad KVM switch that included Ethernet.

In one of these cases they would completely lose network connectivity on all systems. Power cycling the switch fixed it for 24-72hrs. Replaced the switch, but it didn't fix the problem. Ended up being a faulty KVM switch that had a NIC port being used.

The second case all systems on the network would randomly drop about 12-18 packets a day. The dropped packets would happen at different times during the day. For example system A would drop some packets in the AM, system B would drop some in the PM. It might be the exact opposite the next day. No system was worse or better than any other system. It would happen to all of them. Ended up being a faulty NIC in one of the systems.

Basically, the only way I knew to work on these problems was to use a constant ping -t and gauge the results. Segment the network until we eventually found the culprit causing the issue.

This is very time consuming and not efficient.

Does anyone have any suggestions on how to better handle these situations?

I appreciate you taking the time to help.

 

[Martyn]

I've seen problems like this a few times over the years often to do with speed auto negotiation issues or poorly made Ethernet cables. Try setting the speeds down to 10 mb fixed on the problem ports to see if it stabilizes. Also use ping xxx.xxx.xxx.xxx -l 1500 -t to force the problems. Poor cables will often work at 10 mbit but at faster speeds will intermittently fail due to the more critical inter frame gap.

 

[i2omani]

wireshark and set some filters

 

[sethay]

Wireshark. Thank you for the tip, I will look into it.

From the basic overview on their website...I would use this to find the bad packets from the faulty device?

 

[AdamsAPlus]

I agree with wireshark. We used this tool in college. It will take a little bit of instructions to be able to get the filters correct. If you do not set the filters then you will get way too much info. It's a great tool once you learn how to use it.

Do a google search for wireshark instructions. There are instructions all over the internet. (Or at least there was 5 years ago when I was learning it in college)

 

[Martyn]

Wireshark? More useful for tracking packet/frame types. I've used it quite a bit but to track packets that are there not that aren't there, i'm not so sure? If you're getting packet loss then the problem will be in the physical layer, layer 1 imho or maybe layer 2/3. So cables. nics, switch ports. On the suspect cables/ports use the ping as I said above should help.

 

[sethay]

Thank you for taking the time to respond Martyn.

"Try setting the speeds down to 10 mb fixed on the problem ports to see if it stabilizes."

In these situations the primary goal was actually locating the problem port. All devices were having problems.

"Try setting the speeds down to 10 mb fixed on the problem ports to see if it stabilizes. Also use ping xxx.xxx.xxx.xxx -l 1500 -t to force the problems."

"On the suspect cables/ports use the ping as I said above should help."

In these two cases we did use ping IP -t to monitor all of the devices, but there were no patters telling us which device might be causing the problems. In the one case every device dropped at the same time, in the other case all devices lost a few packets throughout the day.

Would changing the time to live setting help something? We just did default.

 

[Martyn]

Ah sorry I thought you knew which port it was on. What are the symptoms again? Is there a problem or have you just noticed there is an occasional dropped packet?

 

[phaZed]

I would just add that because you are seeing a network wide outage/issue that you should be looking a Switches, routers, and gateways. Any modern switches, routers, gways are protected against the "lone gunman" scenario. In other words, a single bad NIC should not cause a problem because the network ports should be isolated from each other (usually optically isolated). Only the offending system should fall off the network.

Just curious to know what type of switch/router are they using? Perhaps this could be a DHCP server conflict?

 

[sethay]

Thanks again for the replies guys. I do appreciate people taking the time to reply.

However, I must not be very good at explaining myself. I have outlined two situations that occurred in the past that have already been resolved. One was a bad NIC one was a bad KVM switch with an Ethernet port. They both displayed odd symptoms.

The only way I could come up with to locate the offending network device was to segment the network and do process of elimination. It seems like there has to be an easier way.

One thing that was brought to my attention by another tech was a smart switch that monitors every port. It seems like I could then look at activity on each port and may be able to locate the offending network device that way.

I am looking for ideas on how I could more easily troubleshoot similar situations when I run into them in the future.

Thanks!

 

[WeyantEnterprises]

i have also run into this problem. In my case it was a faulty netgear wireless router, reconfigured to act as a guest wireless access point. Unfortunately, I had to do the same network segmentation until the problematic device was identified. I ran wireshark and it did not help in identifying the issue.