Seeking thoughts on Comcast Internet issue

[pceinc]

Here are the details.
Client has Comcast SMC Gateway with an RV082 firewall configured with a static IP. Two 24 port switches, WAP, couple of IP printers, Windows 2008 server, Datto box, 2 desktops, and 2 or 3 wireless laptops that come and go.

Back in November I get a call that the client is losing internet daily at the same time between 3:15 and 3:30. It will come back on it's own after a few minutes. They called Comcast first which came out and said the problem is not on their end. Shocker. They waited a week before they called me. When I actually go onsite it was after the time change so now the outage is after 2pm. I'm thinking something is rebooting at the same time daily. I've ruled out any scheduled event that could be taking place. The RV082 has lengthy up time.

DAY1: I arrive onsite, get plugged in with laptop and start some pings and basically wait. Sure as the sky is blue 2:19pm internet goes out. At this point I'm connected to the RV082 directly. I have pings to their server, RV082, 208.67.222.222, and 8.8.8.8. Only the WAN IP's are dropping. I unplug my laptop from RV082 and plug directly into the SMC 4 port. Pings come back instantly. I switch back to RV082 and pings are still replying. It's an old RV082, they've been known to die, so I swap it out for a known working RV082 that I brought with me and tell them I will be back the next day.

Day2: Onsite, get plugged in. 2:19pm, WAN pings start dropping. I know it's not a firewall issue. This time however, the connection does not come back right away. So I pull the uplink to the primary switch and the connection comes back. I monitor it for some time and tell them I will be back again the next day. It's difficult to troubleshoot this when it's not dropping packets.

Day3: Onsite, get plugged in. This time I disconnect RV082 from SMC, connect laptop to SMC. 2:19pm comes along, no loss of connection. Weird. So this time before I leave I install a spare switch, update firmware on WAP, configure SMC to have LAN IP as RV082, and basically have them operating on all spare equipment excluding the SMC.

All along I will add that at no time did I ever have dropped packets on internal lan devices such as server or printers.

Day4: Onsite, get plugged into SMC, 2:19 no loss of connection. I reconnect all original equipment and basically put everything back to square 1. The only change that has been made thus far is updated firmware on an Asus WAP.

Day5: Monitoring remotely. 2:19 no loss of connection. Hmmm, what could it have been.

Fast forward to last Friday and I get a call the problem is back. This time they lose connection at 12:19pm. WTF? It's as if it jumped an hour per month. It started happening the beginning of last week so they say.
They called Comcast again, tech comes onsite, not their problem. Shocker. An employee was working on Saturday and states he did not lose internet at 12:19. Monday I go onsite. Get setup. This time at 12:19 I notice I do not actually lose connection but it slows down so slow you would think the connection is lost. I start some traceroutes to a few WAN IPs and monitor. Everything looks okay. Pings are all good as well. I monitor remotely on Tuesday due to snow storm. Connection slows at 12:19. I can actually maintain an RDP connection to their terminal server albeit slow as ever. It was difficulty to do anything. I could see pings and tracert responses but mouse and keyboard input was very spotty. After 20-30 minutes of this I rebooted the RV082 and the connection came back. I looked at the tracert logs and can see the first hop after the modem is where the packet loss occurs. It's the CMTS of Comcast. So now I will call Comcast and present my findings. What on earth could be happening at the same time every day for a few minutes that could cause this? I've Googled and cannot find anyone reporting similar issues. I'm going to request a new SMC from Comcast but I do not think that is the issue. I'm using VisualRoute so I have a lot of comparables to show Comcast.

What are your thoughts and has anyone experienced anything like this before?

 

[Knightsman]

I'd almost wonder if one of the client machines runs some weird update at that time, and it bogs the bandwidth. But with it being Comcast, i'm sure they have a decent pipe.

Very weird. I would suspect Comcast also. I have a client on an old downtown road. Their internet goes out constantly at different times for about 20 seconds. I know its Comcast, because they have 2 stores, 2 spots down from each other with different types of equipment, and they both go down at the same time. Comcast has always maintained it has to be my clients equipment.

 

[pceinc]

I'd almost wonder if one of the client machines runs some weird update at that time, and it bogs the bandwidth. But with it being Comcast, i'm sure they have a decent pipe.

They don't have the fastest Comcast account, 20mbps, but it would take a serious misappropriation of bandwith to cause this.

Very weird. I would suspect Comcast also. I have a client on an old downtown road. Their internet goes out constantly at different times for about 20 seconds. I know its Comcast, because they have 2 stores, 2 spots down from each other with different types of equipment, and they both go down at the same time. Comcast has always maintained it has to be my clients equipment.

Unfortunately I don't have any other Comcast customers on the same node to compare with. There is a small motorcycle shop business next door I may speak with. The chances of them saying "oh yeah, that problem drives me nuts" are slim but it won't hurt to ask.

 

[YeOldeStonecat]

We've done dozens and dozens and dozens of those RV082 and Comcast biz gateway (SMC) installs....great reliable setup. Those RV082's were tanks! Only time I'd seen them die was when lightening hit and made it through.

I just deployed a big Ubiquiti wireless network at a client that had one (RV0 with Comcast 50 meg pipe, SMC gateway)...I was about to replace their router until I saw they had an RV082 in there...kept it, it's fine.

First question, I am assuming that the SMC is configured to pass the first usable public IP address to the RV0? So that the RV0 gets that public IP, and its gateway is the WAN IP of the SMC.

Firmware updated the RV0? Although they've not released new firmware in a long time...it's as up to date as possible?

Else....since the common denominator is the time of day here, I'd look at something happening with a workstation or server. Offsite backup kicking off, or some trojan spamming out stuff.

 

[YeOldeStonecat]

Another thing I always did with older routers, manually set the MTU on them, I never trusted the auto detect. Since cable internet.....1500.

 

[Markverhyden]

How many usable IP's do they have? If they have more than one I would configure a laptop with wireshark, hang it off of the SMC as a sensor. Do the same on the LAN side. You can match times and traffic to see if there is anything interesting going on.

As far as Comcast themselves. They have very poor reporting on their end. As we all know you have to provide them with information showing there is a problem on their side before they really do much.

Not sure if you are aware but if you just turn on wireshark it will run with defaults but when the buffer fills up you won't be able to save the info. So for long term scanning you need to set it up with some options to write files to disk every so often.

 

[pceinc]

Just left the client's office. The outage today ended up being longer, in and out for 40 minutes. I removed the RV082 from the networkj despite already swapping it out with another unit a couple of months ago. They are now temporarily on the SMC gateway. To answer the above questions, yes the RV static IP is passed on from the SMC. Firmware is latest. I was not able to be onsite today during the outage but someone in the office took it upon themselves to start rebooting the modem and routers so my logs are compromised for the most part. I'm going back tomorrow with an Untangle box and leaving it there for a few days so I can monitor traffic from each device on the network.

I called Comcast and all their scripted techs are willing to do is dispatch a tech. I did request to have the modem replaced in case there is some flaky firmware issue.

 

[YeOldeStonecat]

What do you use for an RMM?
If you have one, stick a laptop with a probe agent behind the SMC, and if you don't have a probe agent behind the RV0...stick one on one of their nodes, and watch 'em for a few days. See if both disconnect at once, or just the RV0.

Try another port in the back of the SMC? Another patch cable?

 

[pceinc]

We use Continuum. I have a server agent installed plus a Datto box onsite. We receive no alerts that either go down or become unreachable. There is enough connection to respond to pings so an alert is never triggered plus I am able to maintain an RDP connection directly to the server during the slowness but it is painfully slow. I built an Untangle box today. Going to take it onsite tomorrow and put it behind the Comcast SMC.

 

[pceinc]

Update: Comcast switched the modem out this morning. I arrived onsite afterwards to install the Untangle box. No loss or slow connections issues were found during the time when it was previously happening. So time will tell I guess.

The Comcast tech actually came back during my visit and spoke with me about what he saw before swapping the modem. He said the modem was not reporting it's upstream data consistently. I'm not sure if he was referring to the upstream power level or something else. He did not elaborate. It would have been nice if they replaced the modem the previous two times it was requested.
The new modem is a Netgear.

 

[smashedbotatos]

Comcast modems send upstream connection data. Not sure if it is scheduled or on demand, or even both. It is like a diagnostic of sorts. The modem itself doesn't send the data though. The equipment at the local office handles it.

They bascially get info like MAC address, current IP, recieve power level, transmit power level, and signal to noise ratio. They normally use this iinformation to diagnose connection issues.

Under certain circumstances modems will have intermittent signal issues, and Comcast won't be able to get an upstream data report. This isn't monitored, unless you call and they specifically look. I can go unnoticed for quite a while if the internet connection doesn't have issues.

 

[FremontPC]

Sounds like another thread:

http://www.technibble.com/forums/showthread.php?t=50816&page=3

"By far the toughest part of this problem was trying to get BT to acknowledge that the problem was at their end. Slugged it out though and got there in the end!"

It's a small world after all...