Retail store intermittent network upload timeouts

[lassenpc]

Hi all,

Grocery retailer store called with 'network bogged down' problem, started 10 days ago without any previous hardware or software changes to the network. ISP changed out DSL Modem 1 day after problems began with no change. Went onsite and have discovered thus far:

-Email (provided by ISP) from all workstations (XP, Vista, Win 7) and all email clients (Windows Mail, Thunderbird, OLE) will receive, but timeout on send (intermittently)
-Speedtest downloads are fine.. all uploads fail / timeout
-Malwarebyte scans clean; SaS clean; Various a/v programs (Norton, WSE, AVG) all clean

Spoke with ISP (Frontier), they blamed corrupted email on all workstations. Deleted / recreated pop3 account, worked once, then timeout on send again.

Spoke with store corporate IT, they checked the zywall FW (I do not have access to that); made adjustments, able to send mail now, however speedtests on all machines still fail on upload.

Local Frontier ISP Tech dispatched again, says 'it sounds like a physical short in a switch or ethernet cable; the problem is not on our end'. The infrastructure is a nightmare, multiple 8 port switches daisy chained throughout the store. Attempted disconnect of several workstations / switches to isolate, fail.

Current troubleshooting steps:
-Continue to check / search workstations for possible botnet / sniffer?
-Check POS Server event logs for additional info
-Check POS Server to see if hacked / compromised
-Start replacing switches
-Replace hardware FW - ordered, enroute
-Speak with ISP Business Support again to check for changes on their end
-Start replacing cable??? (last resort)

The main concern is it is starting to (randomly) affect the credit/debit transaction processes at the (ten) point of sale terminals, they are losing communication (unable to upload info)

Has anyone seen similar? What else should I be doing, where else should I be looking? Thank you!

[CarrobesIT]

First thing I would look to do is see if you can hook a single PC directly into the ISP's DSL modem and run the SpeedTest/try sending an e-mail from there. I would do this potentially with a couple of PCs and several ethernet cables to the modem (possibly try a brand new one too). This way, you isolate any & all of the customer's infrastructure and can either prove/disprove that it is the ISP's problem.

At the end of the day, they cannot argue with the results of two PCs, each plugged directly into their modem. And if you find that it is working fine directly into the ISP's modem, you can start adding infrastructure components back in until you find the cause(s).

Rob.

[deutschnaftula]

as carrobesIT said i would take my own laptop and connect it the the ISP modem and go from there

[lassenpc]

Update (Solved):

Corporate IT kept pointing the finger at the ISP as there had been no software/hardware changes made. ISP came onsite and checked lines, had already changed out modem once (after this had started) confirmed with ISP business support, and they said everything looked good on their end.

So we changed the ISP Modem settings to allow PPOE Passthough, then changed settings from 'PPOE' to 'Bridged'; on the next device, the Zywall FW, entered the PPOE settings.

Everything is working fine now. Plausible culprit, authentication issue with the ISP.