IP Credit Card terminals connection issues

[HCHTech]

This is a new setup in a retail store. New Cat6 wiring and all new equipment, save for 2 four-year-old POS terminals.

So....two checkout lanes, each with two network drops, one for the POS terminal (POSx units running Windows 7 embedded standard, and one for the IP credit card terminal (new Ingenico IP320s). The building has 25/5 DSL service, a single static IP, and the ISP gateway is in bridge mode. There is a Sonicwall SOHO-W and a small Netgear 8-port switch. Besides the POS & cc terminals, there is a single office computer and a shared network printer. There is also a single Ubiquiti AP & Cloud Key. 8 total networked devices.

The problem we're facing is credit card transactions being cancelled because of connection issues. What we've found in our initial analysis is that a simple ping command to credit card terminal #1 has about a 14% failure rate, and pinging credit card terminal #2 has about a 35% failure rate. Not good.

All devices are currently on the same subnet. I get 100% success on pinging the network printer from any device, I can also ping from one POS terminal to the other or from the office computer to either POS terminal or from either POS terminal to the office computer with no dropped packets.

Both CC terminals have static IPs, and I have excepted them from the various scanning by the firewall security services (Content management, IPS, AV & anti-spyware).

If I connect a laptop to the same network port previously occupied by either cc terminal, I can ping it with 100% success from any other device on the network, and vice versa. I even tried giving it the same IP as the cc terminal it replaced so that it's traffic was treated identically by the firewall. I still had 100% successful pings.

Disabling all other devices on the network doesn't seem to impact the problem. Disabling most services on the firewall doesn't seem to impact the problem. Disabling the Windows firewall on the POS terminal doesn't impact the problem. Disabling the antivirus on the POS terminal doesn't change the problem. I moved the cc terminals connections from the switch to two unused ports on the firewall just in case the switch was slowing things down somehow - this didn't improve things. I ran an IP scanner to check for address conflicts on the network, there were none. I did NOT try removing the switch entirely, they are in the middle of a soft open and have to checkout stray customers now and then, so I've got to be careful what I take down.

So, it's got to be the cc terminals themselves, right? They have those proprietary network cables with an HDMI connection on one end and a DC power connection inline, so I tried swapping cables between the terminals to see if the problem followed the cable - it did not. I took one back to the shop so I could see how it responded to pings in a completely different network. Unfortunately for me, there was no packet loss over several hundred pings on my network (also protected by a Sonicwall, btw).

So something about the network in the store is causing this problem, but where to start for diagnostics. All of the other network devices are working just fine and showing no network problems at all. I did take several minutes of wireshark logs before I left today, but I'm not seeing anything that sticks out as a problem.

Where would you start?

 

[GreyWolf]

Well for one of my clients he had something similar and believe it or not I disabled ipv6 from the router and all the problems vanished to this day I still have no darn clue why that fixed it but it did only clue I had was the sharing printer was crossing 2 networks. Just out of curiosity did you power cycle the whole thing down? I know with the system I was on with VPN to the main corporate office needed a full system reboot from time to time until they had changed the main box that communicated with them. As for the cc machines they definitely look like the same beasts I had to deal with.

 

[Diggs]

I've had a spate of residential issues where the client lost connectivity to the Internet and the issue was resolved by disabling IPv6. Yet in 99% of the time IPv6 is enabled with no issues on an IPv4 network.

 

[HCHTech]

Ok, thanks for the suggestions - I have disabled IPv6 on all internal interfaces on the Sonicwall, and I've disabled it on the 2 POS terminals and the office computer as well. I have NOT done a "whole network" power cycle, because they are open for business, we'll need to schedule that carefully. I'm also wonder if I need to enable legacy TLS in the Sonicwall - they have a setting in the diagnostic interface titled "Enable TLS Compatible Mode". That requires a reboot of the firewall, so again, I'll have to schedule that carefully. I suppose once I do figure this out I might have learned something for future setups - I hope anyway. I just wish I could get something to give up a clue or two. Blind troubleshooting isn't a fun way to spend your day.

Edit: I found that TLS 1.2 was NOT enabled on the Windows 7 POS terminals, so I did that, but it also doesn't seem to have made a difference.

 

[GreyWolf]

Well put it this way my client wanted to have his printer working on both networks and having one network that could see one but the other not so believe me I know how you feel about blind troubleshooting.

 

[nerd2u]

Have you checked the cabling to verify it's good?

Also call the payment terminal vendor they can likely help.

Sent from my SM-G870W using Tapatalk

 

[Markverhyden]

Have you checked to see if there is a firmware update for the devices?

Forgot to add. Did you re-terminate both ends? I know it sounds strange but you can have a line that actually tests good, like you did with the laptop, but actually have a problem with another device. I was racking and turning up a new Cisco router couple of days ago where I had a similar issue. Customer had run a dmarc extension from the fiber media converter to their server room. Weren't getting a link light on the Cisco. After short cabling my laptop determined that the problem had to be with the run even though it tested to cat6 standards. Re-terminated the run on both ends and we were good to go.

 

[fencepost]

Is that small switch Gigabit? Do you have a 10/100 switch that you can try plugging the terminals into? They may be having autonegotiation issues.

 

[HCHTech]

Some progress. I did reterminate one of the runs, but that didn't help. I unhooked everything from the network (including the switch) and connected just those 4 runs directly to the firewall. I found that as long as the terminals were connected directly to the firewall with a short cable, I could ping them just fine - 4 50-ping runs with 0 dropped. I could do this from a computer or either POS terminal, and I tested all 4 drops in the register lanes this way.

If, however, I reversed this, and put the computer directly on the firewall with a short cable and put the credit card terminal on any of the 4 drops in the register lanes, I got packet loss. I tried forcing the firewall ports the register drops were attached to down to 100mb instead of auto-negotiate, no difference.

The vendor checked the firmwares and we apparently have the latest. I will try to confirm this, but this was done while I wasn't onsite.

The thing I didn't do is take an 80-foot cable (because I didn't have one handy and didn't bring a spool of wire) and put the cc terminal at the end of that and test it. If that works, then it's definitely the runs. If that also fails, then it's definitely the cc terminals having trouble with cable length.

Also, If I put a computer on the firewall and a computer on any of the 4 drops at the register area, I got 0 dropped pings. If I put two computers on any 2 of the drops at the register lane and ping from one to the other, which means the signal had to go to the firewall and back, I also got 0 dropped pings.

So, the cc terminals seem to be sensitive to the long runs, or there is something wrong with those runs that the computers don't care about, but the cc terminals do. Reterminating one run on both ends didn't help this situation, so we're in a corner.

The mechanical solution would be to move the firewall to the register area and cable the cc machines directly to the firewall. This is obnoxious and not the solution I want, but we'll do it if the vendor doesn't have any bright ideas to help.

I might add that this client has two other stores using the same equipment. In both of those stores, we had to deal with already-existing infrastructure and there was only a single drop at each register. These runs are at least as long as the runs in the problem store. So we put 4-port switches at each register drop and connected both the POS computer and the cc machine to those switches. Both of those locations work. I will definitely try this solution at the problem store, even though I got to consult on the initial setup and recommended individual drops. Makes me look bad, but at this point they need to get things working, so there it is.

I'll report back with the final solution.

 

[HCHTech]

Is that small switch Gigabit? Do you have a 10/100 switch that you can try plugging the terminals into? They may be having autonegotiation issues.

It is indeed. My testing was without the switch in the loop, so I don't think it is at fault. I might finally understand why they still make 10/100 switches, though!

 

[YeOldeStonecat]

Yeah I've headaches with some specific CC terminals.....it's been a while but one of my clients that owns a few restaurants got tired of them and went back to ones on dial up (instead of the networked ones).
Things he tried:
*Bitching to the ISP...having ISP run fresh runs to his restaurants....swap out modems, newer modems (latency)
*I put in new switches..HP ProCurves
*I dialed down the ProCurves to 100 meg on those ports...helped a bit
*Replaced his router with an EdgeRouter (low latency)
*Had his electrician run brand new data runs....terminate properly in a patch panel, replace all patch cables round around the messy place (originally it was an old old buncha CAT 5 strewn everywhere...old, no proper patch panels, poor terminations..I thought for sure this would be the answer to his issue but it wasn't)

His networks were small..not like there was a bunch of chatter 'n em. So yeah, like you...I got frustrated and after exhausting work above and testing like you did...nothing left but to point to the CC terminals. So since that's the only model his CC vendor had...he put them back on dial up. That was a few years ago, dunno if he went to another CC clearing house with different terminals and is back on ethernet.

 

[nerd2u]

Are the pin pads also connected to a telephone line? I've seen this cause issues in the past

Sent from my SM-G870W using Tapatalk

 

[fencepost]

we put 4-port switches at each register drop and connected both the POS computer and the cc machine to those switches.

That may not be pretty, but if the terminals are sensitive to the run length that may be the simplest and cheapest option. It may be that the terminals have poor network hardware that's either not sensitive enough or not powerful enough for the amount of attenuation over the longer runs, but that's going to be hard to test without appropriate equipment. I'm assuming that if you were going to go all Marvin Bee and throw a Fluke on it you'd have already done so.

 

[HCHTech]

So we put 4-port switches at each register drop and connected both the POS computer and the cc machine to those switches.

Well huzzah and saints be praised, it freakin works. Rather than moving the firewall, I just installed 2 4-port switches, one by each register. I plugged the switches into the network jacks for connection back to the firewall and then plugged both the POS computer and CC terminal into those switches. 100% ping return over 500 pings. I guess I wouldn't have been sure, but this must mean that traffic between two devices on a daisy-chained switch is smart enough to NOT go back to the router. I think. I'll wait for @NETWizz to chime in on that. I only hope this hard-earned knowledge will come in handy at some point in the future.

I'll admit another success - by keeping the client in the loop every. single. step. of the way on this problem, I managed to remain the hero. They got copied on the entire tech-support email chain with the POS vendor, and I saw both the main manager or the owner every day while we were working through this. On my way out this afternoon, the owner shook my hand and said "Don't forget to send me a bill!". Oh, I won't, don't worry - haha.

 

[HCHTech]

I'm assuming that if you were going to go all Marvin Bee and throw a Fluke on it you'd have already done so.

Ha ha. I thought about trying to rent one of those, but I'm glad I didn't.

 

[mikeroq]

I guess I wouldn't have been sure, but this must mean that traffic between two devices on a daisy-chained switch is smart enough to NOT go back to the router. I think.

All switches work on at least Layer 2 of the OSI model. They use the mac address of the devices to decide where to send data. Hubs on the other hand are Layer 1 physical only. That is why when data is broadcast to a hub is just repeats it on every port.

Since the switch works on Layer 2, the data link layer, it stores the MAC addresses of the connected devices and maps them to their ports. When both devices are on the same switch the traffic swill stay on the switch.

Here are some animations that might help: