Weird network / website problems, would love input

[neotechnet]

Hi everyone,

I'm up against this problem for a month or two now that quite frankly makes no sense to me at all, here is the scenario.

The client has a cloud server hosted through us on intermedia.com. On this cloud server, in addition to it being their file server, also hosts a 3rd party website with a custom url.

This client has multiple locations, NY, DC and Long Island. They use LogMeIn Himatchi on their desktops to access the cloud file server as a local drive letter.

Yesterday at their DC office (Comcast ISP) only at that office they can't reach this website. They can't ping it, can't get to it via web browsers. I can visit it, their other locations can visit it.

We asked them to reboot the comcast modem, once they rebooted the modem the website was available again.

The next day in the NY office with Time Warner, the exact same problem happens again. They can't visit this website, reboot the time warner modem and it starts to work again. We also this problem in their long island location too. I think their NY office is a static IP address.

We've been in touch with intermedia about this, the first incident it took them "rebooting" their firewall and it worked. After that they say it's not on their end when we speak to them.

I have never experienced this, rebooting multiple ISP modems to allow access to a particular website? Besides pressing intermedia, can anyone think of anything else?

 

[RCD_Technology_Solutions]

DNS? could it be the issue?. I would use a common one non ISP like norton or google dns...

 

[YeOldeStonecat]

I agree with DNS being an issue. No so much from the ISP end, but I see the common denominator here being that hamachi VPN.

You can always PING by the way, the key is..what's the results? *Could not resolve? *Can resolve...but no replies? *Replies?

Next time the issue happens,
Have them ping the host by DNS name....and then have them ping it by IP address. Difference in results?

If they cannot get replies when they ping...don't go rebooting the router/gateway/modem first....run some more thorough tests to narrow things down. I bet that quirky hamachi VPN adapter is getting in the middle and messing things up. Try killing that service...re-run ping tests. And IPCONFIG /FLUSHDNS after the VPN is dropped...try ping tests again (both to host name and IP address..always run both tests).

 

[neotechnet]

Hi Cat,

If I remember correctly when I tried to ping it did resolve but just timeout.

Also if I recall we also switched one or more sites from inhouse DNS to google which instantly fixed the problem. Literally when we changed one workstation to google it worked but the above symptoms happened after that.

How could it be the Himatchi VPN? I'm not getting the connection? Is it trying to use that tunnel all of a sudden instead of the general WAN? Why would rebooting the ISP modems resolve it?

 

[YeOldeStonecat]

How could it be the Himatchi VPN? I'm not getting the connection? Is it trying to use that tunnel all of a sudden instead of the general WAN? Why would rebooting the ISP modems resolve it?

Dang Steve didn't even notice you were the OP.

That's a very good question. Software VPN puts in basically a dial up adapter in network components. It wraps around things in there...gets in the middle of the OS and the NIC and TCP/IP and just...muddies things up.

I'm generally not fond of software VPN...I avoid it when I can. Fine for short term things. But for full time office use, like an office spread around multiple locations that needs network access 9-5 every day...I'd setup VPN routers at each location and have hardware based full time 24x7 VPN tunnels.

Another thing with VPNs...they are not fond of NAT. Different routers/gateways can product mixed results. Home grade...typically gives issues sooner than business grade NAT routers than can handle VPN traffic better. Having multiple workstations behind that NAT router..all doing VPN at the same time..can compound issues.

A combination of...going through NAT, and software VPN clients making network stacks act fuzzy....I could see things like a DNS service getting lazy and falling down.

Different NICs...and versions of NIC drivers, interfacing with software VPN adapters....can also give mixed results....or, known issues.

Now..add to this...if "split tunneling" is used, or not. I'm not sure in this case, I don't know if Hamachi supports enforcing the disable of split tunnel or not..but if it does...that furthers the travel of the DNS request. Split tunnel is a common VPN where the workstation can access LAN resources and hit the internet out its own local gateway, while still accessing remote resources available via the VPN tunnel. Many places disable split tunnel...once you VPN in, you cannot access your local resources and cannot go surf the net or hit other internet resources out your own gateway, you have to do that through the tunnel.

Add to this even further, lack of giving QoS to the VPN. Typically if you setup offices with hardware VPN via routers, they can be setup to guarantee bandwidth to VPN traffic, so that someone listening to panboringa can't take down the network hogging traffic.

 

[neotechnet]

Haha yes it's me - Ever since buying the IT company in Long Island and buying a second one in Manhattan my goal in my daily life is to survive another day!

I never thought about Himatchi being the problem, the next time this happens I guess we can disable Himatchi on a workstation and see if that fixes that problem for that workstation.

I've spoken to intermedia several times and their engineers are telling me they aren't blocking / black listing IP addresses so it's not them.

 

[NETWizz]

YeOldeStonecat gave a very good response. Really, are will find your solution in what he said.

I am 99% certain the issues you are having are DNS related or the ISP is just blocking certain ports or protocols. I would try 8.8.8.8 and 8.8.4.4 for DNS servers (they are Google).

My tool of choice for troubleshooting DNS though is nslookup.

For example:

nslookup servername.domain.com

Server: somedns.someisp.com
Address: x.x.x.x

*** somedns.someisp.com can't find servername.domain.com: Non-existent domain

Force the lookup using a specific DNS server:

nslookup servername.domain.com 8.8.8.8
Server: google-public-dns-a.google.com
Address: 8.8.8.8

Non-authoritative answer:
Name: servername.domain.com
Address: 66.96.162.92


If you get a result with one DNS server and not the other, you can pin-point the problem to the affected DNS server.


***************************

Now the other thing I was going to say, which is completely off tangent is to stop using Hamachi etc. They all work but have convoluted sign-in processes, require additional troubleshooting etc.

My personal recommendation would be to firewall off the connection to the remote fileserver such that only communication through an IPSEC tunnel is allowed to it. Secure it with strong public/private RSA encryption by simply self-signed certificates on both sides for the public/private key encryption. Then you KNOW your stuff is private and it is NOT something that needs to be configured connected/disconnected etc.

Just tell the end computers that ALL data going to the fileserver goes through the IPSEC rules (and nothing else does). Tell the end sever to only accept inbound data from IPSEC such that it rejects all other connections unless it talks out first.


Oh, and if you want recurring business, self-sign the certificates for only a year, such that like clock-work the IPSEC tunneling breaks at the expiration of the certificate ;-) Nah, just kidding. Sign it for like 30 years and never mess with it again.

 

[neotechnet]

Hi all,

Soooo it happened again yesterday.

Office in long island can't connect to the cloud server via web browser, it times out. We try to ping the public IP address and it's timing out but here is where it gets interesting.

We have a second cloud server on that account, same subnet just the last 3 digits are different. I can ping that other server no problem but can't ping the one they are having trouble with. We also couldn't ping it from a brand new computer that didn't have Himatchi installed.

I rebooted the cloud server and they can instantly ping it as well as access it via web browser, they use a product called Ajera.

I spoke to Intermedia and they are saying from a network point of view both servers are on the same firewall so it can't be the network firewall and has to be a problem with the server itself.

Himatchi is running on this server as well as managed anti-virus from GFI. Can anyone think of anything where I could check that would cause this server to become unavailable for only 1 location and clears after a reboot? I'm at a loss, or maybe I just have a headache, or maybe a little of both.