Anyone with Wireshark experience?

[BrandonTech]

I've been having an intermittent issue with Chromebooks dropping from our network. I am hoping someone here with Wireshark experience can maybe point me in the correct direction if it is possible from the capture shared at the link below. Around packet 210,000ish if you filter by eth.addr == 10:08:b1:61:96:eb, there seems to be weird things happening. Lots of DNS lookups that append the local domain to the query. ARP lookups for the default gateway of the affected subnet also appear.

The issue only seems to affect the Chromebooks (and the Acer c720, particularly more often).


Affected devices during this capture were...

10:08:b1:61:96:eb
38:b1:db:47:9f:c7
10:08:b1:61:92:55

Link to capture...

https://drive.google.com/file/d/0B-fsB7bguAOhTVJxQnRXaDJyRHc/view?usp=sharing

 

[TurricanII]

Long shot, and I haven't looked at the capture file, but what is the DHCP lease length on your WiFi?

I have seen some TPLINK Wireless Access Points ship with a 60 second lease time. This caused intermittent dropoffs but only for Apple devices.

I'd also be checking that capture file for duplicate IP addresses - two MAC's replying to ARP requests claiming the same IP.

 

[BrandonTech]

Long shot, and I haven't looked at the capture file, but what is the DHCP lease length on your WiFi?

I have seen some TPLINK Wireless Access Points ship with a 60 second lease time. This caused intermittent dropoffs but only for Apple devices.

I'd also be checking that capture file for duplicate IP addresses - two MAC's replying to ARP requests claiming the same IP.

Thank you for the reply. Lease length is controlled by our DHCP server on Server 2012, and is set to 6 hours. Doing a quick filter of the particular MAC's "or" the known IP address of that MAC returned the same number of packets displayed, so I guess it's safe to say no other device received that IP during the capture.

Does anyone know if it is normal for DNS traffic to begin to act like the following?...client sends a request for clients4.google.com... Our DNS server responds with the IPs (there appear to be several for that particular domain name). Then, right after that response, client sends out a second request for the same domain but it is appended with our local domain name like this... clients4.google.com.contoso.local... DNS server responds that no such name exists.

I don't understand why it would still be looking for something it just resolved and on top of that adding our local domain to the query.

Also, during this time is when I see more BADTCP traffic than usual and I start to see ARP requests for that subnet's default gateway.

 

[TurricanII]

I would look for ddos protection or similar on your gateway or wifi and see if you can disable.

The DNS lookups might be related to prefectching where chrome tries to look up the IP before you finish typing it. That might explain the put of order lookup followed by the .local lookup.

Some DNS clients append the local domain issued by DHCP to see if the host is local.

 

[NETWizz]

I have not had a chance to look at it, but any reason you are predominantly looking at the IPv6 traffic?

At any rate, all of the items you describe are typical. What you will not see is a reason for a dropped connection most of the time unless you capture a wireless reset frame (unlikely). Typically, when a drop occurs, you simply see nothing until it re-establishes in which case you see the DHCP negotiation, TCP three-way handshakes etc.

 

[Markverhyden]

Server 2012 DNS <> Chromebook Linux? That's the problem I would guess. I'll look the logs over to see if I can add to the conversation.

Few questions

You say dropping from the network. That means they loose their IP address and cannot be seen via a network scan? Is that correct or did you just mean that they have intermittent Internet connectivity issues?

Do these machines access any local resources, like drive shares? If not I'd try to manually specifying the DNS to Google or your ISP.

 

[Markverhyden]

Looks like it's a Chrome, assuming you are using it on a Chromebook, problem. Try FF and see what happens.

https://isc.sans.edu/forums/diary/Google+Chrome+and+weird+DNS+requests/10312/

 

[BrandonTech]

I have not had a chance to look at it, but any reason you are predominantly looking at the IPv6 traffic?

At any rate, all of the items you describe are typical. What you will not see is a reason for a dropped connection most of the time unless you capture a wireless reset frame (unlikely). Typically, when a drop occurs, you simply see nothing until it re-establishes in which case you see the DHCP negotiation, TCP three-way handshakes etc.

No reason for the IPv6 traffic. Maybe I should have filtered that out for the capture. We do have the ability to pull diagnostics from our access points that do appear to include packet captures. I have sent these to the access point manufacturer, and their first glance response was that it appeared that the device was going into power save mode.

The issue with that is that the problem happens during active use of the device, i.e. the user will lose internet connectivity while browsing.

 

[BrandonTech]

Server 2012 DNS <> Chromebook Linux? That's the problem I would guess. I'll look the logs over to see if I can add to the conversation.

Few questions

You say dropping from the network. That means they loose their IP address and cannot be seen via a network scan? Is that correct or did you just mean that they have intermittent Internet connectivity issues?

Do these machines access any local resources, like drive shares? If not I'd try to manually specifying the DNS to Google or your ISP.

As you determined, these are Chromebooks (Acer c720's predominately). From everything I can tell, the device is only loosing internet connectivity. I would like to set them to all use Google's DNS servers. I have 499 of these (not all are Acer c720s), and I am trying to determine if there is a way to set this via Google's admin console.

 

[Markverhyden]

I think if you set the DNS to something else that should solve the problem. Between looking at your pcap file and the link I posted above it looks like it's related to the pre-fetch stuff that @TurricanII mentioned. For some reason the client wants to add the .local even after it has successfully resolved the IP. Generally speaking you can manually specify a DNS server even if the client is getting an IP via DHCP. By Google's admin console I'm guessing this is some global tool? Is this related to the domain or is it a local domain tool?

Have no experience with Google/Chrome and the Edu market, but do know it's very popular. I'm in the middle of a switch and upgrade project at a nearby school system and know they are planning on rolling out Chromebooks as well as allowing BYOD. So I'll bring this up with them to see if they have tested Chromebooks and see if they have run into something like this as they have their own DNS as well on a M$ server.

 

[BrandonTech]

I think if you set the DNS to something else that should solve the problem. Between looking at your pcap file and the link I posted above it looks like it's related to the pre-fetch stuff that @TurricanII mentioned. For some reason the client wants to add the .local even after it has successfully resolved the IP. Generally speaking you can manually specify a DNS server even if the client is getting an IP via DHCP. By Google's admin console I'm guessing this is some global tool? Is this related to the domain or is it a local domain tool?

Have no experience with Google/Chrome and the Edu market, but do know it's very popular. I'm in the middle of a switch and upgrade project at a nearby school system and know they are planning on rolling out Chromebooks as well as allowing BYOD. So I'll bring this up with them to see if they have tested Chromebooks and see if they have run into something like this as they have their own DNS as well on a M$ server.

The Google Admin console is not tied to our local domain seen appended to the DNS lookups in the packet captures. It does allow you to manage Chrome devices and Users with policies. I guess it could be called Google's version of AD and Group Policy. You can pretty much set any option for both users and devices in there, but I have yet to find a place to push out Google DNS servers to all devices at once...which is sort of strange to me considering this is an actual radio button/check box on the devices themselves to switch to Google DNS.

Please do see if they have experienced in issues with their Chromebooks intermittently and suddenly losing Internet connectivity. If you can, see what access points they are using. We have all Xirrus AP's.

I saw one post on Acer's forums referencing what @TurricanII was saying about ddos. They were able to determine that their Chromebooks were sending out UDP floods. I haven't found a way to turn off ddos monitoring on our AP's, but I have come across some filtering options that may do the same thing. It has a pre-configured filter called Air Cleaner, that appears to block out certain types of multicast traffic among other things. I don't have it open in front of me right now, so I can't remember everything it tries to filter.

I'm beginning to wonder if a combination of the number of devices (say, 30 at most, but predominately Chromebooks) on "weaker" 4 radio AP's is causing the issue if they are flooding the AP with unnecessary traffic. These same model Chromebooks are used in other locations, yet the problem doesn't appear to exist or isn't as noticeable.

 

[BrandonTech]

Yesterday, I decided to see how the Chromebooks would do if placed on a network that didn't require wireless authentication/encryption. We had them on one that was with WPA2 AES/PSK.

In initial tests, the issue appears to be gone. So something may be going on with these devices and the way they are authenticating to the wireless. ...Still looking into it.

 

[Markverhyden]

Interesting. Are there any certificate servers or anything like that involved or just a straight forward PSK?

 

[TurricanII]

I'd be looking at various key lifetimes to see if some part of the authentication is not rekeying.

 

[BrandonTech]

Just a straight forward PSK.