SBS 2011 - Terrible network performance

[cyabro]

This one is really doing my head in

Got a SBS 2011 HP ML110 G7 server that is having network issues.
The problem is that network transfers from the server to any workstation are very very slow, like 300KBps slow!
Reading or copying stuff from the server is the main problem as writing stuff back is a bit better but still slower than it should be.
Transfering data from one workstation to another workstation is fine and speeds are what you would expect on a 100Mbps switch.

I've checked all the hard drives.
I've replaced the network card in the server, the cable to the switch and the switch itself.

I found a few posts on the net about others having similar issues and they all point to a couple of Microsoft KB articles, one of which says to install SP1 for SBS2011, which I did when the server was brand new.
The other KB article goes on about network offloading and TCP chimney and stuff that I have no idea about or what they do but I followed through the KB article and that made no difference either.

I've uninstalled the Eset Mail Security for Exchange and that made no difference.

Anybody got any other ideas?

I'm going to the client's site tomorrow (Saturday) to do some more troubleshooting to make sure it's not another device on the network that's causing the issues but I doubt it will be as network speeds are fine between workstations.

 

[Geeks4MacIT]

When files are transferring back and forth from workstation to server and server to workstation its best to have a gigabit network. I have a client that's also running SBS 2011 and before they moved to there new office they did not have a gigabit network. Accessing quickbooks took forever and there home care business software was also slow as it was accessing a database. Once they moved to there new location we made sure to have it wired for gigabit(not cat. 5e but cat. 6) and quickbooks was opening as if it was located on that machine and there database program also was quite faster. Most new machines have gigabit Ethernet ports. I would look and see if the workstations offer gigabit and if so test and see if that improves things.

 

[cyabro]

Sorry forgot to mention that I tried a gigabit switch at the site and it also made no difference.

 

[edtech]

Is there antivirus installed on the server? If so, is it doing real time scanning?

 

[Geeks4MacIT]

But did the router and computer both have gigabit ports?

 

[Geeks4MacIT]

Do you have an offsite back up going? I know this will also cause slow network speed.

 

[cyabro]

Is there antivirus installed on the server? If so, is it doing real time scanning?

From my original post:
I've uninstalled the Eset Mail Security for Exchange and that made no difference.

The only backups going is SHadowProtect, which I've also disabled and made no difference.
When I was doing the tests I made sure there were no backups running anyway.

This site is not a unique setup for me, I have 4 other sites using pretty much the exact same setup and they aren't having any issues.

 

[FremontPC]

Anything revealing in the Event Viewer, server or workstation(s)? What are you using for a router?

When you swapped the nic, did you use another of the same model or a different brand?

Copy works as expected from a flash drive to the server disk?

 

[cyabro]

No nothing in the Event logs.

The speed of the server itself is fine and doing copies from one HDD to another or backups with ShadowProtect to another HDD or USB drive is perfect, it's just the network speeds.

The network card was changed to another of the exact same type but I did update to the latest drivers.
That's one thing I am going to try this weekend is a different type of network card.

There is no router as such onsite as they have an Ericsson ONT for their fibre internet connection, which is faster then the internal network at the moment!
The ISP has a Microtik router at their end of the connection that manages the routing and firewall to the internet.

 

[FremontPC]

"There is no router as such onsite as they have an Ericsson ONT for their fibre internet connection, which is faster then the internal network at the moment!
The ISP has a Microtik router at their end of the connection that manages the routing and firewall to the internet."

This is the same as your other sites?

 

[datagnostic]

There is no router involved on internal file transfers, all is handled by the switches.

Is this a managed switch? 100 Mbps?

Try setting your speed and duplex manually on the server, could be a mismatch, although this shouldn't happen a lot, especially not on different switches.

 

[FremontPC]

Good point, datagnostic. Though if locking the nic down fixes it, I'd be tempted to toss it out and try a different one. What would be next up if the nic doesn't help?

 

[datagnostic]

The first thing you need to do is to verify that everything is correct on the physical side. All cables good (four pairs undamaged for Gbit, speed and duplex correct on both ends, ...)

It would be a lot easier to diagnose all of this with a managed switch. It's quite easy to spot mismatches or other electrical/cabling issues then.
Not so easy with a "dumb switch".

Alternatively, you could check the server's network statistics for excessive errors. There are some commands that you can use to find CRC errors and other errors that are the effect of physical issues:

netstat -e

C:\Windows\System32>netstat -e
Interface Statistics

                           Received            Sent

Bytes                     419926886        81049698
Unicast packets              562660          473664
Non-unicast packets         1203678           17312
[B]Discards                          0               0
Errors                            0               0[/B]
Unknown protocols                 0

netstat -e -s

IPv4 Statistics

  Packets Received                   = 323865
[B]  Received Header Errors             = 0
  Received Address Errors            = 3074[/B]
  Datagrams Forwarded                = 0
  Unknown Protocols Received         = 0
  [B]Received Packets Discarded         = 17253[/B]
  Received Packets Delivered         = 307831
  Output Requests                    = 134274
  Routing Discards                   = 0
  Discarded Output Packets           = 16
  Output Packet No Route             = 15
  Reassembly Required                = 0
  Reassembly Successful              = 0
  Reassembly Failures                = 0
  Datagrams Successfully Fragmented  = 0
  Datagrams Failing Fragmentation    = 0
  Fragments Created                  = 0

You might be able to tell from that if there are lots of bad frames received.

If there's nothing out of the ordinary there, you'll have to look at your server configuration.

 

[YeOldeStonecat]

What is the MTU set at on the NIC?
Whatever switch is being used....does it support jumbo frames? How large? If it supports jumbo frames, enable it..find out how large it can take..and enable it and set the size on your server.

Broadcom NIC?
I see it's an ML100 series....probably means SATA drives, which are pokey with SBS, (I'm not a fan of putting SBS on desktop drives) are they at least two pairs of RAID 1? RAID 1 for the OS and another RAID 1 for the data?

 

[datagnostic]

What is the MTU set at on the NIC?
Whatever switch is being used....does it support jumbo frames? How large? If it supports jumbo frames, enable it..find out how large it can take..and enable it and set the size on your server.

You would need to set the same MTU on all the client devices too. Otherwise they would receive oversized frames from the server and just discard the frames.

Unless the problem is actually caused by fragmentation I wouldn't touch the MTU.

 

[cyabro]

It's just a basic 10/100 switch unmanaged.

Yes it is also Sata drives in raid1, just one pair of 250gb drives for system and data.
This isn't a big company and they don't have a lot of data.
As I said this isn't a unique setup for us and we have other sites with pretty much the exact same setup and they have no issues.
We even have the same setup at our office except we're still on sbs2008.

Anyway I'm 100% sure it's got nothing to do with the storage system as I've tested that and speeds are fine.

I'm also 100% sure it's not a physical problem with cables or the switch as these have been replaced.
Also tested with a gigabit switch and made no difference.

I suspect either another faulty device on the network that's perhaps hitting the server with packets almost like a dos.
Or the network cards that I've used in the server as they are both the exact same model.

I'm going to test by disconnecting everything from the network except the server and one other machine and see what happens. If that works then start reconnecting devices one by one until the problem comes back again.
If that doesn't work then I'll put a different nic in the server which is different model to the others.

If that still doesn't fix it then not sure where I'll go next, hopefully won't have to think about it.

 

[Martyn]

Run Wireshark and have a look at the traffic

 

[cyabro]

Tried wire shark but it didn't do anything out of the ordinary.
I'm onsite at the moment and restarted the server and tested network speed and it's back to normal. Great! However I haven't done anything yet.
Keep on testing

 

[YeOldeStonecat]

You would need to set the same MTU on all the client devices too. Otherwise they would receive oversized frames from the server and just discard the frames. )

Correct..but since we're talking SBS here...can assume a relatively small network where this is easily done.

But now that the OP stated it's just a 100 meg switch in there...can't do jumbo frames, because jumbo frames are only possible on a gigabit network.
Seeing it's just a 100 meg switch in there...I'd look at number of workstations, other nodes on the network....and...well, that 100 meg switch would be my next target to replace.
As well as adding a 2nd pair of drives and going another RAID 1 volume and moving the infostore, user folders, shared apps, shared drive..and span the pagefile to that.

 

[cyabro]

So after some more troubleshooting it looks like it is the Ericsson ONT causing the issues.
With that disconnected from the switch I could ping the server from the workstations while transferring test files back and forth without any issues.
As soon as the ONT is connected back in pings to the server start timing out at random times.
I spoke to the ISP who said that they've come across this issue before.
They want me to write a letter to them about the issues so they can take it to the fibre provider (the company that actually installs the fibre and ont to the customers premises)

Glad I got to the bottom of it!