Need help with a network that's gone through major growth spurt recently

timeshifter

Well-Known Member
Reaction score
2,171
Location
USA
In the span of about 18 months one my bigger clients has gone through a growth spurt and has about tripled the size of their network.

At the beginning, and for the last 8 years it was pretty simple. One server running LOB. Traditional phone system. About 15 PCs, two printers. Now they have VoIP phones, fiber Internet, 40 PCs, added on to their building. New wiring for the new section and now rewiring original section. Every location now has at least two wires, one for phone and one for a computer. I think it's about 190 ports on the patch panels now.

The wiring vendor is done and we're now in the process of moving all the gear into the new rack and patching everything together neatly. Trying to decide on how to patch everything in.

There will be three switches, all 48 port, every port PoE. There are three 48 port patch panels above each switch with a wire manager in between.

Here's a list of the network gear:
Internet connection 1 - 25/25 fiber connection (primary connection)
Internet connection 2 - 200/10 coax (used as failover and some high data devices are routed to it)
Gateway - Cisco Meraki MX68CW
Traffic Manager - VoIP provider Microtik box, sits between gateway and rest of network so they can manage call quality
(3) 48 port UniFi PoE network switch (two old version, one new version)
(4) UniFi 6 APs

Connected to all that:
41 PCs / servers
39 VoIP phones
16 iOS / mobile devices
10 misc printers and other devices

I haven't quite decided how to interconnect all the switches. That also influence how I'll patch in the phones and PCs, potentially.

OPTION A:

Shortest path and short cables. Phones and computers get plugged in to the switch that corresponds to their patch panel.

OPTION B:

Patch devices to function specific switches. Thinking I would put all PCs and servers on one switch. VoIP phones on another switch. APs, printers, other items on the third switch.

Seems like this might be helpful. All PC and server traffic would be contained in one switch. If there's a problem with the PCs and the server application I could reboot just that switch if needed, and not take out all the phones. Or if the phones are acting up the phone switch can be rebooted or whatever.

The rack may not be as neat, but it might be more serviceable. Note that although all phones have a PC port, so we could just use one run for a phone / computer, but they've put in dual lines to every location. So I'd like to take advantage of that.

ALSO, should I get different colored patch cables for different devices? Is there an industry standard like one color for phone and one for PCs?

There's a lot I need to sort through and may need some outside help to get this right, but for now I need to know OPTION A or OPTION B as the vendor will be supplying the patch cables and we're meeting tomorrow to get that ordered. Will need to know what lengths to order and colors.
 
My preferred approach, leaving more room for growth....
You have 3x 48 port switches. I'd put a TOR switch above them (Top Of the Rack)...a Unifi 16 XG for example.
Not sure exactly which model 48 switches you have, hopefully the ones with at least the SFP+ options..
This allows you to take each of the 48 port switches...and uplink them to the 16XG via 10 gig uplinks. You link your gateway into the 16XG also, and the server(s),and VoIP Edgewater/PBX.

The alternative approach, not adding more equipment, take the top 48 port switch and treat that as TOR. Link gateway into it, server(s), and uplink each of the two lower 48 ports into it via SFP+.

For these uplinks, 10 gig DAC is fine, and low cost.

Right off the top of my head I see 3x VLANs.
Default data/production VLAN
Guest VLAN..we usually pick 6 for this
VLAN for the VoIP...we usually pick 2 for this

You can set LLDP MED for VLAN 2...which is an "auto discovery" protocol most IP phones support...meaning they will automatically search for the VLAN advertising LLDP....makes it easier to setup IP phones, don't have to go to each one and program which VLAN they should look for.

For the APs...I don't know the size of this building, but with three 48 port switches, and that amount of workstations, 4x APs may be a little low for good coverage. More APs (higher density) at lower TX power is better. Don't forget those in-walls to help some areas...they still allow data and POE ports coming out.

On larger setups like this, is where you get to enjoy the power of the Unifi controller.....and how they handle VLANs....a feature called "profiles". A profile controls which VLANs are members, as well as...POE settings. You want to have a profile that includes POE=>OFF....for ports facing your servers, and your gateway, and other expensive equipment. For the ports that have APs on them, I have a profile that has "isolation mode=>enabled" (unless you have wireless printers). Port isolation helps cut down on broadcast chatter across the APs...which improves airtime quality, which improves wireless performance.

Don't forget to disable wireless uplink connectivity monitor if you're not using wireless uplinks on your APs....also helps performance by removing those regular broadcast heartbeats.

The port facing the Edgewater(PBX) box for the VoIP...have a Unifi port profile for that, setting VLAN2 as default, with no POE.

The guest VLAN, you can utilize one of the available ETH ports on the Meraki gateway, setup a different IP network there....like 192.168.10.0/24, run DHCP on it, and you can have a Unifi port profile on the port facing that effectively is untagging VLAN 2 and no POE.
 
Also, if you're getting into Ubiquiti more for clients...and they're biz clients, I highly recommend you use Hostifi for your hosted Unifi controller. Reilly and Safwan are great, responsive, and have a good service. Yes it costs money, but it beats trying to maintain your own controllers, worrying about backups, upgrades, security, etc. We started adding a line item to our MSP plans years ago for our clients with more network hardware. Now we just cover a lot of various little services costs in 1x first line item called a Core Service.
 
Now they have VoIP phones, fiber Internet, 40 PCs, added on to their building. New wiring for the new section and now rewiring original section. Every location now has at least two wires, one for phone and one for a computer. I think it's about 190 ports on the patch panels now.

Just wondering about the count. If you really have 190 runs, then 3x48=144 switch ports is going to be a little short. I will say that setting up the patch panels and switches so that phones are segregated from data does save troubleshooting time down the road, IMO. That size setup isn't really my world, though, so take this with a grain of salt.
 
That size setup isn't really my world
Mine either :D

So I'll show some more ignorance... how do VLANs help here? I know they segment and isolate things on the network, but since everything is running over the same infrastructure, I don't really see how it helps.
 
Here's what the closet looked like a few weeks ago, don't have a more current picture. Most if not all of the stuff on the left hand wall is the old stuff. The things I've boxed in orange have been rewired and will be relocated to the new rack, like the switch that sticking out. All the 66 blocks are gone now.

So basically its:

48 PORT PATCH
WIRE MANAGER
48 PORT POE SWITCH
48 PORT PATCH
WIRE MANAGER
48 PORT POE SWITCH
48 PORT PATCH
WIRE MANAGER
48 PORT POE SWITCH
48 PORT PATCH
WIRE MANAGER
BLANK
24 PORT PATCH for cameras
BLANK
BLANK
BLANK


We'll have some shelves and misc items down low for modems, etc to come over.

If I do a TOR switch, does it really need to be at the top. I could have them move everything down 1U.

closetcleanupwip.png
 
VLANs help isolate the traffic...and ensure it flows more smooth. Less collisions. Lessens the broadcasts from mixing it up too much. Even though physically the same total bandwidth....it helps separate things so that they can share the same physical bandwidth better. Also, depending on how configure, can provide security...keep networks separated for security purposes.
 
Regarding your color question. That's up to you. Years ago I used to do that. Green for the gateway, black or white for the servers, something for workstations, and now something for phones. But I stopped doing that.

Label the patch panels
Label the switch ports ...and map out on paper what you have for ports and VLANs.

And use those new "slim" cables for jumping from patch to switch. They just look SOOO much neater, and easier to work with. I use as short as possible. If you're in any of the IT related FB groups, look for some pics that Chris Tiffany from Sprinter Tech posts of his work...he does a lot of wiring/network closet work. Incredible work to aspire to!
 
What about physical switch separation - all PCs and servers to one physical switch, all phones to another, etc.?
 
I'm going to echo the advice of the TOR switch. And, if that isn't available just make sure you lay things out as a STAR, not a CHAIN.

Chain too many switches and annoying things happen. You want to ensure that there are no more than 3 switches involved with any network transaction if you can avoid it, 4 works too... but much more than that and things go wrong. I like 3, because that's plenty of depth to get to a router, from there you have 3 more!

As for how you plug it in, it's up to you! Assuming you have 10gbit links between the switches you've got enough bandwidth to largely ignore any real concerns locally. So you're just down to how you want to organize them. Assuming these devices are tagged into their own VLANs, having them connected to a contiguous block of switch ports helps keep the configuration clean. I like to color code the ethernet cables too if I can.
 
As I sit here thinking about it I’m liking the thought of having the phones all physically on their own network using the fiber connection, then all the PCs and related items on their own network using the coax.
 
What about physical switch separation - all PCs and servers to one physical switch, all phones to another, etc.?

Highest load components....spread across the TOR.
So the TOR....
Port 1...uplink to gateway
Port 2..uplink to PBX
Port 3...uplink to a server
Port 4...uplink to another server
Port 5...uplink to US-48-1
Port 6...uplink to US-48-2
Port 7...uplink to US-48-3
Port 8...AP1
Port 9...AP2
Port 10..AP3
Port 11...AP4
Port 16...uplink to ETH3 (for example)..on the Meraki...which is setup for the guest wifi network. VLAN 6 untagged

Like Rob said...perfect analogy, "Star" topology, not daisy chain. Avoid switch 4 uplinking to switch 3 which uplinks to switch 2 which uplinks to switch 1. All that does is congest traffic...and clients on switch 4 have a few hops to get to the servers and gateay(internet). Always take the "star"...or "spoke 'n hub" approach...each and every client should have the SHORTEST path to what they want (servers and internet).

Not really a need to separate stuff "per switch"....just, focus traffic so the shortest path to the goods.
The APs could be on any switch really...no big difference if they're TOR or not, but "on paper"..."in theory"..best to have them TOR, since each AP can have multiple clients on it.
 
As I sit here thinking about it I’m liking the thought of having the phones all physically on their own network using the fiber connection, then all the PCs and related items on their own network using the coax.

On some really large setups I've done that. For example, a school, where I have one of Robs NG-500 firewalls running Untangle....I'm using 5 or 6 of the ETH interfaces...and it's multi WAN. Well, I still let the phones get mixed up with the computers...on the lower switches. But I'll use the VLAN management...and the gateway, to dump the PBX traffic out of a backup/failover WAN connection....since that is not used by regular internet traffic. You may have to play with this...as ISPs vary. You may find one WAN connection, or the other..works better for the VoIP.

But trust the horsepower of todays switches...and gigabit...you can mix up VoIP with the computers in the switches. Honestly....since phones have gigabit passthrough ports now (unlike just a few years ago when many were just 100 megs)....you can have 1x data jack in an office. Patch cable from that to phone...and out the other side of the phone (the 2nd ETH port under it)...a 5 foot patch cable to the computer. The VLANs will work their magic.
 
And use those new "slim" cables for jumping from patch to switch. They just look SOOO much neater, and easier to work with. I use as short as possible.
Like this?

 
Back
Top