Alerts!

[HCHTech]

We have an admittedly cobbled-together-but-working system for turning device & service alerts into job tickets. We have a 'notifications' user on our M365, and setup anything sending alerts to use that address. We have an SMTP2GO subscription as the mail server to do this. Then, we use Exchange transport rules to identify which client that alert belongs to (through a variety of ways, depending on that information is contained in the alert). Exchange then redirects that email to our ticketing system in such a way that it makes a ticket for the correct client. It's the next best thing to a real SIEM, without the expense. And before @Sky-Knight pops up to say that's a terrible way to protect our clients, I'll say it has served us just fine. (just kidding SK!) In fact, generally, it works way better than i thought it would when we first started to use this a couple of years ago. Before that we just didn't actively monitor those things - we depended on manual access to assess reported problems one-at-a-fricken-time. So it's a BIG step up from the old days.

We currently get alerts from iDRAC, Some HP switches, Unifi APs & switches, Synologies and EDR software. We have a couple of hundred Exchange transport rules to make this all go.

Whenever we add some new vendor's equipment or service to the system, it's a big job of filtering through the available alerts that equipment can generate, to find the alerts that are actionable. I don't want to know that the malware scanner finished a scan and didn't find anything, just as a quick example. There is no action implied by that alert, so I don't want it.

I'm in that process determination process now with firewalls. I want to add our estate of Sonicwalls to this process. Unfortunately, the job of picking worthy alerts is kind of big. With Unifi, for example, there are a total of about 20 alertable events. With Sonicwall, however, there are approximately 2,000 alertable events. Some are obvious (thermal hardware alarm is one example) but many, many of them are...less than obvious. I've read a couple of different 200 page manuals on events, but even so, many events remain unexplained, so it's difficult to know for sure whether I want to know about that particular thing.

I'm working with a test group of 10 firewalls (we have somewhere around 100 total in the field). So far, I've learned the hard way what alerts NOT to pick, haha. I used up my 40,000 monthly email allotment with SMTP2GO in one night when I had alerting enabled for Malformed IP Packets, and TCP connection aborts. After cleaning up THAT mess, I've been much more careful.

I'm focusing now on the various types of external "attacks" that the firewall can detect. Things like "SYN Flood", "DOS Detection", "FTP Port Bounce Attack", "DNS Tunnel Abuse", "Ping of Death", "Smurf Attack", "TCP Xmas Tree Attack", and my favorite "Spank Attack", haha. There are at least a couple of dozen others.

At first glance, knowing that all of these things are happening seems like an important thing. But honestly, as long as the firewall is protecting the network, I don't think ANY of them are actionable. What would be the expected action if I knew a client's firewall was fending off a Smurf Attack? I think the answer is "nothing", but wanted to run this by the brain trust here before I wholesale disabled probably 50 different alerts. It just seems wrong to ignore anything with the word "attack" in it...

 

[britechguy]

I don't think ANY of them are actionable.

About which I concur for those you've described. Getting alerts that protection is functioning as designed, and against attacks that are well documented, just means "all is OK." You can't expect common attacks not to be attempted, and the fact they're actually attempted means nothing if the alert tells you they've been repelled.

If the alert tells you something could NOT be neutralized/repelled, then *that's* actionable.

As you've recently relearned, there truly are more "normal errors/alerts" than there are actionable ones.

 

[ThatPlace928]

I turn off all information only alerts. I only want to know if something comes up I need to actually deal with.

 

[Sky-Knight]

@HCHTech All security tools are visibility tools. The first line of those tools are the notifications on all the things. SIEM is a collection point of all of that noise, but you still have to configure the bits that make the noise too.

You're right, it's not a SIEM. But it's also an essential state of operation before a SIEM, and one that's very wise. You're doing the right things! As you walk this road, eventually you'll find a SIEM is easier. But before you get there, you have to generate enough noise, that takes enough time to sift through to justify that investment.

So again, this is all wonderful stuff, and yeah... keep it up!

The only thing I'll suggest is get your Microsoft Partner portal stuff going, and get your customers linked in via GDAP. M365 Lighthouse isn't perfect, but it centralizes this configuration to a large degree so you can get to standardized alerting faster for anyone that's got at least a single Biz Premium license.

And Firewalls? Goodness... there's no good way to deal with those! Put them into a SIEM? Sure... but they're VERY chatty, and the data hard to parse, all of this means EXPENSIVE. Pull those logs out of your view, and a huge portion of your telemetry on the attack chains goes away, leaving you blind and in the dark. We're CONSTANTLY tuning the noise from firewalls, you're already dealing with this fact, buckle up!

 

[HCHTech]

I turn off all information only alerts. I only want to know if something comes up I need to actually deal with.

I agree 100% with this idea. The trouble is the determination of "what alerts represent something I need to actually deal with" is.....hard. Way harder than i thought.

 

[HCHTech]

You're right, it's not a SIEM. But it's also an essential state of operation before a SIEM, and one that's very wise. You're doing the right things! As you walk this road, eventually you'll find a SIEM is easier. But before you get there, you have to generate enough noise, that takes enough time to sift through to justify that investment.

Yes, I'm definitely not there yet, and not sure I'll ever get to the point where I think a SIEM is a good investment. Even with a SIEM, I suspect there is still a LOT of configuration that needs to be done so they know what I want to know about. I am 100% sure the process is not just "send them all the logs and they will figure out without your input what you need to worry about". For this reason, because we are a small company dealing with small clients, I don't think selling that service to my clients is in the cards. I could be wrong, ask me again in 2 years.

We're a Time & Materials shop, even though we quack a lot like an MSP. Any new service I want to bring on board means a sales pitch to my clients to pay for that service (including a profit margin). The sweet spot is turning every alert into a billable opportunity. It takes a bit of fine tuning for sure. I freely admit the size of the job in figuring out firewall alerts is WAY bigger than it was for any of the other stuff we get alerts for, but we'll figure out a workable setup. Eventually. haha.

 

[YeOldeStonecat]

We've had similar for years....
We have a legacy distro group, ops@...which goes to all of us.
A bit over 5 years ago started pointing those alert emails to our helpdesk@ email that dumps into Syncro (our RMM)

Endpoint security...yup
Servers and Workstations...done via RMM internally which opens a ticket, for any clients on one of our level plans
Network hardware..easily done in Unifi, as well as Syncro. Only for clients on level plans. if APs or switches or the gateway go down.
Microsoft 365 has an alerting section I'd configured as a "poor mans SEIM"...but we have (currently) SaaS Alerts that handles that anyways. May be changing SaaS Alerts soon.
Synology has their own multi tenant dashboard you can use for that monitoring..granted we don't have many of those left

Firewalls...I don't want to know all of the activity they are blocking. I remember back in the Windows 9X days when I started using Sonicwall SOHO3 firewalls..and then even more so when Win 2K came out..(and I started using Microsoft ISA Server via SBS2K Premium)....how NOISY firewalls were. My first glimpse at ISA Server logs was.."HOLY SMOKES"...but you gotta remember, yup, the internet is noisy, worms, trojans, grinding tools, probes...they're all out there bumping up against everything. Let NAT do it's job!

The ONLY time I set up alerts for suspicious/malicous activity on firewalls, was if I had port forwards going on..to an internal resource. Like back in the days of Terminal Server, RWW, OWA, RD Gateway. I'd have Untangle configured to sent our helpdesk alerts for things such as....if a grinding attack was happening against port 3389 back in the days of terminal server or RD Gateway. Then I'd quickly go into Untangle and block the IP that attack was coming from.

If I don't have any port forwards on the firewall....I don't want to know about the noise bumping up against its WAN. We have good endpoint protection on the workstations/laptops...as well as DNS Filter to help keep them from going to bad stuff. Windows firewall has been good in the past quite a few years.

The bigger attack vector is email accounts....and online services....so we move to protect those. I want their Microsoft 365 tenants properly armed and configured....and monitored (or if you also do Google Workspace...that too). Thus...have a form of SEIM watching over those. The amount of small businesses falling prey to compromise is staggering. You only hear about the "big names" in the news. You don't hear about the poor little small businesses in the news. Just had another "unmanaged" client fall prey to token theft the other day, a small restaurant management company...their boss got targeted..and token taken. He never let us kick up his 365 licensing to BizPrem....nor put him on a managed (thus monitored with SaaS Alerts) plan.

 

[Sky-Knight]

@HCHTech I don't have a ton of experience with a SIEM outside of SNORT growing "into one" over time, and Microsoft Sentinel.

And in that open source -> closed source leap the largest difference for me is that Microsoft knows how to grow an ecosystem. Every major firewall vendor worth a crap, has a supported, documented means to connect to Sentinel. That may be via syslog, it may be via dedicated connector the firewall vendor built, but there's some means of getting sanitized, and ready log details into Sentinel.

Which saves much of the faffing about trying to get things to work.

But it's still a TON of work!

Which is why I look at this Iron Scales / Cloud Strike "SIEM" BS and laugh. People "love it", "because it's so simple". And I'm sitting here going... yeah... it's simple. And missing 99% of the information to make effective security decisions. But it's "cheap", and "ticks the boxes".

 

[YeOldeStonecat]

We use a formal SEIM....from SOCSoter...
It's a little NUC device that sits on their network.
Scans the network....logs all sorts of traffic
Agents are installed on every computer ...and they report to it. That agent on the computer plugs deep into Windows, as well as the endpoint security.
It talks to the network hardware via SNMP....plugs into the Unifi gateway which has mirrored ports for the top switch and the uplink to the gateway...monitoring all traffic in and out.
It plugs into the GCC High Microsoft 365 tenant...gathering lots of info,
When they had their on prem server...the agent hooked deep into that for logging....
Monitors all software installed on all computers, notifies of exploits, patches, etc.

Yeah, detailed stuff. THEY monitor it in their SOC as part of their service and alert us of actionable things...I don't have enough time in a century to pour through those logs.

 

[HCHTech]

Firewalls...I don't want to know all of the activity they are blocking.

Yes, agree. What I'm finding out is that there is no alert for "I couldn't block this thing". I think identifying the things they ARE blocking is all focused on the dumb reports you can get that point out to the client that "We blocked 18,732 potential intrusions this month, aren't we doing a good job?" The sum total of the actionable alerts are basically either hardware issues, user problems or configuration problems. Those are still important things to know about, but obviously something less than I was hoping for.

For our EDR, for example there are several alerts for "unmitigated detections". THOSE are important and actionable. The firewall choices don't have an equivalent.

I like the system we have developed, I think it fits our company and our clients. We haven't added alerts for the RMM yet because the RMM dashboard itself serves that purpose. Once I get the firewalls figured out and we live with that for a while, I'll probably start focusing on the RMM, we'll see. The evolutionary step up is to farm this all out to a service like you guys have, but I think we have to go through the process ourselves to get to the point of considering that. I do appreciate the elucidations you've provided. This forum is just about the only reasonable sounding board in our industry. Reddit has a few subreddits on point for sure, but the whole interaction is different and much less valuable.

 

[Sky-Knight]

Honestly, firewall logs are bogus in general. If you're using the cloud correctly there are no ingress services. What is there to block? The security controls that matter already did that at the endpoint.

The firewall traffic logs are there for investigation of lateral spread, not notification of compromise. They are a sensor sure, but if things are designed and working correctly, that dataset isn't the one that gets your attention something has gone wrong.

There is a reason Palo Alto is making moves to get into the identity space to compete with Okta, Microsoft, and Google. They know, the days of the firewall are over. All anyone needs anymore is a basic monitor-able connectivity solution, which is what Unifi does really well.

You are correct to focus on the EDR, because it matters, and will continue to matter.