Tips for selecting a managed detection and response company based on Actionable, Relevant and Timely response criteria.
Selecting a Managed Detection and Response Company is a Complex Task
Cybersecurity is often difficult to understand, especially for the average business owner. These people devote their time to managing their enterprise, not studying cybersecurity tactics and trends. Unfortunately, that means these businesses may misinterpret basic cybersecurity positions.
For instance, many business managers interpret the absence of data breaches or cybersecurity attacks as a metric of success. This interpretation can complicate vendor selection and review. (There are, however, other ways to look at the absence of a breach. Perhaps there were ‘no incidents’ because the attack was not detected. In this case, an adversary can siphon off intellectual property without the company’s knowledge.)
Another influencing factor is the perception that your business experienced a breach (AKA a ‘security event’) because you did a bad job protecting your data or you got very unlucky. This interpretation allows shady vendors to hide behind probabilities – at least until an event occurs where they cannot deny the fault in their stance.
At that point, they simply move on to the next victim customer.
How can a business owner or manager tell if their cybersecurity vendor is providing value?
Selecting a Managed Detection and Response Company Based on Actionable, Relevant and Timely Detection & Response
Evaluating and selecting a Managed Detection and Response (MDR) solution and provider is more of an ART than a science.
By ART, I mean the Actionable, Relevant and Timely detection and response.
This approach requires a lot of assessment work from the business owner to make the MDR solution work.
Focus on Detection
In terms of performance metrics for MDR, I am going to focus on detection for two reasons.
First, you will never respond to threats you do not detect. If you never find a problem, the response is irrelevant.
Second, incidents should be rare. If you find yourself having reliable statistics about response, you probably have more pressing issues than how you respond.
For these reasons, I’ll reserve response for the “perspiration” section of this post.
The ART of Detection
Let’s take a look at our three evaluation metrics.
Actionable
When you receive an alert from your MDR provider, the required action should be clear.
If you ever receive an alert that makes you ask yourself: “What does this mean?” or “What am I supposed to do with that information?”, you’re likely dealing with a provider that fails to offer actionable advice.
There are two main causes behind non-actionable alerts: jargon and lack of service.
The Use of Jargon
The use of jargon is an indicator of a communication problem. The provider’s alert language may be designed for a user with a lot of cybersecurity literacy. This is especially true when the vendor targets high–end customers with in-house security experts.
Jargon can be addressed by asking follow–up questions. This may impact timeliness of response, and in some cases, incur a professional services cost.
Lack of Service
Lack of service is a serious matter. It is entirely possible that the provider is forwarding raw alarms generated by their security tools.
While this may be reasonable for AV alerts (these tend to be non-actionable. The AV tool automatically blocks or quarantines the malicious item), a number of EDR products generate alerts based on an activity’s potential to cause harm.
These alerts are intended to prompt threat hunting. They are not necessarily indicators of malicious actions. If you are not a threat hunting expert, these alerts seem obtuse and do not prompt specific actions.
Receiving raw alerts is a sign that the provider is not investigating the alerts. They are simply passing them along to you. If you find yourself in this situation, it is often better to change providers or invest in the detection software.
Relevant
When you receive an alert from your Managed Detection and Response Company, the alert should be related to a security event within your organization.
If your conversion rate between provider alerts and security incidents is low, you might have an alert irrelevance problem. There are two main cause behind irrelevant alerts.
The first cause of irrelevant alerts is false positives. These alerts indicate an activity is malicious when it is actually benign. A high volume of false positives may indicate that your provider is not thoroughly investigating the alarms generated by security sensors. Failure to investigate alarms suggests the provider is offloading costs to your account (charging you for alerts that don’t impact your operations). This is another indicator you may need a new service provider.
The second cause of irrelevant alerts is tracking attack attempts. This is a common deception used by vendors to pad their detection statistics. Assessing attack attempts may demonstrate true positive detections, but since the attacks were denied – the alerts are ultimately worthless.
Let’s say a perspective attacker scans the external firewall with a network reconnaissance tool. There is little to no risk to the organization if there is no known vulnerability. So now you have an alert that lets you know someone is scanning your firewall. Can you actually do anything to stop the scan? Does the knowledge that someone is scanning your defenses influence your behavior in any way?
If the answer is no, the alert is irrelevant. If the bulk of your alerts is like this, you should request that your provider trim them out. If the provider is reluctant, this is a sign that they are relying on irrelevant stats to pad their SLA. This is another indicator you may need a new service provider.
Timely
When you receive an alert from your MDR provider, it must be timely.
A fire alarm that indicates the building is on fire only after it has burned to the ground is not particularly useful.
That being said, each company must set reasonable expectations for what type of attack requires the timeliest alerts. When providers have knowledge of malicious players or attacks, they can use tools (like an AV) to quickly detect and effectively block an attack.
However, unknown activity requires manual investigation. These investigations will be triggered by alerts. The provider must also alert you so that they make take action and to ensure your system is not sitting unprotected for weeks. You want to receive relevant alerts so that you can effectively investigate potential threats.
With unknown threats, those that are investigated by your MDR provider, the performance metric measures the time between when you receive the alert in relation to the time the incident actually occurred.
Most providers use a trick to make their SLAs appear timely, while making sure to minimize their liability. If you read the fine print for detection SLAs, the provider usually starts the timer when they see the alert – not when the alert actually occurred. An alert flagged low priority may sit in a queue for three days before an analyst reviews it. Then they will send an alert to notify you, the client, within 30 minutes of realizing the alert was important. In doing do, the vendor meets their 30-minute notification SLA.
Perspiration vs. Inspiration: Checking the Effort Required
The last important metric is evaluating the time required to make a vendor relationship work for your organization. In conducting this assessment, consider three factors: agent/collector management, incident validation, and incident response.
Agent/Collector Management
In order for detection to occur, telemetry must flow between your devices and the MDR provider. Generally, the provider collects data by deploying agents or collectors in the system. These agents and collectors must be continually maintained to ensure telemetry is collected for review. Maintenance is often the most time–intensive task and one that must be performed continuously.
As such, you need a clear understanding of what the vendor expects you to do, and what the vendor will do. Are they responsible for maintenance, or are you? Understanding this division of labor provides a clear “total cost of ownership” that plays a key role in comparing prices for each service/vendor.
Incident Validation
Incident validation is linked to detection metrics. Investigation and validating incidents take a lot of time. Understanding how a vendor works can help to understand the cost of the service and the labor required.
If the vendor sends raw alarms, you may be expected to investigate the alerts. Or, perhaps, the vendor conducts investigations to filter out false positives – only relaying raw alerts requiring your action.
The “rawer” the alerts are, the more time you should budget to validate and investigate the alerts.
Incident Response
Incident response is based on pre-established playbooks. When an incident occurs, you need to know what playbooks are available and who is responsible for what. Some portion of the playbook will be executed by the provider, while other parts will be your team’s responsibility.
If you have a malware incident, do you expect your provider to run an AV scan? Who is responsible for deleting the files or reimaging the machines?
Evaluating the value of incident response efforts is perhaps the most difficult step. Unlike routine management of the collectors and incident validation, incident response is ad–hoc based on an incident. Depending on the scope of the incident, it might fall outside the scope of your provider’s contract.
In the case a machine needs to be reimaged or a domain needs to be rebuilt, your provider might leave the work for your IT team. If a malicious file requires deletion, the provider will do the work remotely.
To calculate the true cost, you must understand the expected value of having an incident occur, multiply it with the odds your provider will perform work, and then factor in the work that is provided in the playbook.
That is why this particular metric is the one with the least value.
Summary: Steps to MDR Vendor Selection
When evaluating a Managed Detection and Response Company, we suggested the following framework:
- Start by calculating your labor costs, including your team’s efforts to manage the telemetry, validate alerts and respond to incidents.
- Establish the cost of the product and service offered by the vendor.
- Understand how your company will receive valuable detection in the form of actionable, relevant and timely alerts.
- Compare vendors and decide what your capacity to respond to threats is.
If you are in the evaluation process and are still unsure where to go, please contact us. We will be more than happy to offer a proof of value so you can see for yourself.
