This post was originally published on this site.
Security controls can be a bit of a cat and mouse game—you block one attack, new ones spring up. Malicious actors continue to innovate new ways to hack your software, so responses end up being attack-specific and often manual. It’s not just your software, it’s your third-party dependencies, too. So Exaforce built software that can automate some of the responses and attack detection.
I spoke with Ariful Huq, co-founder and head of product, and Marco Rodrigues, co-founder and head of product, at Exaforce last month at AWS re:Invent.
————————————
Q: Tell us a little bit about what what Exaforce does.
Ariful Huq: We are focused on helping organizations of all sizes, starting from high growth startups all the way to mid enterprises, depending on where they are in their SOC journey. If you do not have a SOC, we enable you to build one in days, literally without having to go buy tooling, get detection, engineers, get analysts. If you do have a security operations center when you have analysts, our goal is to amplify the capability of these analysts. Think about a team of two or three analysts—how do you make them a team of ten? That’s essentially what we do.
Q: Where do you find that organizations are the most lacking, either pre-SOC II audits or after?
Marco Rodrigues: In our experience at least, customers tend to come to us once they have the SOC II compliance or ISO that’s clearly an attestation and an evidence-driven security compliance framework. When it comes time to actually start putting together incident response plans or where there’s legal liability that’s being driven through their customer contracts, that’s where they tend to get a bit more serious.
A lot of these companies are at the early stage startups. They barely have one or two security engineers to begin with. Usually where they’re lacking depends on the journey of the company. A lot of them can be where they have no tools at all, and they need some detection framework. They need individuals monitoring and actually writing those detections. You need a routine that actually responds and remediates to it. So we’ve seen a kind of a variance of companies in that space.
Some of the larger companies, they just can’t keep up with the growth of detections as they come in. They need to augment their teams. The reality is that the skill set is not there—they can’t hire these people even if they wanted to. They’re using AI SOC, as an example, to augment and fill in that gap.
Q: When you do construct these sort of detection frameworks for these operations, how much existing infrastructure are you building on? I know a lot of folks have a CloudFlare base to help with that, or HAProxy to route traffic. What are you coming in to? Does anyone just have nothing?
AH: Surprisingly, even in the largest organizations that we work with, sometimes they have nothing, specifically around cloud and SaaS.We found in starting our journey in building this AI SOCplatform is that most of the market thinks about this as an AI analyst problem.
But we think about four primary tasks in the SOC and detection is one of them: detections, triaging, investigations, and response. If you’re a very small organization, typically two to three person security organization, you don’t even have the bandwidth to actually go think about detection engineering or building detections.
What you’re really looking for is getting off the ground, right? So you come with out-of-a-box detections: great! If you have existing detections from, for instance, CloudFlare, we’ll leverage those detections for enrichments and those sorts of things.
Even the larger organizations, like Fortune 2000 companies that we work with, what we find is a lot of them don’t even have detection coverage for SaaS services that you would think they would consider very critical.
Q: Open to the internet.
AH: Exactly. Like GitHub, Snowflake, OpenAI. These are critical services where a lot of critical data resides today. And they don’t have detections on top of it. We help those organizations with detection and coverage for those SaaS services.
If they already have an endpoint technology, email, that they’re getting the right detections. There’s no value that we can add there. Where there is value in creating additional coverage for critical data, we help there.
Q: We wrote something about our own DDOS mitigation. We got hit by bunch of attacks, but it was almost like whack-a-mole. How do you do detections in a reliable and almost permanent way?
AH: It’s a tricky problem to solve. Anomaly detection has a little bit bad rep for being noisy. I’ll give you a little bit of how the approach and the market has evolved, how the industry overall has evolved.
Most anomaly detection has been statistical in nature. It’s based on baselining and those sorts of things. Sometimes these things are bespoke to every organization. What we find with anomaly detection that we’re doing now is we still have statistical modeling because you certainly need to understand what is normal and then you can figure out what is known good from potentially what’s bad, right?
But what’s really interesting now is we’re leveraging our large language models, our AI agents, to actually do the triaging for these detections. We’re helping make anomaly detection much more reliable. We leverage statistical modeling behavior across lots of different types of data and then layer it with what we call a knowledge layer that’s based on the large language models where we take business context from. Every customer has different business context, right? Different types of ways they leverage their technologies.
From there, we try to weed out what should be potentially good behavior in the environment that’s being flagged as potentially as something anomalous, like developers working within your cloud environment. Sometimes they may be doing things that an attacker may be doing, but it’s normal for this person.
That’s how we think about anomaly detection and leveraging this new wave AI agents. In the past, you couldn’t create higher fidelity because you didn’t have enough people to look at these detections. Now we actually have machines looking at them, so we can actually take even the lowest signals, put it all together, let machines do the stitching and bring up the fidelity.
Q: Do you find that AI triaging reliable? Do you have guardrails to make it more reliable?
AH: It’s really how much guesswork you try to avoid. If it’s you and me and somebody asks a question without directional guidance. Likely our responses could be in one direction, but it could deviate quite a bit. Yeah. With LLMs, we try to give them as much directional guidance as possible.
That’s where we leverage a lot of the data—we gather, build semantics around it, build relationships, figure out, get a lot of context. Then we essentially give the LLMs reasoning capabilities. We answer a bunch of questions that are critical to understanding the specific detection, and then we let leverage the LLMs to do reasoning by giving it sufficient context, by actually narrowing the amount of data we give it too.
That’s the other thing that people have to avoid. You give too much data. It’s you reading a hundred page book. The first page versus the last page, what are you most likely to remember?
So we try to reduce the scope of the data. We try to give it as much context as possible, remove the guesswork. We get a lot more predictable results, and that’s the approach that we’ve taken. Goes well beyond just LLMs. We do a lot of statistical modeling.
MR: I think of that as human reasoning at scale, machine scale. A lot of our critical value is in all the upfront data processing work that we do to make sure we present the right data in the context of that protection or alert, even an investigation that comes up relative to the LLM.
Q: So for that data sort of identification, I assume you have to do a bit of fine tuning on your LLMs, if not train your own LLMs. How do you go about that process? Where do you get that data? And how exact do you need to get it?
AH: This is part of the reason why the approach that we’ve taken in solving this problem was a data first approach. A lot of competitors in this space have taken an overlay approach. They rely on detections from third-party sources that they try to do triages on top of.
We’ve taken a fundamentally different approach where we try to ingest the data and build semantics around it, build a bunch of enrichments. From our perspective, it’s a combination of LLMs plus the data engineering work.
As far as fine tuning is concerned, we do some aspects of fine tuning. As an example, when we convert natural language to actual queries in the system, we do some level of fine tuning because making it understand natural language to SQL conversion will help it be very exact. But what we also find is that doing a lot of the upfront data engineering and enrichment work actually reduces the need for doing tuning.
We leverage a lot of the LLMs just through APIs primarily for recent capabilities. We find they’re very good at general intelligence, which is what you want them to be good at. We give them all the domain specific context. So they use a combination of general intelligence and domain-specific context to give you really good results.
MR: There is a level of measurement in terms of measuring the LLM output precision. The team is constantly measuring that as new models come out. There’s a constant reassessment of that in terms of pipeline.
Q: You mentioned the four aspects were investigations, detections, triaging, and response. We haven’t talked about the response. I believe is the response part of this is actually building features to protect like hardening systems, right?
AH: It’s actually much more focused on it responding to a potential threat, right?
The whole response based SOAR (Security Orchestration, Automation, and Response) have existed for quite some time, right? Even with SOAR, you need to build playbooks. These are typically done as step-by-step processes. If this happens, go do this.
We’re starting to see that it’s not as easy as that. Response actions are very dynamic in nature. It could be many numbers of things that happen, and then it may result in a specific response act.
We have automated response actions—simple things like go reset a password, isolate an instance, reset a session token. Those are things that customers are happy to have out of box. But more interestingly, we’re helping customers build automation agents on top of our platform so they can go build their bespoke response actions.
I’ll give you a simple example. I have some actions I need to take based on monitoring a set of IPs for a specific type of behavior. Let’s say I get a password spray attempt from a bunch of IPs. I’m gonna record these IPs. Anytime I see any activity from these IPs that’s successful, I wanna know about it. Successful or unsuccessful, I wanna know about it.
It’s preventative response actions, things that people have to go threat hunting. They can build an automation agent for that. You write it in natural language—you say, “Hey look for all threats where I see password spray attempts. Extract the source IPs. Now give me a daily report of any type of activity from these source IPs specifically if there’s been a successful authentication attempt. That may be an attempt where somebody did a brute force attack and then there was successful,
MR: Once our agents have investigative capabilities, a response to those investigative capabilities is the system itself will actually also reach out to users. An example would be if this detection was fired through some third party system or detection, or we decided to trigger our own detections. The agents themselves will respond to request more information to determine whether something is false, needs to be investigated further, or is deemed a cause positive.
The system itself will ping a user on Slack saying, “Hey, was this really you? Did you attempt to do these things?” That’s taken into account as the investigation and reassessed.
AH: And you’d be surprised how much time SOCs spend on just doing that task that we talked about. It’s very asynchronous. You send somebody a Slack message, but you don’t know when they’ll respond. You may forget about it.
Q: What’s the installation lift of this? I know a lot of systems like this will live in the cloud. How does their hook into the existing system? And is there compute inflow, outflow cost to it too?
AH: We have a fairly flexible approach in deploying. Most customers choose deployment in our cloud. Every customer of ours gets a separate cloud account. It’s a single tenanted deployment because we care a lot about the data. It’s very sensitive data that we’re handling. Each and every customer gets their own data warehouse. We use Snowflake as our backend for storing this data.
As far as getting the data is concerned, it’s all API based. We tap into the major cloud providers through roles that they may give us access to. In the case of GitHub, OpenAI, Snowflake, it’s just three APIs: read access to get logs, event data, and configurations.
I’s a fairly easy lift—a matter of three to four hours. We’ve seen customers start a POV, onboard four to five data sources, and then start to see the value.
Q: To get the signals out of the system, do you require them to instrument first? Are you able to instrument? Are there preferred ones, like Open Telemetry?
AH: If you’re storing historical data, in most SaaS services, they at minimum have 30 days of historical data. Some even have 15. If you have any historical data, we’ll pull it in. We typically try to do baselining statistical modeling on about a 90 day window. If you have any historical data, We’ll automatically ingest it.
Within the first run of this entire data, we start to build the behavioral models automatically. Literally within the first date, we’re looking at 90 days of data if you have it and starting to baseline and figure out what is potentially anomalous.
A lot of sims today, even UEVA, universal entity behavior analysis, we’re doing historical analysis on it. We built the technology in such a way that, if you have historical data, we’re gonna read it and leverage it for better outcomes.
