Your SOC Uses AI. Who Decides What It Can Do?

0:00–1:33 | Welcome and introductions

TYLER MOFFITT: Welcome to the very first episode of the TENEX Signal Podcast, where we explore the threats, ideas, and decisions shaping cybersecurity with practical insights you can take back to your team. I'm your host, Tyler Moffitt, and I've spent over 15 years in cybersecurity researching malware and how attackers operate.

And today we're going to talk about AI and the SOC, what it should be allowed to do, and when a person needs to step in, and who ultimately makes that call.

So I'm joined by Sam Moss, founding member and VP of Global Forward Deployed Engineering. But before we get into it, Sam, why don't you give us three things that listeners should know about you.

SAM MOSS: So I'll stick with the first theme here. I was actually really fortunate to be part of the first class of cyber warfare operators in the US Air Force.

I was also one of the first to get an ATO and authority to operate for the US Air Force bomber fleets. I was also fortunate to be the first employee here at TENEX

TYLER MOFFITT: Good stuff. So continuing with the theme of three things, you're VP of Global Forward deployed engineering. Go ahead and tell us three things about that and that team.

SAM MOSS: Yeah, so the three things that forward deployed engineers deliver is making our security platforms rapidly operational. Providing expert security advisory to our customers and then leading the agentic security transformation.

1:33–5:37 | 1. What are buyers getting wrong?

TYLER MOFFITT: Obviously, when you hear about security vendors, they talk about how much of their SOC is automated.

What do you wish buyers would ask next?

SAM MOSS: Yeah, that's a, that's a great question. If, if I was to put myself in that buyer seat or the buyer's shoes, I would want to know a few things When I hear about automation and. some of those initial questions are like, did the right task get automated?

And was there a, was the right approach taken? When we talked about automating that, was it simple script? Was it a workflow? Is it. Machine learning or are we going all the way to a full blown agent solution? Mm-hmm. So I think it's making sure that we understand what are the right tasks to automate, when's the right time to automate them, and then what is the right way?

So I, I think we see a lot of, of stories around automation. But I would, I would challenge customers and if I was a customer, I would challenge us to go a little bit deeper into that, to make sure that we understand the why, the how, the, when. Those types of things. Another thing that I think of when I hear about automation is did the, did we go and approach it from automating like a, a documented well established repeatable process, or are we even shifting left of that and are we trying to, to get to first principles when we're automating some of these things?

And are we trying to automate some of the, the thinking, the dispositioning versus just like complete task one, complete task two. So I think that's another thing that, that I would, I would be challenging if I was in the customer seat and then I would want to know , you know, why was it important for us to do that?

Like, did we get time back? Did we save money? Did I free up resources to allow them to upskill and uplevel themselves? And then I think my final question would be like, how do we QA this process? Right? I think it's, it's easy to automate something, but then how do we monitor that, refine it either reinforce what we put in there, or, you know, tune and tweak over time.

I think those are all things that I would want to explore and I'd want to double click into as, as I hear things about automation,

TYLER MOFFITT: it sounds like. Yeah. Focusing on the outcomes, which is definitely important. You have any strong opinions about this that maybe other people in the cyber, you know, industry might disagree with?

SAM MOSS: I think right now we're seeing a very heavy reliance on. An AI only approach.

TYLER MOFFITT: Hmm.

SAM MOSS: And I, I would challenge that right now. And, and to be clear, I think there is a difference between AI native and, and AI only, or AI always. And it seems like what I see a lot of right now is just let's build an agent and throw it at a problem.

I want to challenge folks in the thinking of an agent isn't always the solution. And there are definitely times and places and situations where it absolutely makes sense and we want to leverage that power. But again, we don't have to jump from, you know, A to Z there, there are steps in between that we can take.

TYLER MOFFITT: Yeah.

SAM MOSS: And the reason I say it's important to challenge that is because like there, there are costs in doing some of these things and there are risks in doing some of these things. So yeah, that brings, this is something that's on the forefront of customers' minds right now is like cost token burn. And so we, you know, we want to be mindful of that.

And then we have this, this unaddressed, but something that is definitely growing a concern is, is these non-human identities a lot of times have a lot of privileged access. And so how do we limit, how do we monitor? So those are, those are some of the things I consider when we think about is AI is an agent the best solution for it, or again, it is machine learning.

Appropriate for this situation? Or is it a simple workflow? So,

5:37–7:44 | 2. Where should the line sit?

TYLER MOFFITT: yeah, no, this, this goes straight into my next question, like where do you draw the line between work and AI agent can handle vs. Like a decision that needs an actual person that needs to make that?

SAM MOSS: I think it's going to vary based on each team, each business, each risk appetite.

We we're definitely seeing, we as a, as an agentic, MDR. Yeah, every one of our customers is different, so we certainly cannot follow the same approach for all of them. E each one of them is a little bit unique, and so I think it, it really, this, it's an easy answer, but it really depends and so I think it's important that every team can really assesses what resources do they have available, what are the biggest pains, what are the biggest problems?

Where are the biggest opportunities? Where do they have like the, the deepest skillset right now? Right. And even thinking forward, how do they want to be able to leverage the, the resources that they have in the future?

TYLER MOFFITT: Yeah. Yeah. And it, it likely changes too based on like, if you're for a hospital or a bank or a software company, I imagine that, that those lines change.

SAM MOSS: Absolutely, like each one of those verticals will have, again, like different risk appetites and different approaches to those risks and different ways that they're going to address compensating controls. And that's one thing that we spend a lot of time on with our customers early on in the process is understanding like, where is that line right?

And we can certainly come with a prescriptive approach. What we believe fits for most customers. Based on, again, the team size, the security controls, like all of these factors. But we spend a lot of time working with our customers to try to understand where are they truly at. We're trying to figure out, you know, where are they at right now?

Where do they want to get, it's important for us to meet the customer where they're at. And then again, to go back to what I was talking about earlier, it's our job in, in FDE to, to lead that transformation for our customers and so to, to bring them on that journey and help give them that guidance.

7:44–12:12 | 3. Walk through one incident

TYLER MOFFITT: Let's put that situation into something listeners can picture.

Let's say it's two in the morning when your SOC flags phishing campaign, right? Several employees have already clicked and that same email is sitting in other inboxes. Walk me through what you'd want AI to handle immediately and then what, like a human needs to make a call.

SAM MOSS: How about I walk through the process and then maybe we can figure out where do we think AI should come in and where do we want the to rely on that human expertise?

Sure. Yeah. So Bad email comes in or well suspicious email comes in. We don't know if it's bad yet,

TYLER MOFFITT: right?

SAM MOSS: So first thing we want to check in an email are the headers, okay? Is the the Demark, those types of things, right? Mm-hmm. Where's the reply to, right? Did it come from. One email address that we trust, but then the reply to goes to a Gmail maybe, where do the links actually go?

Is, is the link that we see in the email actually what's hyperlinked if, if the user was to click it? So just some of those initial checks in the email, the, the headers, the body are going to be important, assuming that this is bad. The next thing I would want to know is who all received this exact version and are there any other variants that others have received?

TYLER MOFFITT: Yeah,

SAM MOSS: so. This is when like the investigation truly starts. Once we establish the scope, the next thing that, that I would want to answer is, did anyone click on it? And then what happened for those users after they clicked? Then I think the next thing after that, and this is really important, is, is like, what is the context?

And, and when I say context, I mean I need to know more about the users. Is this someone in marketing who maybe doesn't have access to a lot of privileged information, right? Or is this the CFO? So I need to know about the users. I need to know about what they do. I need to know about what they can access.

That context is really important. This is where we typically spend a lot of time in a traditional human only process, doing the scoping, doing this investigation, building this context, right? And so assume that we've gone through all these steps, then now it's about taking action and closing the loop, right?

So maybe this is resetting a user's credentials. Maybe this is quarantining a machine,

TYLER MOFFITT: right?

SAM MOSS: Maybe this is reimaging. Again, every customer is going to be a little bit different. But these are generally the steps that, the major steps that I would take in in a case like that. And when I think about where AI comes in and where humans stay in the loop is what is the thing that is, is fairly repeatable and not specific tasks, but like general checks, right?

Like. I want to check the headers. I want to check the artifacts in the body. I want to bring in context about the user. The specifics may change, but the general process is the same. That to me is a great time to automate something like that. That's a great time to introduce an agent that will go and, and accelerate that part of the process so that ideally what a human expert gets is like a pre triaged or a predispositioned case, right?

Like they don't just get an alert. They get a full case. And so really quickly an analyst can come over that information and then either validate, perform additional investigation, or invalidate that process. So like in this instance, in general for me, I would say it makes a lot of sense to introduce AI and a lot of that information gathering, context filling, and I would still want to leave the decisioning to a human expert.

Most situations. Now, in this specific case, we actually see customers approach two things. They, they either stop there and say, Hey, yes, before I take any response, I want a human to look at it and validate or invalidate. And then we also see some customers that say, Hey, you know what, if it is user and marketing that doesn't have a lot of access to privileged information, go ahead and reset the password.

But if it's my CEO or my CFO, oh no off, I want to give involved off. Yeah. And so like we, we actually see customers again that they can start to create different paths. But again, you can see how, based on the customer, based on the vertical, based on, you know, the risk appetite, the user impacted how some of these things could change.

12:12–15:46 | 4. How do you control the agent itself?

TYLER MOFFITT: Do you, I'm curious here, do you think there is things that an, an agent should never have access to by default? I

SAM MOSS: think as a, as a security practitioner, I would say absolutely yes.

TYLER MOFFITT: Yeah,

SAM MOSS: there are agents can do phenomenal things right now. Would I still trust them with the keys to the kingdom? I, I don't think I would do that yet.

I don't think we're there yet. I think there's still a lot of work that can go into managing those non-human identities, the, the things they can access, the times they can access. I think even just the information security itself, just being able to monitor and kind of control that is really important. But I would say in general, I, I wouldn't trust an agent with all access to everything.

I actually think that this is when we should, if we think about an agentic approach, agents that can have clear lines of delineation and do handoffs, either between other agents or between humans, is really important. So I think this, this goes back to like the role-based access principles and what, what do I really want this agent to do?

And then when do I want to say, Hey, stop. It's time to hand off again. This could be to another agent. This could be to a human. But I think it's really important to establish what is that role that we're trying to execute? What is that goal that we're trying to accomplish? Making sure that the agent executing those things only has what it what it needs.

Right. Least, least privilege. The same approach that we follow with people.

TYLER MOFFITT: Yeah. Onboarding,

SAM MOSS: we should follow with agents.

TYLER MOFFITT: Totally. I completely

SAM MOSS: agree. And so I think what we're seeing a lot of right is, is people are really trusting agents almost more than people at this point, right? They're trusting that, hey, just run and change this line of code and like, don't give me the popup, or, Hey, I'm going to give you this, this full admin key to this app.

And just, but only, only grab this data. So I think we're,

TYLER MOFFITT: it's popular.

SAM MOSS: I think we're seeing lot of, of that approach of like, Hey, this is, this is the easy way to do it. But again, like I, I would challenge us to say like, Hey, no, let's still follow the security principles here. Let's still follow role-based access.

Let's still follow least privilege. Let's make sure that what the agents are doing is, is still happening in a secure way.

TYLER MOFFITT: Agreed. Now it's sort of the age old premise of like convenience to be security is still very prevalent here in over permissioning of the, the agents that we have based on leadership a lot of the times and some of these companies is expecting everyone to use it as much as possible and for their prospect of basically numbers.

Everybody will be more productive and the security sort of consequence is sort of an afterthought and I think that people need to consider that. Especially right now as AI has taken over. So we had talked about ai, making sure to have it do the task at hand only what you want it to do and stop when you want it to stop.

Who can stop it and what record should exist of what it did.

SAM MOSS: I think it really is that that security operations team needs to be able to have insight into what the agent is doing, how it's processing through things, what task it is on, what step it is on, and then. Essentially have a kill switch to say, okay, hey, I want you to go ahead and stop executing that task and I'm going to take it over from here.

And then I think it's also important for us to maybe when we allow those agents to fully execute and do you know that task that we have the ability to go back and, and process what did it do? What was it, chain of thoughts, what was it? Chain of actions and. If that's a good opportunity for us to, again, either reinforce, Hey, yes, you, you did exactly what you were supposed to.

15:46–20:46 | 5. What makes this a different way of operating?

SAM MOSS: Or, Hey, you missed a step and this is something manual that we had to execute. Yeah, I would love for you to do that next time. Or maybe, hey, this is something that you, you either did do or started to do that we want to say, Hey, no, we want, we're going to draw a line there. Don't, don't go past that one. So I think it's really important to have that visibility.

have that view into the chain of, again, chain of thoughts, chain of actions, and then be able to, to reinforce and, and kind of retrain. And, and that's one thing that that's unique to our approach in, in our platform, the TENEX platform, is that we essentially reinforce with every single case that we work.

So we're reinforcing through human input. We're reinforcing through this agent at QA process. So we have a lot of pieces that allow us every time we work a case to, to step back and then analyze again what, what went well, and then what could we do better, and then what should we not do the next time.

So part of that is just having that auditability and then having that, that ability for a human to quickly step in and flip that kill switch if it's needed. To say, Hey, you're, you're executing a task and, and I want to kind of stop you right there and I'm going to take this over. So I think those two elements are really important.

Again, kind of getting left of that, just how we approach it, following least privilege, incorporating RBAC, like all of these things are important. They allow us to have that auditability, they allow us to, to put some of those compensating controls in place. So hopefully we don't have to initiate that kill switch often.

So I think it's a combination of all of these things. There's a little bit of a proactive piece, there's a little bit of a, a real time reactive piece as well.

TYLER MOFFITT: Sounds what like most people would call, sort of modernization of our times, but you've actually described this as, as a transformation rather than modernization.

So what actually has to change beyond just adding AI to the tools that the team already uses.

SAM MOSS: When I think of the two I, I do think of them different. They're close and we'll hear them interchanged sometimes, but yeah, modernization and transformation to me are, are different. When I think of modernization, I think of essentially getting from a legacy tech or just maybe another tech to something more modern, or in this case something agentic.

And so, you know, really what that looks like is bolting on AI or trying to somehow integrate it into something that exists. Again, sometimes it may make sense, but I think again, we're, we're all on a, an agent transformation right now, it really is a matter of where are you at on that bell curve?

Are you an early adopter? Are you somewhere in the middle? Are we a late lagger? Again, I think that's why it's really important, and this is, this is how we approach forward deployed engineering is, is really trying to meet the customer where they're at, trying to solve some of those initial problems and then bring them a along that journey, right, because this transformation is happening and Absolutely.

And then to get back to the difference between modernization and transformation, I think modernization is focused on just the technology. It's just focused on a new platform or an agentic platform, but it's, it's solely tech focused. And transformation, I would say, is actually a combination of, of like three really important things, the people process and the technology.

And it's about leveraging all of these things together to get to. Better outcomes to get to those faster and to get to those in a more affordable way. And so I think when we talk about transformation, it's, it's really stepping back and, and assessing how are we even approaching the problem and being willing to adapt the people process, technology pieces.

And so when I think of each of those people, for example, it's how are we leveraging those experts? We, we all have phenomenal resources. Are we using them in, in the best way possible? When I think of technology, how are we able to accelerate through some of the things that take a lot of time and that that cause a lot of like context switching.

How can we use that technology to accelerate through those pieces and be a force multiplier for the experts that we have? And then I think the process really connects those two, right. Again, we can, we can approach AI a few different ways. We can. Either take things, the, the processes, the standard operating procedures we have, and then just try to have an agent do those same things, or we could say, Hey, I want you to really understand these first principles.

And then from there, I want you to step through, you know, this problem and try to solve it.

20:46–23:29 | 6. What should listeners ask next?

TYLER MOFFITT: So if a, if a CISO takes three questions from this conversation to their security team or MDR provider tomorrow. What do you think those questions should be?

SAM MOSS: I think the first one is, what can your AI do without asking a person?

TYLER MOFFITT: Yeah.

SAM MOSS: I think asking, can you see what it actually did? And can you again see the chain of thoughts in the chain of actions? That's critical.

TYLER MOFFITT: Definitely.

SAM MOSS: Again, we, we want to see. Full visibility into everything the human identities are doing. We should have the same approach for the non-human identities.

And then I think a third question would be, who sets the boundaries and when did we last review them? Again, I think following the same security principles we've done for a long time, we need to understand. Who has the authority to draw the line, and who's going to do that qa, qc on the process and make sure that everything is still working as intended.

And even doing a review to see are there opportunities to improve and get better,

TYLER MOFFITT: what would a convincing answer look like?

SAM MOSS: Well,

We go back to the first question of what can the AI do without a person, I think having that list of those actions and then the specifics that are associated with it, can it only do this action if it's cer certain type of user or if it's in a certain type of environment?

Just that's what I would want to see is. that list or that audit trail of here's, here's what it can do by itself without consulting a, an, an expert. The second one around, can we actually see what it did? I think every agent action needs to be captured and searchable in the platform.

So whether it's a SIEM or whether it's just. Cold storage solution. Having the, again, the chain of thoughts, the chain of actions, I think is critical for being able to answer the questions, what did it actually do? So capturing all of those things. And then for that last one of like who sets the boundaries and, and who does the review.

TYLER MOFFITT: Yeah,

SAM MOSS: I think it's having named resources on both sides. Knowing who has that authority to sign off before something go goes live. And then who's going to own that constant review? Is there a schedule for that review? Just understanding that process is going to be super important.

23:29–24:49 | Takeaway and close

TYLER MOFFITT: So as we bring this to a close, what, what's one thing you would want a security leader to check this week?

SAM MOSS: I mean, if I want to be cheeky, I could say. You should go check the TENEX Signal newsletter.

TYLER MOFFITT: Absolutely.

SAM MOSS: But I think in all seriousness, I think it's checking your non-human identities. Like do you have a, do you have those accounted for in your security policies? Are you following the same processes to secure the non-human identities that, that you follow with your human identities?

TYLER MOFFITT: That's good stuff. Thanks for helping us kick this off. Sam, it was really great to have you on.

SAM MOSS: I appreciate it. It's fantastic that we're kicking off a podcast and I'm really excited to, to hear some of the topics and some of the guests that, that you're going to bring on. So Again, appreciate the opportunity.

TYLER MOFFITT: It's good stuff, and thanks everybody to listening to our first TENEX Signal podcast. And for more security news and the stories we didn't get to make sure to check out the TENEX Signal Newsletter from our TENEX SOC team. You'll find the newsletter and related reading in the episode notes. And as always, please follow, like, and subscribe to the show and we'll see you next time.

Sources

• Episode 1 on Buzzsprout, with show notes