Scaling security alert automation with agents and ChatOps
Sakil (Lead Security Architect)
This post is a step-by-step breakdown of how Tracecat uses Tracecat to resolve security alerts in real time without slowing the team down.
By the end of this technical breakdown, you'll be able to build your own agentic system that autonomously:
- Turns any detection event into a human-readable Slack notification
- Identifies the owner of the affected identity, host or infrastructure
- Tags the owner and requests verification from them
- Closes or escalates the alert based on the owner's response
It all runs on Tracecat with one agent, six skills and a single 12-action agentic workflow.
Automations-as-code
Everything in this post is open source: https://github.com/TracecatHQ/tracecat
What you need
- Tracecat. A workspace with the Tracecat MCP server connected to your coding agent.
- Model provider. An API key for the provider Socky runs on.
- Agent skills. One preset, Socky, with six skills:
guardduty-case-lifecycle,business-context,hypothesis-driven-triage,aws-cloud-incident-response-core,case-outputandslack-case-threads. Write a few lines on what is normal for your organization intobusiness-context.
- AWS GuardDuty. Enabled in every region you want triaged. Socky lists and reads findings through a read-only IAM role.
- SIEM, optional. Read-only access to the logs Socky should search, such as CloudTrail and VPC flow logs. Without one, Socky investigates with GuardDuty and AWS alone.
- Code, tickets and docs, optional. Read-only connections to tools such as GitHub, Jira or Confluence. See "Giving Socky the places to look".
- Threat enrichment, optional. VirusTotal, AbuseIPDB and urlscan lookups for IPs, domains and URLs.
- Slack. Create an app at api.slack.com/apps with "From a manifest" and paste the manifest below. Install it, invite it to the triage channel and copy the bot token. You add the request URLs for mentions and the Yes/No buttons after you deploy.
{
"display_information": {
"name": "Socky",
"description": "Security alert triage bot"
},
"features": {
"bot_user": {
"display_name": "socky"
},
"rich_previews": {
"is_active": true,
"entity_types": [
"slack#/entities/incident"
]
}
},
"oauth_config": {
"scopes": {
"bot": [
"app_mentions:read",
"channels:history",
"chat:write",
"links:write",
"reactions:write",
"users:read",
"users:read.email"
]
}
}
}
The manifest covers a public triage channel. For a private channel, add groups:history.
| Integration | Secret | Keys |
|---|---|---|
| AWS GuardDuty | aws | AWS_ROLE_ARN, AWS_REGION |
| SIEM, optional | MCP integration | Your SIEM's MCP server, with read-only access |
| Code, tickets and docs, optional | MCP integration | One read-only MCP integration for each tool |
| VirusTotal, optional | virustotal | VIRUSTOTAL_API_KEY |
| AbuseIPDB, optional | abuseipdb | ABUSEIPDB_API_KEY |
| urlscan, optional | urlscan | URLSCAN_API_KEY |
| Slack | slack | SLACK_BOT_TOKEN |
Tracecat assumes the AWS role with a workspace External ID. No long-lived access keys are stored.
slack_work_objects: the Slack card for each case. The workflow reads it when someone opens the card's details panel.
| Column | Type | Notes |
|---|---|---|
external_ref_id | TEXT | Unique |
case_id | TEXT | |
channel | TEXT | |
message_ts | TEXT | |
entity | JSONB |
slack_events: the ID of each Slack event already handled. A repeat delivery is dropped.
| Column | Type | Notes |
|---|---|---|
event_id | TEXT | Unique |
detection_events: alerts an intake workflow stores by alert_id, for backfills and replays.
| Column | Type | Notes |
|---|---|---|
alert_id | TEXT | Unique |
payload | JSONB |
Deploy it
Click your coding agent to get a prompt. The agent connects to Tracecat over MCP, builds in dependency order, checks each step before moving on and keeps the schedule off and Slack quiet until you have seen a test case.
Click to deploy through prompts
Add the Slack request URLs
The deployed workflow has a webhook trigger. Copy its URL from Tracecat, open your app's App Manifest editor in Slack and add these two blocks at the top level of the manifest, next to oauth_config:
"settings": {
"event_subscriptions": {
"request_url": "https://<your-tracecat-host>/api/webhooks/<workflow-id>/<webhook-secret>?echo=true",
"bot_events": [
"app_mention",
"entity_details_requested"
]
},
"interactivity": {
"is_enabled": true,
"request_url": "https://<your-tracecat-host>/api/webhooks/<workflow-id>/<webhook-secret>"
}
}
Save, then verify the event subscriptions URL when the editor asks. Slack needs to be able to reach the URL. The change adds no scopes and needs no reinstall.
The event subscriptions URL must end in ?echo=true. Slack verifies the URL by sending a challenge and expects it back. With echo=true, the Tracecat webhook returns the request body containing the challenge. Without it, Slack reports the URL as unverified. Leaving it out is the most common setup mistake. The interactivity URL does not need it.
The Slack card is a Work Object. rich_previews.entity_types in the manifest and the entity_details_requested event above enable it. The event fires when someone opens the card's details panel. The workflow answers it with entity.presentDetails. Neither needs a scope. Slack's implementation guide covers the setup.
Distributed alerting
One of the fastest ways to reach a verdict about an alert is to ask the person involved whether it was them. Slack popularized this model of alerting in Distributed Security Alerting. Dropbox followed with Securitybot. Brex described a similar approach when scaling alert management with automation.
That is easy when the alert is about a user: a suspicious sign-in belongs to someone. It gets much harder when the detection relates to some host activity or infrastructure change. An unusual command run by a scheduled ECS job, a suspicious DNS lookup from a production EKS pod or an overly permissive Terraform apply from GitHub Actions has no clear owner to ask for a review.
The following section covers the technical challenges behind distributed alerting. If you'd like to jump straight into building, skip to Meet Socky the triage agent.
How distributed alerting breaks at scale
Before Tracecat, I was a security engineer and incident responder at a large 2,000-person technology company. We had a deterministic SOAR workflow, similar to the one described in the previous section, that redirected our SIEM alerts into Slack.
This was before AI agents existed. Every detection event that entered the SOAR required someone to:
- Review the alert payload
- Identify the field that names a human owner to review the alert
- If that field does not exist, identify repeatable drilldown queries that accurately pivot to an owner
User-centric alerts are often easy to resolve. Take this Okta detection event for example. It matches user.session.start when securityContext.isProxy is true, with the owner already in actor.alternateId.
{ "eventType": "user.session.start", "displayMessage": "User login to Okta", "published": "2026-03-14T16:22:08.412Z", "severity": "INFO", "actor": { "id": "00u1ab2cd3ef4gh5ij6k", "type": "User", "alternateId": "jordan.lee@example.com", "displayName": "Jordan Lee" }, "client": { "ipAddress": "198.51.100.23", "geographicalContext": { "city": "Amsterdam", "country": "Netherlands" } }, "outcome": { "result": "SUCCESS" }, "securityContext": { "asOrg": "M247 Europe SRL", "isp": "M247 Ltd", "isProxy": true }}Owner: jordan.lee@example.com, one field lookup away. Send them the Slack message and ask them to confirm the login.
An EKS audit event that matches the Potential Remote Command Execution In Pod Container detection is harder. The caller is the pod service account system:serviceaccount:prod:checkout. No human identity exists in this alert payload.
{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Request", "stage": "ResponseStarted", "verb": "create", "requestURI": "/api/v1/namespaces/prod/pods/checkout-7d9f8b6c4-xk2pq/exec?command=ps&command=aux&container=checkout&stderr=true&stdout=true", "user": { "username": "system:serviceaccount:prod:checkout", "groups": [ "system:serviceaccounts", "system:serviceaccounts:prod", "system:authenticated" ] }, "sourceIPs": ["10.0.12.34"], "userAgent": "kubectl/v1.29.0 (linux/amd64)", "objectRef": { "resource": "pods", "namespace": "prod", "name": "checkout-7d9f8b6c4-xk2pq", "apiVersion": "v1", "subresource": "exec" }, "responseStatus": { "code": 101 }}Owner: unknown. system:serviceaccount:prod:checkout is a workload, not a person. Finding a human means digging through pod tags, CI runs, CloudTrail or an inventory sheet.
At this point, the SOAR engineer would have to collaborate with the detection engineering team to experiment with different drilldown queries to pivot to an owner. Before AI, it could take hours to days to add support for a new detection in the SOAR alerting playbook.
From what I've seen, security teams rarely scaled distributed alerting beyond user-centric detections. Host and infrastructure alerts rarely have a human identity in the payload. The owner lives elsewhere: in tags, in past CI/CD runs, in SSO sign-ins, in inventory spreadsheets (if you happen to have one that's up to date), or inferred from searching through audit logs.
In our user interviews, most security teams that have implemented distributed alerting stop at user-centric alerts after about 30 hours of build time.
The math
Take one detection source as an example: GuardDuty, AWS's native threat detection service. It is one of the first detection sources a cloud-native company enables for SOC 2. At the time of writing, there are 204 active GuardDuty finding types covering user (IAM), host (EC2, Lambda, runtime) and infrastructure (EKS, S3, RDS) anomalies.
How long would it take to deterministically map every one of them to an owner, pre-AI? Assume a SOAR engineer who already knows the environment. User-centric findings carry a principal in the payload and take an hour each. Host and infrastructure findings need pivots into tags, CloudTrail, CI runs or inventory. Those take three to four hours each.
| Finding type group | Centric | Finding types | Hours per type | Total hours |
|---|---|---|---|---|
| IAM | User | 28 | 1 | 28 |
| AI Protection (IAM principal) | User | 3 | 1 | 3 |
| EC2 (VPC flow logs, DNS logs) | Host | 40 | 3 | 120 |
| Lambda | Host | 7 | 3 | 21 |
| Malware Protection for EC2 | Host | 8 | 3 | 24 |
| Runtime Monitoring (instance, EKS, ECS, or container) | Host | 46 | 4 | 184 |
| S3 | Infra | 20 | 3 | 60 |
| Malware Protection for S3 | Infra | 1 | 3 | 3 |
| Malware Protection for Backup | Infra | 4 | 3 | 12 |
| RDS | Infra | 9 | 3 | 27 |
| EKS Protection | Infra | 33 | 4 | 132 |
| Attack sequences (multiple resources) | Mixed | 5 | 4 | 20 |
| Total | 204 | 634 |
634 hours is 21 engineer-weeks at 30 focused hours a week. That is two engineers for one quarter to cover one detection source.
That's just the build. GuardDuty's document history lists 230 updates since January 2018. Of those, 54 add, retire or change a finding type. Call it 30 minutes to triage an update and an hour to rework a finding-type change:
176 x 0.5 h + 54 x 1 h = 142 h over 35 quarters ≈ 4 h per quarter
That is the average since 2018. At the current pace of 12 updates per quarter, it is closer to 7 hours.
GuardDuty is one of roughly ten detection sources a mid-size security team runs, alongside Okta, GitHub, CrowdStrike, Google Workspace and whatever the detection engineers write themselves. Multiply by ten: 6,340 hours to build, or three engineers for 16 months, then about 6 hours a week to maintain. No in-house security team can afford that. Most distributed alerting stays stuck on user-centric alerts, the 15% that already name a human identity.
Meet Socky the triage agent
Socky is the agent that owns a finding end to end: it investigates, writes the case, publishes it to Slack, answers questions in the thread and records the owner's answer. Because the same agent that did the investigation also writes the case and answers questions about it, nothing gets lost or reworded when the work moves from one step to the next.
The preset says which skill to load for which kind of input. It also lists the tools Socky can use (read-only AWS calls, SIEM, Tracecat cases, Slack and the enrichment services). Because the model is a setting on the preset, nothing here is tied to one AI provider.
The skills carry the rest. Socky loads each one only when the job in front of it needs it:
| Skill | What it gives Socky |
|---|---|
guardduty-case-lifecycle | What to do with a new finding: match it to a case or open one, group related findings and decide whether to investigate |
business-context | Background on the environment for judging findings against what is normal for the organization rather than in isolation |
hypothesis-driven-triage | The investigative method an experienced responder would use to work through a finding |
aws-cloud-incident-response-core | The AWS-specific checks, the evidence and identity rules, plus the structured record every investigation ends in |
case-output | How a case, a Slack card, a brief and a thread reply are written |
slack-case-threads | How to handle anything that comes back from Slack: questions in the thread and the owner's answer |
Where to look for owners (the skills)
Attribution is the hardest part of triage, especially for alerts that have no person in them. Take the EKS exec event from earlier in this post. This section works it by hand first, then shows how we turn that work into skills that Socky uses for every other non-user alert too.
Working one by hand
Because the caller is the service account system:serviceaccount:prod:checkout, there is nobody to ask yet. One way to start is to work outward from the workload to the people around it:
- Name the workload. The pod
checkout-7d9f8b6c4-xk2pqbelongs to thecheckoutdeployment in theprodnamespace. - Find its owner. Check the labels and annotations on the deployment and namespace, then the service catalog. If they are empty, look at the repo that deploys it: who is in
CODEOWNERSand who merged the last change. - Look for planned work. A change ticket, a deploy in CI around the same time or an open incident where someone is legitimately debugging production.
- Find the person. The source address
10.0.12.34is internal. Match it to a VPN or bastion session, then match that session to an SSO sign-in. - Compare with normal. Has anyone exec'd into this service before? Does a runbook say this is how it gets debugged?
If a ticket explains it and the steps land on one person, the finding is benign and a quick confirmation closes it. If nothing lines up, that is also an answer: unexplained activity on a production workload with no change behind it.
Every one of those steps means finding information that lives somewhere else in the company: the cluster, the code, the ticket queue, the sign-in logs. For an alert with no person in the payload, that adds up to a lot of research before anyone can ask the right question.
From one alert to a skill
This is the work we hand to Socky through skills. Strip out the Kubernetes details and the same questions come up for an S3 policy change, an unusual DNS lookup from a host or a Terraform apply from GitHub Actions. Only the places to look change.
| Question | Where the answer lives | In the EKS example |
|---|---|---|
| What is this identity or resource? | Cluster, cloud IAM, inventory | Service account checkout in prod, the identity of the checkout deployment |
| Who owns it? | Service catalog, CMDB, team labels, code owners | Team label on the deployment, CODEOWNERS in the deploying repo |
| Was it planned? | Tickets, CI/CD runs, incidents | A change ticket, a deploy or an on-call incident |
| Which person acted? | SSO, VPN, bastion logs | The session that held 10.0.12.34 at the time |
| What is normal here? | Past cases, docs, runbooks | Earlier exec sessions into checkout |
We write that method down once in skills, instead of writing a drilldown for each of the 204 finding types. hypothesis-driven-triage sets out the order of work: form the possible explanations, then look for the evidence that separates them. aws-cloud-incident-response-core holds the AWS-specific checks and the identity rules. business-context holds what is normal for your organization. A new finding type reuses the same method instead of costing another few hours of build time.
Giving Socky the places to look
Most GuardDuty findings flag unusual activity that turns out to be someone doing their job. The difference between unusual and unexpected is context: was this change planned, who owns this service, is this how this system normally behaves? Not all of that belongs in a skill. Socky can also read the places where it already lives, through read-only tools connected to Tracecat. Every organization uses different tools, but these types add the most context:
- Code and CI. Who owns a service, what the infrastructure code says should exist and what was deployed around the time of the finding (for example, GitHub, GitLab or Sourcegraph).
- Tickets. Whether a change ticket covers the activity and marks it as planned work (for example, Jira, ServiceNow or Linear).
- Docs and knowledge search. What runbooks and team pages say about a system and who to ask about it (for example, Glean or Confluence).
- On-call and service catalog. Who is on call and who owns what (for example, PagerDuty or incident.io).
With that context, a finding a ticket explains can be closed as Benign and owners get pinged less often. Activity that matches no planned change or documented behavior is a stronger reason to escalate. Access stays read-only and scoped to what the investigation needs. Anything the agent reads from these sources is treated as data rather than instructions.
Keeping context fresh
Context eventually goes stale as teams reorganize, services change owners and what counts as normal drifts. We keep this manageable by giving environment facts their own skill. business-context holds facts only: what the company is, how its AWS estate is laid out, what normal activity looks like, which identities and patterns are expected and which telemetry exists. The rules for reaching a verdict live in hypothesis-driven-triage. Updating what Socky knows about your environment never changes how it decides.
Facts that change often, such as owners, tickets and recent deployments, do not need to be copied into that skill at all. A skill can declare its own tools in its SKILL.md frontmatter under metadata.tools, including MCP integrations connected to Tracecat. Socky then reads those facts from tools like GitHub or Jira at investigation time, as fresh as the source. The skill keeps only the slower-moving facts, such as what normal looks like for a given service.
When those slower-moving facts need to change, such as a new service or a retired account pattern, the team makes the change the same way as any other fix. Ask your coding agent, connected to Tracecat MCP, to edit the skill's draft, review the change, then publish it. Publishing creates a new immutable version. The preset uses it on its next run with no change to the preset itself.
From finding to verdict
Follow one anonymized GuardDuty finding through the six steps of triage, from the scheduled poll to the owner's answer. Steps 3 to 6 show what Socky produces at each point. Answer as the owner in step 5 to see how the thread plays out in step 6.
Every 15 minutes the workflow asks GuardDuty in every enabled region for findings updated in the last 20 minutes. The window is longer than the schedule so that a finding at the edge of one run is caught by the next.
Each finding goes to Socky on its own. To test or re-triage, run the workflow by hand with a single finding ID.
The case Socky writes
For each finding, the workflow hands off to Socky to investigate and write a Tracecat case. The case holds the verdict, what was flagged, whether it is expected, the evidence behind it and anything Socky could not check.
The alert lands in Slack
Socky posts a card to the triage channel with the finding, the verdict so far and the evidence behind it. The full brief sits in the thread, with the next steps for the owner and a Yes/No prompt.
The owner asks questions
The owner does not have to answer cold. They can reply in the thread and ask Socky for more context. It answers from the evidence it already collected instead of starting again.
The owner gives the verdict
One click on Yes or No records the owner's answer. Socky updates the case and posts the outcome back in the thread. If the owner confirms the activity, Socky also closes the case.
The case updates
Whoever opens the case later sees the owner's confirmation alongside the original evidence.
Staying in control
Socky can only do what its preset allows and you can see what it did. The limits are set on the preset, not left to the prompt.
| Control | What it does | Socky |
|---|---|---|
| Allowed tools | An allowlist of Tracecat actions. Skills can declare tools too. Review the preset's tools plus those of its skills | VirusTotal, AbuseIPDB and urlscan lookups, plus case search, list and get |
| MCP integrations | Which MCP servers the agent can call. A namespace restriction does not filter MCP tools. Attach only the ones you want | SIEM and GitHub |
| Internet access | Lets the agent reach the web from its sandbox when a tool needs it | Off |
| Approval rules | Requires a person to approve a tool call before it runs | Required for URL lookups |
| AWS access | The role the agent's AWS calls run as | A read-only role that reads our account but cannot make any change |
- See what it did. Agent runs can export OpenTelemetry metrics, logs and traces to your own collector, including token usage, cost and tool calls (setup guide). Telemetry is off until an organization admin turns it on. Because prompt and tool content is opt-in, you decide what leaves your deployment. Each case also carries its own record: the verdict, the evidence behind it and anything Socky could not check.
- When something fails. If a workflow action fails, the run shows it. If a tool call inside Socky's investigation fails, Socky notes it in the case as something it could not check. The gap stays visible to whoever reads the case.
- When Socky is wrong. The team owns the improvement loop. Someone works out why and fixes it in the skill or the preset, using the same edit, review and publish steps described under "Keeping context fresh" above.
What next?
What we learned today
- Attribution is a method, not a drilldown per finding type. Writing the questions down once in skills lets one agent handle user, host and infrastructure findings alike.
- Context turns "unusual" into "expected." Read-only access to the places where code, tickets and docs already live makes that call possible.
- The owner decides. Socky investigates, attributes and writes up the case. The owner's answer sets the verdict.
- The pattern carries over. Nothing about Socky is specific to GuardDuty except its skills. The next alert source, whether that is endpoint, identity or SaaS, is a new skill for the same agent rather than a new set of workflows.
If you want to try this against your own GuardDuty findings, the workflow, preset and skills are on GitHub and in the Tracecat docs.
Coming next
- Metrics
- Deduplication
- How to model entities
- Containment
Stay tuned.