Skip to content

AI Agent Security Event Management: Sending Logs to SIEMs

Picture of Sharan Shirodkar
Sharan Shirodkar

TL;DR
What it is: AI agent security event management turns what agents do into security events you can collect, correlate, investigate, and prove.
The pattern: For SIEM integration, keep the audit record inside your infrastructure and use security event forwarding to send a minimized event, with an ID that points back to that record, to Splunk or Datadog.
The mechanics: Write the record and an outbox entry atomically in one datastore, deliver asynchronously, and expect duplicates.
The delivery catches: Splunk Cloud Platform does not support HEC indexer acknowledgement outside AWS Kinesis Firehose, and Datadog truncates logs over 1 MB while still returning success.
Where it fits: Prediction Guard, a sovereign AI control plane, is the control point that produces these events inside your boundary.

Security teams run detection in a SIEM, yet AI agents now produce activity that never reaches it: a detected prompt injection, a blocked tool call, a quarantined agent. AI agent monitoring shows that an agent ran, but security logs have to show whether each action was allowed and under whose authority. AI agent security event management captures those events in one schema, forwards them to the SIEM you already operate, and keeps the underlying audit records where you control them. It is how a security team extends operational control of agents into the tools it already trusts. Prediction Guard, a sovereign AI control plane that runs inside your boundary, produces these events for the agent traffic routed through it.

This guide follows one event from the first runtime decision to an analyst retrieving evidence, with a schema, request examples, threat detection, and a go-live checklist. Splunk and Datadog details come from their documentation as read in November 2026, and the code is illustrative. It complements Prediction Guard's SIEM gap explainer, which covers why SIEMs miss AI events. Prediction Guard publishes this guide and builds a sovereign AI control plane, so its statements about its own product are limited to what its v1.2 release ships.

SIEM Integration Architecture: Keep Audit Records Inside Your Infrastructure

What AI Agent Security Event Management Means

Three terms carry this guide.

  • Audit record: the system-of-record account of what an agent did, kept at the fidelity your retention requirements and data sensitivity justify. It may contain prompts, tool arguments, tool results, and outputs, or appropriately redacted representations of them.
  • Security event: a structured, minimized record that something security-relevant happened, such as a blocked tool call or a quarantine. It carries decisions and identifiers, not content, and it is what you forward.
  • Evidence pointer: the event ID, and optionally a hash, that an authenticated retrieval service resolves to the matching audit record.

AI Agent Monitoring Is Not Enough

AI agent monitoring tracks traces, latency, tokens, and errors: whether an agent ran and how well. Security event management records whether an action was allowed, under whose authority, which safeguard fired, and what the team did. OpenTelemetry's agent span conventions define attributes such as gen_ai.agent.id, still marked Development, which make good correlation fields. Send traces to your observability backend and security events to your SIEM, joined on a session or trace ID.

Two Tiers, One Boundary

Design rule. The SIEM receives events, not content, and enforcement never waits on the SIEM.

                                  agents and tools
                                          |
                                          v
  +--------------------------------------------------------------------------------+
  |                               control point                                    |
  |                               inside your boundary                             |
  |                               decide, record, stop                             |
  +--------------------------------------------------------------------------------+
                    |                                           |
                    v                                           v
              audit record                              outbox (pending)
              system of record,                                 |
              inside your boundary                              v
                    ^                                   forwarder: batch,
                    |                                   retry, dead-letter
              retrieval                                         |
              service                          +----------------+----------------+
                                               v                                 v
                                            Splunk                            Datadog

Splunk Enterprise runs where you run it, while Splunk Cloud Platform and Datadog run in the vendor's cloud. Analysts query a SIEM broadly, and prompts and tool outputs can contain regulated data, so only the minimized event travels. An identifier alone retrieves nothing, so an authenticated retrieval service resolves each event_id to its audit record.

What Leaves Your Infrastructure: Security Logs and Audit Records

Keeping audit records inside does not mean nothing leaves. The security event is deliberately forwarded to a hosted SIEM, and you decide, field by field, what may travel.

Field group Leaves? Control
event_id, event_type, decision, severity, detector, session_id Yes Schema allowlist
agent_id, owner, initiating_user, target, effective_authority Yes, after redaction Pseudonymize users where required, and strip query strings and tokens from targets
evidence_hash Yes A hash only, never content
Prompts, tool arguments, tool results, outputs No Stay in the audit record, in full or redacted form
Credentials, keys, tokens, personal data values No Never serialized into an event, and redacted before storage unless an investigation needs them

Field-level allowlisting: the canonical schema rejects unknown fields. Redaction: pseudonymize users where required, and strip query strings and tokens from target. Encryption: TLS to the SIEM and the retrieval service, and encryption at rest for the audit record. Access control and logging: only the retrieval service reads the record, each request is authorized by role and case ID, and each access is logged separately.

For high-risk AI systems, the EU AI Act requires automatic event recording (Article 12) and a floor of at least six months for logs under the provider's (Article 19) or deployer's (Article 26(6)) control, subject to other Union or national law. Set SIEM retention to your investigation window, and audit record retention to your legal floor.

Build the Security Event Forwarding Pipeline

Capture, Store, and Minimize

Emit one security event per decision: a blocked action, a detected threat, an authorization failure, an identity change, a containment action, or a configuration change. A routine allowed call is an audit-record entry only.

Store the audit record append-only, with each record's SHA-256 hash chained to the previous one and the chain head signed periodically by a key held outside the control point. Encrypt it at rest, allow reads only through the retrieval service, key it by event_id, and set retention to your legal floor.

The rule that prevents lost events is to make the audit record and the outbox entry atomic, which requires both writes to share a transactional datastore. Write them in one local transaction in the same database, before the decision is returned. If you keep audit records in object storage and the outbox in a separate database, no database transaction can make the two writes atomic. In that case, write the record first under its event_id, then the outbox row, and run a reconciler that finds audit records with no outbox row and enqueues them, so the audit store, not the outbox, is the source of truth.

Record decisions and outcomes separately. For a blocked action, record the denial before returning it. For an allowed action, record the authorization decision before execution, and append the execution outcome (succeeded, failed, or unknown) when the tool call completes, so the audit record never implies that an action happened when it did not. A security event reports the decision, not the outcome.

A forwarder then reads pending rows in small batches of around 100, sends them, and marks a row delivered only after the destination confirms, handling failures as described under retries below.

The Canonical Event and Schema

Every event is built through an explicit allowlist and validated before it enters the outbox. The schema is closed, so it rejects any field not listed, and it caps string lengths at 128 characters. The forwarder also enforces a serialized event-size limit before sending, such as the 256 KB cap in the Datadog section, which protects against oversized payloads and unexpected schema changes. event_type takes one of six values (blocked_action, detected_threat, authorization_failure, identity_change, containment_action, or configuration_change), decision is blocked, detected, allowed, or contained, severity runs from informational to critical, and evidence_hash is sha256: followed by 64 hex characters. Eight fields are required: schema_version, event_id, event_time, event_type, agent_id, decision, severity, and source_system. The rest are optional.

These are recommended field mappings, not one-to-one schema equivalents. Preserve the original event fields and use destination-specific mappings where the platforms expect different semantics. Map to a CIM field only where the value matches that field's meaning, and keep everything else as a custom field.

Field Proposed Splunk CIM Alerts mapping Proposed Datadog mapping
event_id Keep as event_id; alias to id only if it fits your dedup model Custom attribute
event_time time in the HEC envelope Custom attribute; the log timestamp is ingestion time
severity Map to severity (critical, high, medium, low, informational) Keep as a custom attribute, and map to status through an explicit table
source_system app service, source
initiating_user user usr. attributes
detector signature, signature_id Custom attribute
session_id Custom field Custom attribute; use trace_id only for a real trace ID
All other fields Custom fields Custom attributes

Datadog reserves host, source, status, service, message, and trace_id, and its standard attributes use the usr. and evt. prefixes. See the CIM Alerts model and the Datadog attribute docs. Save this canonical event as event.json, because the Splunk and Datadog examples below build on it:

{
  "schema_version": "1", "event_id": "evt-0192", "event_time": "2026-11-18T14:02:11.123Z",
  "source_system": "agent-control-point", "event_type": "authorization_failure",
  "agent_id": "invoice-agent", "owner": "finance-platform", "initiating_user": "a.rivera",
  "effective_authority": "invoices:read", "target": "payments.refund",
  "detector": "authorization", "decision": "blocked", "severity": "high",
  "session_id": "ses-7731", "evidence_hash": "sha256:39773c48e38df6070ddc82cd79af5dfbb0cdd284ab4a9f49332d2fe8027b878f"
}

Retries, Duplicates, and Recovery

This is the one place the reliability design is explained. The Splunk and Datadog sections refer back to it.

Retryable versus non-retryable. Retry with exponential backoff and jitter on server-busy, rate-limit, timeout, and 5xx responses (Splunk reply code 9, Datadog 408, 429, 500, and 503), and split a batch Datadog rejects as too large (413). Dead-letter anything a retry will not fix, such as a bad payload, invalid key or token, or incorrect index. See the Splunk reply codes.

At-least-once delivery and duplicates. A crash after a send resends the batch, and Splunk can accept part of a batch before returning a 503, as this HEC 503 behavior explains, so duplicates are expected. Neither the Splunk HEC nor the Datadog intake documentation describes deduplication on a field you supply, so count unique event_id values in queries and rules.

Outbox buffering. During an outage, events wait in the durable outbox while enforcement continues. Alert as it fills, shed low-severity events first, and spool critical and high events to disk.

Dead-letter handling. A dead-lettered batch raises an alert and waits for a fix, then returns to pending.

Replay and reconciliation. After an outage longer than the outbox holds, rebuild minimized events from the audit store by event_id. Datadog accepts a log timestamp up to 18 hours in the past, so for older replays use ingestion time as the log time and keep the original in event_time. Reconcile delivered rows against unique event IDs in the SIEM on a schedule.

Configure Splunk HEC

  1. Create the index. Create ai_security. The index must already exist before HEC can write to it.
  2. Enable HEC. On Splunk Enterprise, open Settings, Data inputs, HTTP Event Collector, then Global Settings, and enable all tokens. On Splunk Cloud Platform, HEC is enabled by default.
  3. Create a token. Limit its allowed indexes to ai_security. On Enterprise only, you can tick Enable indexer acknowledgment.
  4. Send an event. Use sourcetype _json, as Splunk's own example does, and wrap event.json in the HEC envelope. The command derives time from event_time, converting a UTC timestamp to epoch seconds with milliseconds, so the value always matches the event being sent. Never hardcode it, and keep event_time in UTC, because this conversion does not handle other offsets.
    jq -n --argjson e "$(cat event.json)" \
      '{time: ($e.event_time
                 | capture("^(?<s>[^.Z]+)(?:\\.(?<f>[0-9]+))?Z$")
                 | ((.s + "Z") | fromdateiso8601) + (("0." + (.f // "0")) | tonumber)),
        host: "agent-control-point-01", source: "agent-security",
        sourcetype: "_json", index: "ai_security", event: $e}' |
    curl -sS "https://splunk.example.internal:8088/services/collector/event" \
      -H "Authorization: Splunk $HEC_TOKEN" \
      -H "Content-Type: application/json" -d @-
    # {"text":"Success","code":0}

A reply of {"text":"Success","code":0} means Splunk accepted the request. Verify ingestion by searching index=ai_security event_id=evt-0192. On Splunk Cloud Platform, use https://http-inputs-<your-stack>.splunkcloud.com on port 443, with no port in the URL.

Delivery confirmation differs by deployment. Splunk documents HEC indexer acknowledgement for Splunk Enterprise only, where each request carries a channel identifier and you poll for acknowledgement status. Splunk Cloud Platform supports it only for AWS Kinesis Firehose, so do not assume the Enterprise workflow exists on Cloud. On Cloud, treat reply code 0 as accepted, reconcile delivered rows against unique event IDs in the index, and alert on silence. On Enterprise, acknowledgement state is held in memory, so treat an unavailable acknowledgement as unknown and resend. See the indexer acknowledgment docs, the HEC setup docs, and the HEC event format.

Configure Datadog Log Intake

  1. Create an API key. In Organization Settings, open API Keys and choose New Key. Use one key per forwarder, and keep it in your secrets manager. See the API key docs.
  2. Pick the site. Use the site your organization is on, such as datadoghq.com or datadoghq.eu. The site is also a data residency decision.
  3. Map severity to status. Datadog's status remapper maps status strings onto the Syslog severity scale and defaults unrecognized values to info, so use these exact words: critical to critical, high to error, medium to warning, low to notice, and informational to info.
  4. Send a log. The command below builds the payload from event.json, applies that mapping, moves the user to usr.id, and sends a gzip-compressed array.
    export DD_SITE="datadoghq.com" DD_API_KEY="<key from your secrets manager>"
    
    jq -c '[ { ddsource: "agent-security", ddtags: "env:prod,schema_version:1",
               hostname: "agent-control-point-01", service: "agent-control-point",
               status: ({critical:"critical", high:"error", medium:"warning",
                         low:"notice", informational:"info"}[.severity]),
               message: "\(.event_type) \(.decision): \(.agent_id) -> \(.target)",
               usr: { id: .initiating_user } } + del(.initiating_user) ]' event.json |
    gzip | curl -sS -w '\nHTTP %{http_code}\n' -X POST \
      "https://http-intake.logs.${DD_SITE}/api/v2/logs" \
      -H "Accept: application/json" -H "Content-Type: application/json" \
      -H "Content-Encoding: gzip" -H "DD-API-KEY: ${DD_API_KEY}" --data-binary @-
    # {}
    # HTTP 202

The Datadog log intake docs set limits of 5 MB per payload, 1 MB per log, and 1,000 entries per array. A log over 1 MB is accepted and truncated while the API returns a 2xx, which is silent data loss, so refuse to send any serialized log above the size limit, such as 256 KB. A 202 is always an empty body and means accepted, not indexed. Handle the other codes as classified above.

Verify in Logs Explorer with source:agent-security @event_id:evt-0192, after creating facets for @agent_id, @decision, and @detector to filter and group on them.

Connect SIEM Alerts to Internal Audit Evidence

An alert in the SIEM is a pointer, and the retrieval service turns it into evidence. It resolves an event_id through the store's primary key, verifies the record against the event's evidence_hash, authenticates the analyst through single sign-on, checks a retrieval role, and requires a case ID on every request. It returns a redacted view by default, with whether the hash verified, and a raw view only to a more privileged role. It writes an access record for every request, kept apart from the audit record. Add a link or automation step in the SIEM that calls the service with the event_id.

A worked example shows the whole path. An invoice agent tries to refund a payment for a user with read-only authority:

  1. The control point blocks the call as an authorization failure and, before returning the denial, writes the audit record and an outbox row for evt-0192 in one transaction.
  2. The forwarder sends the minimized event and marks it delivered after reply code 0 or a 202.
  3. The authority-overreach rule fires after the third denial in 15 minutes, and the analyst retrieves the record with a case ID.
  4. The team quarantines the agent, which emits evt-0193 through the same pipeline. Each containment action is an intervention: a countable, recorded act of control.

Test and Operate the Pipeline: Go-Live Checklist and Threat Detection

Go-Live Checklist

Run these before production and after any change. Each line states its expected outcome.

Threat Detection and Forwarder Monitoring

This guide is about getting events into the SIEM, so it shows one detection as the pattern. Repeated authority overreach maps to ASI03, Identity and Privilege Abuse, in the OWASP agentic top ten. Alert when one agent and user accumulate more than three authorization failures in 15 minutes, counting unique event_id values so duplicates do not inflate the count. In Splunk, that is a scheduled search counting distinct event_id values by agent and user in 15-minute windows. In Datadog Cloud SIEM, following its rule creation docs, create a Threshold rule on source:agent-security @event_type:authorization_failure, set Count to unique @event_id, group by @agent_id and @usr.id, and use the condition a > 3. Unit-test it with a sample log copied from Logs Explorer.

Monitor the forwarder as well: alert when no events arrive for 15 minutes. In Datadog, use a Threshold rule on source:agent-security with the below-threshold condition a < 1, which its documentation describes for silent sources. For a broader set of detections, see Prediction Guard's monitoring and alerting guide.

Where Prediction Guard, a Sovereign AI Control Plane, Fits in This Pipeline

A sovereign AI control plane, in this guide's sense, is the layer that identifies, authorizes, enforces, meters, records, and contains agent actions inside your own boundary. It is the control point in the diagram above.

Prediction Guard's v1.2 release produces the raw material for this schema: safeguards for PII, prompt injection, toxicity, credentials, and grounding, and a Security Events dashboard charting blocked versus detected events per detector. A Tool Content control extends PII, prompt-injection, and secrets checks to tool results. Each agent is a managed identity with an owner, and a run is limited to what the key, the agent, and the initiating user can all reach. Admins can quarantine an agent, with release requiring a reason and both actions recorded for audit, and disabling a user in one step cuts off their sessions, keys, and agents. The Console shows a live audit log on its own System tab. It is self-hosted and enforces on traffic routed through it, so events exist for routed traffic.

Related guides cover AI observability versus SIEM, a prompt injection response plan, and the product overview.

From Security Logs to Operational Control of Agents

A SIEM tells you what happened. Operational control of agents means you can also decide before an action runs, stop an agent while it runs, and prove afterward what was decided and who authorized it. Count the interventions the pipeline records: that number, more than alert volume, shows whether control is improving.

How Prediction Guard Solves This

Prediction Guard gives regulated enterprises operational control of their AI agents: one sovereign AI control plane that identifies, authorizes, enforces, meters, records, and contains every action an agent takes, running inside the customer's own boundary. For the pipeline in this guide, it is the control point that makes the decisions and keeps the records: safeguards and a Tool Content control that decide, managed agent identities with owners, quarantine with recorded release, a Security Events dashboard, and a live audit log, all self-hosted.

It enforces on traffic routed through it, so events exist for routed traffic. The forwarding pipeline, the SIEM configuration, and the retrieval service are yours to build, and the code in this guide is illustrative. Ask us to show a blocked call, a quarantined agent, and the event that reaches your SIEM in the same session.

FAQ: Sending AI Agent Security Logs to SIEMs

1. What is AI agent security event management?

It is the practice of capturing security-relevant agent activity as structured events, forwarding them to a SIEM, detecting threats on them, and keeping the audit records under your control.

2. Does Splunk Cloud Platform support HEC indexer acknowledgement?

Splunk documents it for Splunk Enterprise only, and for Cloud Platform only with AWS Kinesis Firehose. On Cloud, treat reply code 0 as accepted and reconcile delivered events against unique event IDs in the index.

3. Do Splunk and Datadog deduplicate events by my event ID?

Their intake documentation does not describe deduplication on a field you supply, so design for duplicates. Count unique event_id values in queries and rules.

4. Should prompts and tool outputs go to the SIEM?

Generally no. They can contain regulated data, and a SIEM is queried broadly. Forward decisions and identifiers, and let an authorized retrieval service return the content, or its redacted form, through the event ID.