Splunk HEC — token generation and index setup

The engine posts batched events to a Splunk HTTP Event Collector (HEC) endpoint using a long-lived token. The token is the only auth model HEC supports, so Enforza stores it (encrypted, in AWS Secrets Manager) and the engine fetches it on demand. The token never appears in the console UI, never goes back to the browser.

1. Enable HEC in Splunk

1. Settings → Data Inputs → HTTP Event Collector.
2. Global Settings → “All Tokens: Enabled”.
3. Make sure the HEC port (default 8088) is reachable from the engine VM. If your engine runs in a different VPC / cloud, you’ll need network connectivity (peering, public endpoint with allow-list, or VPN).

2. Create the token

1. “New Token” — name it something obvious like enforza-engine.
2. Source type — pick _json (or your standard JSON sourcetype). The sink editor lets you override per-sink, but matching here avoids confusion.
3. Index — pick the index your firewall logs should land in.
4. Disable “indexer acknowledgement” unless you have a specific need. The engine retries on failure; ACK adds latency.
5. Copy the token immediately — Splunk shows it once.

3. Verify with curl

Before pasting the token into the sink editor, test it from any machine that can reach the HEC endpoint:

curl -k https://splunk.acme.example:8088/services/collector/event \
  -H "Authorization: Splunk <PASTE-TOKEN-HERE>" \
  -d '{"event":"enforza connectivity test","sourcetype":"enforza:traffic"}'

Successful response: {"text":"Success","code":0}. Confirm the event appears in Splunk under your chosen index.

Heads-up

Don’t use -k (insecure TLS) in production — it’s only there for self-signed cert labs. The sink has an insecure_tls toggle for the same reason; keep it off in prod.

4. Configure the sink

On the Log Export page → “New export” → Splunk HEC:

• HEC endpoint — the full path including /services/collector/event.
• Sourcetype — defaults to enforza:traffic. Set to whatever your Splunk searches expect.
• Index — optional override. If the token has a default index this can be blank.
• HEC token — paste here. Stored in AWS Secrets Manager; never returned in a future API call.
• Insecure TLS — leave off in production.

Rotating the token

Open the sink, paste the new token in the HEC token field, click “Rotate”. The old token is overwritten in Secrets Manager and engines pick up the new value on next refresh (typically < 1 minute). Revoke the old token in Splunk only after the rotation has propagated.

Troubleshooting

• “Token disabled” — Splunk admin disabled the token; check the HEC settings page.
• 403 / “Invalid token” — token typo or revoked. Re-issue and rotate via the sink editor.
• Connection timeout — engine can’t reach the HEC URL. Verify network path + firewall rules between the engine VM and Splunk’s HEC port.
• Events arrive but with wrong sourcetype — the token’s default sourcetype overrides the per-event one unless “Use the sourcetype assigned to the event” is enabled in the token settings.