Observability

Log destinations

Set up Splunk HEC, Datadog Logs, OpenTelemetry, webhook, syslog, Azure Monitor, Event Hubs, Cloud Logging, Pub/Sub and CloudWatch Logs destinations with filters, tests, status and cursor control.

Add destinations in Settings → Log Export, with /api/v1/log-destinations, the MCP log_destinations_* tools, usertasks log-destinations … or config as code (credentials are never exported).

Filters (all optional): categories, types (globs such as workflow.* or *.failed), min_severity and workflow_ids. start_from: now (default) or retention_start to backfill everything still retained.

usertasks log-destinations create -f splunk.json
usertasks log-destinations test <id>                       # one system.test event through the real path
usertasks log-destinations status <id>                     # status, lag, last success / error
usertasks log-destinations update <id> --set enabled:=false
usertasks log-destinations reset-cursor <id> --set from=now

Splunk HTTP Event Collector

In Splunk: Settings → Data inputs → HTTP Event Collector → New token. Allow the index you want, then add a destination with the HEC base URL (https://splunk.example.com:8088) and the token. Events go to /services/collector/event, newline-delimited, at most 1 MB per request.

index=usertasks sourcetype="usertasks:event" category=workflow severity=error
| stats count by type, workflow_id
| sort -count

Splunk with a private CA: set NODE_EXTRA_CA_CERTS on the UserTasks server (certificate verification cannot be turned off per destination).

Datadog Logs

Create an API key (Organization settings → API keys) and pick your site. Logs arrive with ddsource:usertasks, service usertasks (configurable) and tags tenant_id, category and event_type, plus any you add (env:prod). Create facets on @type, @category, @workflow_id and @trace_id; no pipeline is required. Self-hosted servers can send every Datadog destination through a proxy or Observability Pipelines worker with LOG_EXPORT_DATADOG_URL.

OpenTelemetry (OTLP/HTTP)

Point a destination at any collector or vendor endpoint that accepts OTLP logs (JSON encoding): Grafana Cloud, Honeycomb, New Relic, Elastic or your own collector. Header values (API keys) are stored encrypted. Resources carry service.name=usertasks and tenant.id; event fields become usertasks.* attributes and traceId is the event's trace id.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318      # UserTasks endpoint: https://collector.example.com:4318
processors:
  batch: {}
exporters:
  debug:
    verbosity: basic
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

Webhook signatures

Webhook destinations receive POST { "events": [...] } with X-UserTasks-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256(secret, t + "." + raw body)> and X-UserTasks-Event-Count. Verify against the raw body, compare in constant time, reject old timestamps, and dedupe on events[i].id. Several v1= values can be present while a secret is rotated.

import { createHmac, timingSafeEqual } from 'node:crypto'

export function verifyUserTasksSignature(rawBody, header, secret, toleranceSeconds = 300) {
  const parts = Object.fromEntries(header.split(',').map((p) => p.trim().split('=')).filter((kv) => kv.length === 2))
  const signatures = header.split(',').map((p) => p.trim()).filter((p) => p.startsWith('v1=')).map((p) => p.slice(3))
  const t = Number(parts.t)
  if (!Number.isFinite(t) || Math.abs(Date.now() / 1000 - t) > toleranceSeconds) return false
  const expected = createHmac('sha256', secret).update(`${parts.t}.${rawBody}`).digest('hex')
  return signatures.some((sig) => sig.length === expected.length && timingSafeEqual(Buffer.from(sig), Buffer.from(expected)))
}

import hashlib, hmac, time

def verify_usertasks_signature(raw_body: bytes, header: str, secret: str, tolerance: int = 300) -> bool:
    items = [p.strip().split("=", 1) for p in header.split(",") if "=" in p]
    t = next((v for k, v in items if k == "t"), None)
    signatures = [v for k, v in items if k == "v1"]
    if t is None or not t.isdigit() or abs(time.time() - int(t)) > tolerance:
        return False
    expected = hmac.new(secret.encode(), f"{t}.".encode() + raw_body, hashlib.sha256).hexdigest()
    return any(hmac.compare_digest(sig, expected) for sig in signatures)

Syslog (RFC 5424)

For rsyslog, syslog-ng, the syslog inputs of Splunk, Graylog, Logstash / Elastic Agent and SIEMs such as QRadar or ArcSight. Each event is one RFC 5424 message:

<134>1 2026-10-24T11:59:58.250Z usertasks usertasks - workflow.run.completed [usertasks@32473 tenant_id="…" event_id="42" category="workflow" severity="info"] {"id":42,…}

PRI is facility × 8 + severity (debug 7, info 6, warn 4, error 3); MSGID is the event type; the message is the event JSON. tls (RFC 5425, port 6514, the default) verifies the receiver's certificate (NODE_EXTRA_CA_CERTS for a private CA) and can present a client certificate; tcp (RFC 6587) and udp (RFC 5426, one datagram per event, at most 8 KB) are unencrypted and need ALLOW_PRIVATE_EGRESS=true. Streams use octet-counting framing unless framing is lf.

module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1" StreamDriver.AuthMode="anon")
input(type="imtcp" port="6514")
if $app-name == "usertasks" then /var/log/usertasks.log

Azure Monitor (Log Analytics, Sentinel)

Uses the Logs Ingestion API. Create a custom table (e.g. UserTasks_CL), a data collection endpoint and a data collection rule whose stream Custom-UserTasks_CL declares TimeGenerated (datetime), EventId (long), Attributes (dynamic) and the string columns TenantId, EventType, Category, Severity, Message, SourceTable, SourceId, TraceId, RunId, WorkflowId, TaskId, ActorId. Grant an app registration Monitoring Metrics Publisher on the rule and add a destination with the DCE's logs-ingestion URL, the rule's immutable id, the stream, tenant_id, client_id and the client secret. At most 1 MB per call.

Azure Event Hubs

Batched REST sends to https://<namespace>.servicebus.windows.net/<hub>/messages; each message body is the event JSON, with tenant_id, event_type, category and severity as properties. Authenticate with a shared access policy that has Send (auth: sas, sas_key_name, credential sas_key) or an app registration with Azure Event Hubs Data Sender.

Google Cloud Logging and Pub/Sub

Cloud Logging receives entries.write batches in projects/<project>/logs/<log_id> with the event as jsonPayload, its severity, labels tenant_id / category / event_type and insertId <tenant>-<event id> (a re-sent batch is deduplicated); grant the service account roles/logging.logWriter. Pub/Sub publishes one message per event (data = event JSON) with the same attributes; ordering_key: tenant keeps each workspace's events in order (enable ordering on the subscription); grant roles/pubsub.publisher. Paste the service-account key JSON as the credential.

Amazon CloudWatch Logs

PutLogEvents batches (SigV4-signed) to log_group_name / log_stream_name; each log event's message is the event JSON and its timestamp the event time. Batches stay within CloudWatch's limits (1 MB including 26 bytes per event, 10,000 events, 24 hours between the oldest and newest event) and are sorted by time. Throttling (ThrottlingException, sent as HTTP 400) is retried with backoff. CloudWatch drops events older than 14 days, so a retention_start backfill only lands the last two weeks. With auto_create: true the group and stream are created on first use. The IAM key (or role) needs logs:PutLogEvents on the stream (and logs:CreateLogGroup / logs:CreateLogStream for auto_create). Either credential may be a secret://tenants/<workspace id>/… reference to the instance's secret store.

Cloud identities. The Azure, Google and AWS destinations authenticate with credentials the workspace supplies. The server's own managed / workload identity or AWS role (auth: managed_identity / workload_identity / instance_role) can reach whatever the operator's identity can, so it is refused unless the operator sets LOG_EXPORT_ALLOW_INSTANCE_IDENTITY=true (single-tenant installs). Token endpoints are fixed (Microsoft Entra ID per cloud, oauth2.googleapis.com) — a token_uri inside a key file is ignored — and tokens are cached until shortly before they expire.

Self-hosted notes