The reliability story
Server-side event delivery isn't always a one-shot success. Ad platform APIs occasionally hit rate limits. Networks blip. Platform backends experience transient 5xx errors during deploys. Without proper handling, any of these would cause events to silently disappear, producing under-reported attribution and broken audiences.
Echo handles this in four layers: batching (chunk events to safe per-request limits), automatic retry (retry transient failures with exponential backoff), a dead letter queue (preserve failed batches with their failure reason), and automatic resend (redeliver captured batches once the destination recovers). Every CAPI-style destination in Echo has the same baseline so behavior is predictable regardless of which platform you're targeting.
Batching
Each ad platform has a maximum number of events you can include in a single API request. Echo chunks your events to a safe size below each platform's hard limit to avoid 400 Bad Request errors on oversized payloads.
Meta, TikTok, Pinterest, Snapchat, Reddit, and Google Ads chunk at 500 events per request (these platforms have 1000-event hard limits, so this gives a safety margin). LinkedIn chunks at 100 per its documented limit. Google Analytics 4 chunks at 25 per the Measurement Protocol maximum, grouped per visitor. Microsoft Ads and Floodlight are special cases, since neither has a batch endpoint; Echo fires one request per event and handles each as its own batch for uniform retry and DLQ handling.
Per-batch error handling means one bad batch doesn't take down others. If an invocation has 1500 events that split into three batches of 500, and the middle batch fails after retries, the other two batches still deliver successfully. Only the failed batch lands in the DLQ.
Automatic retry
When a batch fails with a transient error, Echo retries up to 3 times with exponential backoff: 500ms, then 2 seconds, then 5 seconds. That's a maximum added latency of about 7.5 seconds per failing batch before giving up.
Not all errors are retryable. Echo retries on HTTP 429 (rate limit), HTTP 5xx (server errors), and network-level failures (connection refused, timeouts). These represent transient conditions where the next attempt might succeed. Echo does not retry HTTP 400 Bad Request, 401 Unauthorized, or 403 Forbidden errors, because those typically mean bad payload or bad credentials and retrying won't help; they go straight to the DLQ.
Retry is only safe on destinations that support idempotency via event_id or similar. Meta, TikTok, Pinterest, Snapchat, and Reddit all support this. If Echo's first attempt succeeded but the response got lost in transit (a common timeout scenario), retrying would normally create a duplicate. These platforms handle the dedup so you never double-count. GA4 Measurement Protocol is not retried, because it has no native dedup and retrying on an ambiguous failure would inflate numbers.
Every event Echo sends includes a unique event_id generated by the SDK. Ad platforms use that ID to deduplicate any duplicate deliveries, regardless of what caused them. This means retries cost nothing in data quality: they just improve delivery rates without inflating counts.
Dead letter queue (DLQ)
If all retries are exhausted, Echo writes the failed batch to the DLQ. The DLQ uses a "claim check" pattern: the full batch payload goes to S3 at a path like s3://tagpipes-echo-dlq/failed/{"{"}date{"}"}/{"{"}destination{"}"}/{"{"}sourceId{"}"}/{"{"}batchId{"}"}.json, and a small pointer message goes to SQS with metadata (destination, source, reason, S3 location).
This preserves the events with their failure reason, so nothing is silently dropped. Every batch records what failed, at which stage, and how many events it carries. Because every event carries an event_id, redelivery is safe on idempotent destinations: any duplicates are deduped by the platform.
Failed and successfully-resent DLQ payloads expire from S3 after 30 days via a bucket lifecycle policy. Parked batches (see below) do not expire, so a resend is always possible once the underlying problem is fixed.
Automatic resend
A captured batch does not just sit in the DLQ waiting for someone to notice. Echo resends it automatically: the batch is queued for redelivery and retried up to 4 times with increasing delays between attempts, so a destination that was down for maintenance or mid-token-rotation picks the events back up on its own.
Two limits keep automatic resend safe. First, batches older than 7 days are never resent, because ad platforms reject conversions that stale and the attempt would only burn API calls. Second, after the fourth failed attempt the batch is parked. A parked batch is still stored safely; it just stops retrying until a system administrator resends it manually after fixing the underlying cause.
The Lapse destination tracks abandonment journeys with its own state. Replaying old Lapse events would re-open journeys and re-fire abandonment signals, so Lapse batches are excluded from both automatic and manual resend.
What the destination page tells you
The Events Failed tile on the destination detail page reports what actually happened to failed events, not just that they failed. "Saved for resending" means the events are captured and will redeliver automatically. "Held" means they are stored safely but automatic resend is not available for them, so they wait on a manual resend. A red banner appears only when events failed and could not be saved anywhere; that banner should never appear, and if it does, contact support.
System administrators additionally see a Dead-Letter Batches panel at the bottom of the page. It lists every held and parked batch from the last 7 days with its failure reason, phase, payload shape, and resend attempt history, plus a reconciliation line that checks failed events are fully accounted for as captured or lost. Each eligible batch has a Resend button; a confirmation dialog shows how many events will be redelivered before anything is queued. Batches older than 7 days and Lapse batches are shown but cannot be resent.
What you see in the dashboard
The Echo dashboard inside TagPipes shows per-destination metrics updating in near-real-time:
- Events Received — count of events the Processor ingested for this destination.
- Events Sent Success — events successfully delivered.
- Processing Errors — events that failed delivery even after retries.
- DLQ Writes — number of batches pushed to the DLQ.
- DLQ Events — actual event count in the DLQ (batches times events per batch).
Healthy operation: received and sent success climb together. DLQ metrics stay at zero. If DLQ writes are non-zero, something is consistently failing and worth investigating in CloudWatch logs for the tagpipes-echo-processor Lambda.
Tips
- Healthy operation has zero DLQ writes. Investigate any climb, even a small one, before it becomes a trend.
- Access tokens expire on most platforms. A sudden DLQ climb for one destination usually means a token rotation is due; once you fix the credentials, the captured batches resend themselves.
- CloudWatch logs for the Processor Lambda contain detailed retry attempts like
Meta CAPI batch 2/3 failed (HTTP 503). Retrying in 2000ms.which let you see the retry loop working. - DLQ payloads contain the full transformed event, so you can inspect exactly what was being sent when it failed.
- GA4 sends are single-attempt at the HTTP layer because GA4 has no native dedup. Failed GA4 batches still reach the DLQ and resend from there.
Troubleshooting
DLQ writes climbing for one destination
Open the S3 DLQ bucket and look at a recent payload. The failureReason field in the payload tells you what went wrong. Common causes: expired access token (401/403), hitting rate limits repeatedly (429 even after retries exhausted), or platform API outage (5xx for extended period).
Processing errors rising without DLQ writes
This pattern usually means the Lambda is hitting transient errors that don't reach the DLQ path, like a Lambda timeout or an exception during event transformation (not during API call). Check CloudWatch for Lambda errors and memory usage. Increase Lambda timeout or memory if needed.
DLQ configured but writes not appearing
Check the Lambda env vars DLQ_S3_BUCKET and DLQ_SQS_QUEUE_URL are set. Also check the Lambda execution role has s3:PutObject on the DLQ bucket and sqs:SendMessage on the queue. If either is missing, the DLQ write logs as [DLQ] Failed to write... AccessDenied but doesn't block the Processor.
A batch keeps failing and then stops retrying
That is the parking behavior working as designed. After 4 failed resend attempts the batch is parked so it stops burning API calls against a broken destination. Fix the underlying cause (usually credentials or destination configuration), then have a system administrator open the destination detail page and click Resend on the parked batch. Parked batches do not expire.