Pulse Ingest collects product analytics events from web and mobile clients and lands them in the warehouse. Roughly 40M events/day at baseline, with bursts to ten times that during campaign launches.
Tip
Ingestion pipelines fail in two directions — dropping events and duplicating them. Almost every design decision here is about picking which one to risk at each hop.
Problem
The original pipeline posted events directly to a service that wrote row-by-row into the warehouse. Under burst traffic:
- The warehouse became the bottleneck, so the HTTP endpoint started timing out and clients dropped events with no retry.
- Retries that did happen produced duplicates, because writes weren't idempotent. Analysts hand-deduplicated in SQL.
- There was no replay. A bad deploy that mangled a field lost that field for the window it was live.
Definition of done: accept events faster than the warehouse can absorb them, guarantee no duplicates in the final tables, and be able to replay any window.
Architecture
Three stages with a durable buffer between accept and load, and deduplication at the point of load rather than the point of ingest.
clients ─► collector ─► Kafka (raw) ─► transformer ─► Kafka (clean) ─► loader ─► warehouse
│ │ │
validate + schema map + merge on
assign event_id reject to DLQ (event_id, day)- Collector does the minimum. Validate shape, stamp
event_id(client UUID + server-received timestamp), append to Kafka, return 202. It never touches the warehouse, so warehouse latency can't cause client-side drops. - Transformer is stateless. Schema mapping and enrichment; anything that fails validation goes to a dead-letter topic with the original payload attached, so it can be fixed and replayed rather than lost.
- Loader batches and merges. Micro-batches every 60 seconds, written to
staging, then merged into the target table keyed on
(event_id, event_day). Re-running a batch is a no-op — that's where exactly-once actually comes from.
Why dedupe at load, not at ingest
Deduplicating at ingest requires a fast shared store of seen ids and a decision about how long to remember them, which becomes its own availability problem. The merge key makes duplicates harmless downstream instead, so upstream retries can be aggressive and dumb.
Warning
This design accepts duplicate rows in Kafka as normal. Any consumer reading the raw topic directly has to handle that — a footgun worth documenting loudly for the next team.
Stack
Durable buffer and replay log; raw and clean topics with separate retention.
Collector and transformer, with schema validation at both boundaries.
Staging-to-target merges and downstream models.
Lag, DLQ rate, and end-to-end latency as the three alerting signals.
Results
| Metric | Before | After |
|---|---|---|
| Peak accepted throughput | ~4k events/s (then timeouts) | 60k events/s |
| Duplicate rows in target tables | ~0.3% | 0 |
| Recovery from bad deploy | none — data lost | replay from raw topic |
| p95 ingest-to-queryable latency | 12 min | 90 s |
What I'd do differently
I chose 60-second micro-batches to keep warehouse costs predictable. That was the right call for cost and the wrong one for the two dashboards that people actually watched during launches — they wanted seconds. A streaming path for a small, explicitly listed set of event types alongside the batch path would have served both without making everything expensive.
I'd also define the schema contract in one machine-readable place from the start. We had Pydantic models on one side and dbt models on the other, kept in sync by review discipline, which held right up until it didn't.