Skip to main content
Back to Blog
Architecture

Architects: Event Driven Architecture for 99.9% Telemetry Uptime

September 202612 min read
Event-driven architecture for telemetry uptime

Event-driven architecture is an asynchronous programming style where decoupled services publish and react to events instead of calling each other directly. Gains are independence and scale in real-time, at the cost of operational complexity, eventual consistency, and more difficult debugging. Read on for the details, topologies, and a concrete implementation checklist.

TL;DR:

  • Using at-least-once delivery with idempotent consumers reduces manual error handling during event processing.
  • Design against a well-defined schema registry and versioning policy to avoid accidental breaking changes across evolving events.
  • The outbox pattern provides atomicity of event publication with the state update that generated it, preventing event loss in the case of a crash.
  • A hybrid topology with brokers and mediators can improve decoupling and error recovery for complex, long-running workflows.
  • A persistent event log enables replay, real-time analytics, and faster incident reconstruction, while adding some latency and operational overhead.

PODTECH

Build smarter telemetry systems. PODTECH develops custom software for telemetry, datacenter management, and reliable integration of mission-critical infrastructure.

Learn more about PODTECH

Table of Contents

What is event-driven architecture, and what counts as an event?

An event is a statement of something that has already happened: an order was submitted, a sensor passed a threshold, a payment went through. It is not a request for something to happen, which is the fundamental difference between event-driven architecture (EDA) and request-response design. In EDA, decoupled components communicate by means of the production, detection, and consumption of events, and no producer need know who, if anyone, is listening.

Four roles make up the model:

  • Producers raise events when state changes, without waiting for a response.
  • Brokers or channels carry events between services and often persist them in a durable log.
  • Consumers read and act on events. Often, many consumers read the same stream of events independently of one another.
  • Event schemas describe the form and semantic meaning of an event, so producers and consumers can agree on structure and intent.

In contrast, a command directly instructs a service what to do, and by implication often requires a synchronous response. Since publishing and consumption are asynchronous and may not occur at the same time, EDA operates on eventual consistency: each downstream service will catch up, but not necessarily immediately. Delivery guarantees such as at-least-once, at-most-once, and exactly-once must be consciously traded off instead of taken for granted.

Core components and who owns what

Each role has clear responsibilities in an event-driven system, and most incidents stem from misunderstanding them.

  • Producers have to determine whether the event is published within the same transaction as the state change or at a later time, which is precisely the issue the outbox pattern aims to address.
  • Brokers split into queues and streams. Queues are point-to-point with one consumer per message, while streams are durable, replayable, and support multiple independent readers. Partitioning strategy influences ordering and throughput.
  • Consumers should be idempotent, as most brokers provide at-least-once delivery semantics, and they should be able to scale horizontally with competing consumers on the same partition or queue.

Schema registries maintain versioning of events so producers can evolve fields without silently breaking every downstream consumer.

Dead-letter queues, retries, and the outbox pattern absorb failures rather than losing events, as Microsoft’s .NET architecture guidance suggests, specifically to make event publication atomic and resilient alongside the underlying state update.

Pro Tip: Write events to an outbox table in the same database transaction as your state change, and have a separate publisher push those rows to the broker. It closes the gap where a crash between “save state” and “publish event” would otherwise lose data silently.

Choosing a topology: broker, mediator, or a hybrid

There are two prevalent topologies for building practical event-driven systems. The Azure Architecture Center summarizes the tradeoff as a function of how much central control you are willing to sacrifice for decoupling.

  • Broker topology: services publish and subscribe directly through a lightweight broker, with no central coordinator. This gives maximum decoupling and broadcast semantics, but end-to-end error handling is relatively weaker.
  • Mediator topology: a central orchestrator receives the first event and coordinates a multi-step workflow. This gives stronger control and easier error recovery, at the expense of a coordination point.
  • Event streaming uses a partitioned, durable log as the base layer, which provides each independent consumer with its own read position.
  • Publish-subscribe fans a single event out to many interested consumers without expecting responses.
  • Complex event processing (CEP) matches patterns across several event streams in near real time, and can be used to trigger alerts such as fraud or anomaly detection.

Mixed forms are prevalent in the real world: one mediator for a critical order-fulfilment saga, with a broker-style pub/sub dispatching analytics events to whatever teams need them, all atop the same underlying stream.

BrokerMediatorHybridSvc ASvc BBrokerC1C2EventMediatorStep 1Step 2Step 3Step 4SourceBrokerAnalyticsWorkflowMediatorMax decouplingStrong workflow controlCommon production pattern

The operational payoff: why teams accept the added complexity

Decoupling is the big win: producers and consumers launch, scale, and fail independently, so a burst in one consumer’s workload never starves the producer or its other subscribers. Message queues and streams absorb bursts as a buffer, insulating services with back-pressure protection instead of cascading failures under load.

Persistent event logs become an audit trail with no extra cost. All state changes are serialized in order and can be replayed, enabling real-time analytics dashboards as well as a replayable history for forensics by operations teams after an incident.

  • Independent release cycles per service, since consumers don’t need producers to redeploy.
  • Buffered load spikes instead of dropped requests during traffic bursts.
  • Replayable history for analytics, reprocessing, and audit.
  • Faster incident forensics because the event log shows exact sequence and timing.

Queues and brokers do introduce latency and overhead to manage, so the benefit only justifies the cost under bursty or highly decoupled workloads, not as a blanket default for every internal call.

When EDA fits, and when it doesn’t

Run through this checklist before committing to an event-driven design:

  1. Multiple subscribers need the same fact. If three or more services care about “order placed”, pub/sub beats three point-to-point integrations.
  2. The load is bursty or unpredictable. IoT telemetry, seasonal e-commerce traffic, and financial market data all spike in ways synchronous APIs handle badly.
  3. You need replay. Retracing the steps of yesterday’s events after fixing a bug, or replaying the state for a new consumer, necessitates a durable log, not a transient queue.
  4. Producers and consumers have to be independently deployable. Separate teams, separate release cycles, separate failure domains.

Typical use cases are e-commerce order processing pipelines, high-volume IoT and telemetry ingestion, financial transaction pipelines with audit logging needs, or datacentre monitoring when thousands of sensor readings come in simultaneously and multiple systems require access to the same feed.

The false positives are as important as the true positives. Most small applications with one database and two services don’t require a broker layer. Workflows with hard real-time requirements or strict consistency, like a single atomic transfer between two accounts, will likely have lower latency with a direct call rather than asynchronous messaging architecture.

What goes wrong: challenges and trade-offs to plan for

Event-driven systems pay for flexibility with an unpredictable call stack, which you feel most during testing and incident response. There is no linear trace to follow when a bug lives across five asynchronous consumers, which is why distributed tracing and durable event logs need to exist before you need them, not after an outage.

  • Testing and debugging get harder without a synchronous call chain to step through.
  • Schema evolution creates semantic coupling across teams even when services are technically decoupled.
  • With delivery semantics, each consumer must deal with duplicates and out-of-order messages, so idempotence is a requirement.
  • Operational overhead increases with the scale of brokers, storage retention policies, and governance required to know which person or team owns which event schema.

This is among a number of what I call “implementation traps”: deciding on delivery semantics later, at production time, instead of sooner. “Solve for X” means leaving the decision until the last minute, rather than baking it in at design time. In this case, delivery semantics are left as an “X” to be solved later, with the consequence that they must be retrofitted rather than built in. Retrofitting idempotence under time pressure is usually bad.

A practical implementation checklist for architects

Don’t wait for the first incident. Bake these decisions in before the first event ships.

  1. Explicitly choose your delivery semantics. At-least-once with idempotent consumers is the most forgiving default for most systems.
  2. Use the outbox pattern to ensure that publication of the event is atomic with the state change that caused it to be published.
  3. Stand up a schema registry early, with a well-defined versioning policy for adding, deprecating, or renaming fields.
  4. Pick patterns intentionally. Use event sourcing and CQRS if you have audit-rich domains, claim-check if you have large payloads, competing consumers for horizontal scaling, and so on.
  5. Instrument for observability first. Tie a correlation ID to every event and log it at every producer and consumer, so that a single request can be traced through the entire flow.
  6. Use contract tests and failure drills. Test schema compatibility, exercise replay in integration tests, and run chaos drills that simulate broker or consumer failure.
  7. Govern access and retention. Encrypt events in transit and at rest, and define retention policies that meet replay and compliance requirements.

Pro Tip: Log correlation IDs everywhere and persist your event log durably from day one. When something goes wrong six months from now, that log is the only thing that will let you reconstruct exactly what happened and when.

Partition key design warrants special attention in streaming systems: trade-offs between ordering guarantees and parallelism are only apparent under realistic load, so test hot-key scenarios prior to production.

PODTECH’s perspective: EDA in mission-critical telemetry

Datacentre monitoring is among the simplest examples where event-driven design seems appropriate, because sensor readings, BMS alarms, and PMS signals all arrive continuously and multiple systems need to read the same feed concurrently. A pipeline architected according to best practices will route telemetry from collection through a durable stream to independent consumers, with replay available specifically for incident recovery, so a fault can be replayed event by event rather than guessed at after the fact.

That’s the thinking behind PODVIEW, our datacentre monitoring platform. It’s also the discipline we’ve applied to achieve a 99.9% uptime SLA on many of the projects we’ve delivered. Event replay for faster incident detection isn’t theory. It’s the difference between discovering a fault in minutes and piecing it together from logs the next day.

If your infrastructure requires an event-driven backbone rather than bolt-on integration, PODTECH’s enterprise automation services build and operate that pipeline end to end, from broker selection through observability. Teams looking to add real-time anomaly detection on top of an existing event stream can also consider PODTECH’s machine learning development services to turn raw telemetry into predictive alerts rather than after-the-fact reports.

Why architects overrate decoupling and underrate operations

Event-driven architecture pitches always begin with decoupling, and that’s the wrong message. Decoupling is a thing, but it’s the easy part: publish an event, allow anyone to subscribe, congratulations. The thing that separates systems that work in production from systems that cause 2AM pages is everything downstream of that first publish call, namely delivery semantics, idempotence, schema governance, and the ability to actually trace what happened when three consumers all processed the same event differently.

Most teams make the decision to adopt EDA after a synchronous call chain became fragile under load. Teams then quickly find that asynchronous messaging architecture trades one class of failure for another. A timeout becomes a duplicate message. A slow consumer becomes a growing backlog instead of an obvious error. That’s not an argument against event-driven integration. It’s an argument for treating observability and idempotence as day-one requirements, not follow-up tickets.

The architects who benefit most from event sourcing patterns and event streaming are those who make peace with eventual consistency as a design constraint up-front. The rest re-introduce tight coupling through workarounds. Design the outbox pattern in from the start. Instrument correlation IDs before you need them. Accept the durable log as your source of truth, not an afterthought bolted on for debugging. Everything else in this architecture is negotiable; that discipline isn’t.

— Harry

Sources

Azure Architecture Center at Microsoft has additional depth for technical background into both broker and mediator topologies. Designing Event-Driven Systems covers event sourcing and streams-as-source-of-truth in depth. Guidance from Microsoft on .NET async messaging describes the outbox pattern. Splunk has a good EDA overview, and it’s also a useful article on observability in async flows. For a construction-sector perspective on coordination challenges similar to eventual consistency, check out this post on process improvement.

Recommended