Skip to content
IP FediuninFull-Stack & AI Engineer
← All case studies

Case 05 · Distributed Systems / Backend Architecture

Event-Driven Backend Architecture

Asynchronous event processing and service-to-service communication

Overview

Server-side architectures where components talk through message brokers and work is done asynchronously, by background handlers and event consumers, rather than inside one HTTP request.

Generalised experience across several systems

This case sums up work on several systems: RabbitMQ and Apache Kafka were used in different projects, so the demo shows them separately too.

The charts are computed by this case's model on synthetic data — the same model that runs the interactive demo. They show what the approach changes, not figures from a client's system.

What I did

  1. Service interaction through brokers
  2. Messaging on RabbitMQ and Kafka
  3. Background task processing
  4. Redis and PostgreSQL in the system
  5. Analysis of delivery failures

Before

  • Clients wait while slow work finishes
  • One slow or unavailable component holds up the chain
  • Work depends on external systems and several services

After

  • The API answers at once, the work runs in the background
  • Handlers scale independently of the API
  • Every service that needs an event gets it

Time to finish 12 tasks

2.8×faster with three competing workers instead of one

  • 1 worker36 time units
  • 2 workers18 time units
  • 3 workers13 time units
The demo queue: tasks of 2–4 time units, prefetch 1, every task acknowledged when done.

How it works

A model of the system you can run: watch requests and data move through the components. It is a simplified reconstruction on synthetic data, not the production code.

What is happening

Switch between RabbitMQ and Kafka — the camera flies to that district. These are two different ideas: a task queue and an event log.

The model runs in the browser on synthetic data. Times on the cards are illustrative, and motion is slowed down so the path can be followed.

Broker

A queue shares out work: a message goes to one handler and disappears once acknowledged. A log keeps events: every group reads them from its own position and can read them again.

RabbitMQ task queue

Workers

Engineering detailsThe full write-up: problem, architecture, implementation decisions and outcome.

Problem

Not every piece of work can be done while a client waits for an HTTP response: processing can take long, depend on external systems or touch several services. Done synchronously, one slow or unavailable component holds up the whole chain. The answer is an architecture where services exchange messages and work runs asynchronously.

Solution

Services that exchange messages instead of waiting on each other: the API publishes a message and answers at once, and handlers do the work in the background — RabbitMQ shares tasks between workers, Kafka gives several consumers the same events.

My role

  • Designed how services interact through message brokers.
  • Organised messaging through RabbitMQ and worked with Apache Kafka.
  • Implemented background task processing and split system components by responsibility.
  • Used Redis and PostgreSQL as parts of such systems.
  • Analysed failures in processing and delivering events.

Technical architecture

API → message broker → handler → database → notification. The API accepts the request and publishes a message; the work is done by handlers that scale independently of the API.

Implementation

  • A queue and a log are different tools. In a RabbitMQ queue a message goes to one handler and is removed once acknowledged — that is how work is shared out. In Kafka, events are kept in the topic's partitions, and each consumer group reads them from its own position and can read them again — that is how several independent consumers get the same event.
  • Acknowledgements. A message counts as processed only once acknowledged; if the handler dies before that, the message goes to another one.
  • Order. Kafka guarantees order within a partition, so events of one entity are published with one key.
  • Redelivery. Because of redeliveries a handler can receive a message twice — processing must cope with that.

Functional outcome

Services exchange data through message brokers, and long-running work happens in the background: the API answers quickly, the load is spread across handlers, and events are available to every service that needs them.

Need a system like this?

Tell me about the problem — I'll come back with questions and a proposed approach.

Discuss your projectHiring? See my engineering profile