Kafka is not a typical message broker: it’s the distributed nervous system that keeps Netflix, LinkedIn, and Uber running. It handles millions of events per second, loses none of them, and maintains guaranteed ordering per partition. This first installment explains the core concepts — topics, partitions, offsets, consumer groups — using the Smart City project from the Advanced Data Modeling course at Politecnico di Torino as a narrative thread: 50 ARPA Piedmont stations, real-time environmental data, full pipeline to InfluxDB and Grafana. Tenth installment of the Digital Stack 2026 series.
Imagine you’re the technical team at a regional environmental agency. You have 50 weather and air quality stations scattered across Piedmont. Every five minutes, each station sends readings of PM10, PM2.5, NO₂, O₃, temperature, humidity, and noise in dB. Fifty stations, twelve readings per hour, 365 days — roughly 5 million data points per year. How do you handle that?
The obvious answer — “write directly to the database” — works while volume is low. When stations grow to five hundred, or sensors start sending data every thirty seconds instead of every five minutes, or you want the same event processed simultaneously by three different systems (InfluxDB for time series, Elasticsearch for alerts, a notification service for critical spikes), that model collapses. Kafka is the solution to that problem. And not just that — it’s the solution Netflix, LinkedIn, and Uber chose when they faced the same challenge, scaled to hundreds of millions of users.
What Apache Kafka Is: Not a Message Broker, a Distributed Log
The most common definition of Kafka — “message broker” — is technically imprecise and practically misleading. A classical message broker (RabbitMQ, ActiveMQ) receives a message, delivers it to the recipient, then deletes it. Kafka works differently: messages are written to a persistent, ordered, immutable log — and they stay there for a configurable period (7 days by default). Consumers read from the log at whatever pace they prefer, from any point, as many times as they want.
This structural difference has enormous consequences. With a classical broker, if the consumer is offline when a message arrives and there’s no robust queuing system, the message is lost. With Kafka, the message is in the log: the consumer can retrieve it whenever it wants, go back in time, re-read the last six months of events to retrain an ML model, or connect a new service that needs to process the entire event history. It’s this persistence and reproducibility that makes Kafka fundamentally different from any other messaging system.
The Log Model: Why LinkedIn Invented Kafka
Kafka was born at LinkedIn in 2011 to solve a specific problem: every action of every user — profile view, link click, accepted connection — had to arrive in real time at dozens of different systems (recommendations, analytics, feed, notifications). Classical brokers couldn’t handle the volume or the need for multiple independent consumers on the same stream. The solution was to model the problem as a database log — append-only, ordered, distributed — rather than as a message queue.
The Four Concepts You Need to Understand
Kafka’s architecture is understood through four concepts. Once these are clear, everything else is operational detail.
Topic: the Logical Channel
A topic is the logical channel where messages are published — the equivalent of a “subject” or “category”. In the Politecnico Smart City project, there’s a topic called sensor-data: all readings from all ARPA stations go into that topic, regardless of which station or sensor type produced them. Another project might have separate topics by type (pm10-readings, temperature-readings) — the choice depends on how you want to organize the consumers.
Partition and Offset: Parallelism and Order
Every topic is subdivided into partitions — physically separate subsequences on disk, distributed across the cluster’s brokers. Partitions are Kafka’s parallelism mechanism: more partitions means more consumers can read in parallel, and higher maximum achievable throughput. As shown in the diagram above, the sensor-data topic has three partitions (P0, P1, P2): each partition is an independent sequence of messages with its own counter — the offset. The offset is the sequential number identifying each message within its partition: offset 77 means “the 78th message written to this partition” (counting starts from zero). Ordering is guaranteed within a partition, not across different partitions — an important detail for producer design.
Consumer Group: Parallelism on the Read Side
A consumer group is a set of consumers cooperating to read a topic: Kafka guarantees that each message is delivered to only one consumer within the group. In the diagram, Group A (InfluxDB Writer) has three consumers — one per partition: Consumer 1 reads only P0, Consumer 2 only P1, Consumer 3 only P2. Group B (Alert Service) has a single consumer reading all partitions. Both groups receive all the topic’s messages, but independently — adding a consumer group doesn’t slow down the others. This is Kafka’s superpower: the same data stream can be consumed simultaneously by completely different systems, each at its own pace.
The Smart City Project: Kafka in a Real Use Case
Project P1 of the Advanced Data Modeling course at Politecnico di Torino is the practical demonstration of all these concepts. The scenario: a regional environmental agency’s technical team needs to handle streaming data from 50 fixed stations with 12 readings/hour each — roughly 5 million data points per year. Kafka is the first layer of the pipeline after the sensors.
The pipeline above follows this flow: 50 stations send every five minutes their environmental readings (PM10, PM2.5, NO₂, O₃, temperature, humidity, noise) to Kafka as JSON events. One consumer group reads from Kafka and writes time series to InfluxDB — roughly 5 million data points per year. A second consumer group syncs station metadata (GPS geolocation, sensor model, configured thresholds) to MongoDB. A third consumer indexes operational events in Elasticsearch: an alert fired, maintenance was performed, an anomaly was detected. Grafana queries InfluxDB and Elasticsearch for the operational dashboard with maps, trends, and WHO thresholds. Four different technologies, all fed by the same Kafka stream — without any of them knowing the others exist.
Confluent Cloud: Kafka Without Installing Anything
Installing Kafka locally requires Java, Zookeeper (or KRaft in recent versions), broker configuration, and at least twenty minutes of terminal work. Confluent Cloud — the managed service from the company founded by Kafka’s creators — provides a working cluster in two minutes, with a free plan that includes enough storage and messages for development and experimentation.
The screenshot shows the sensor-data topic in the Confluent Cloud dashboard — exactly the topic for Project P1, created on May 9, 2026. The Messages tab shows 340 messages produced in the last hour and 631 consumed — a consumer group is processing the queue faster than the production rate, indicating a system that’s correctly draining its backlog. The table shows individual messages with all metadata: ISO8601 timestamp, partition (0 or 3 in this case), sequential offset (77 and 100), key (vehicle_id: 6396, 7166), and full JSON value ({"vehicle_id": 6396, "engine_temperature": 243, "average_rpm": 2301}). The bar chart at the top shows minute-by-minute traffic — useful for identifying production spikes and anomalies.
The Configuration That Matters: Partitions and Retention
The Settings tab shows the topic’s operational configuration: 6 partitions — more than the theoretical 3-partition diagram, to handle higher real-world throughput with a consumer group of 6; cleanup.policy: delete — messages are removed at retention expiry rather than being compacted (the other available policy); retention.ms: 1 week — messages remain available for 7 days, then are automatically removed; max.message.bytes: 2097164 — approximately 2 MB per message, sufficient for rich JSON payloads including aggregated data; retention.bytes: Infinite — no space limit, cleanup happens exclusively by time expiry.
The number of partitions is the most important design decision for Kafka performance: it determines the maximum parallelism on both the production and consumption sides. The practical rule: start with the number of consumers you expect in your most critical consumer group — that’s your lower bound for partition count. You can’t reduce partitions after topic creation (only increase), so moderate overestimation is safer than underestimating.
Why Kafka Scales Where SQL Doesn’t
A relational database like PostgreSQL is designed for complex queries, ACID transactions, and referential integrity. These guarantees come at a cost: every write must pass through the write-ahead log, acquire locks, update indexes. At a few thousand writes per second, PostgreSQL starts to struggle. Kafka is designed to do one thing extremely well: sequential disk appends. Sequential writes are an order of magnitude faster than random writes — Kafka exploits this physical characteristic of hardware to achieve throughput in the millions of messages per second on commodity hardware.
The right comparison isn’t Kafka vs PostgreSQL — they’re tools for different problems. It’s Kafka vs traditional message queues, where Kafka wins through persistence, reproducibility, and horizontal scalability. In the next article we’ll see how Kafka doesn’t just transport messages: with Kafka Streams it can transform them in-flight, aggregate them, enrich them, and connect them to destination databases via Kafka Connect — all inside the cluster, without external application code.
Up Next: Kafka Part 2 — Event-Driven Architectures and Integrations
This first installment gives you the foundational concepts: topics, partitions, offsets, consumer groups, brokers. Next week we get into the operational side: Kafka Streams for stateful in-flight transformations, Kafka Connect for connectors to InfluxDB and Elasticsearch, and the Lambda and Kappa architectures that use Kafka as their backbone. The narrative use case shifts: from P1 Smart City to P4 Social Network Analytics Platform — where Kafka feeds both the Neo4j graph and the Elasticsearch search engine in real time. Apache Kafka for stream processing: the nervous system that holds distributed data stacks together. In the next installment, we put it into production.
