Designing Near-Real-Time Data Pipelines with Apache Kafka and Debezium

Modern enterprises must process business events as they happen, not hours later. Building near-real-time data pipelines—where data flows from source to target systems with minimal delay—has become critical for analytics, personalization, and operational agility.

What Is a Near-Real-Time Data Pipeline?

A near-real-time data pipeline is an architecture where data moves from source systems (often OLTP databases) into downstream consumers (data lakes, analytics platforms, microservices) with latencies measured in seconds, not minutes or hours. This is typically accomplished by capturing changes as they happen (change data capture, or CDC), streaming them via scalable message brokers, and processing them incrementally.

Here’s a minimal, production-grade example using Debezium (v2.5), Apache Kafka (v3.6), and a PostgreSQL source:

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka:7.5.1
    depends_on: [zookeeper]
    ports: ["9092:9092"]
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
  postgres:
    image: postgres:15.5
    environment:
      POSTGRES_USER: demo
      POSTGRES_PASSWORD: demo
      POSTGRES_DB: demo
    ports: ["5432:5432"]
  connect:
    image: debezium/connect:2.5
    depends_on: [kafka, postgres]
    ports: ["8083:8083"]
    environment:
      BOOTSTRAP_SERVERS: kafka:9092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: debezium_connect_config
      OFFSET_STORAGE_TOPIC: debezium_connect_offset
      STATUS_STORAGE_TOPIC: debezium_connect_status
      CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      CONNECT_REST_ADVERTISED_HOST_NAME: connect

This configuration sets up a local environment for testing, but the same patterns scale to production with managed Kafka (e.g., Confluent Cloud, Amazon MSK) and cloud-hosted databases.

Key insight: Near-real-time pipelines demand careful tool selection and configuration to achieve both low latency and high reliability.

How Debezium and Kafka Enable Scalable Streaming

Debezium: Reliable Change Data Capture (CDC)

Debezium is an open-source CDC platform that streams row-level data changes from databases such as PostgreSQL, MySQL, SQL Server, MongoDB, and Oracle. It reads the database write-ahead logs (WALs or binlogs) to produce a reliable, ordered stream of inserts, updates, and deletes.

Apache Kafka: Scalable, Durable Event Transport

Kafka is a distributed, horizontally scalable message broker designed for high-throughput, low-latency event streaming. It decouples data producers (Debezium) from consumers (analytics, microservices, sinks), ensuring durability and replayability.

Connecting Debezium to Kafka

A Debezium Kafka Connect connector is typically registered via a REST POST to the Kafka Connect API. Here’s a real example for PostgreSQL:

{
  "name": "debezium-postgres-source",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "demo",
    "database.password": "demo",
    "database.dbname": "demo",
    "database.server.name": "postgres-server",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_slot",
    "table.include.list": "public.orders,public.customers",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
  }
}

Key insight: Kafka and Debezium together provide a foundation for building robust, near-real-time data pipelines that are resilient to failure and scale horizontally.

Step 1: Provisioning and Hardening Kafka for Production

1.1. Choosing the Right Kafka Deployment

In production, you typically deploy Kafka via managed services (Confluent Cloud, Amazon MSK, Azure Event Hubs for Kafka) or self-managed clusters (on Kubernetes with Strimzi or Confluent Operator). Managed services reduce operational burden and offer built-in encryption, autoscaling, and monitoring.

1.2. Networking, Security, and Fault Tolerance

Ensure your Kafka brokers are deployed in multiple availability zones (AZs) for resilience. Enable TLS encryption (set listeners=SSL://:9093 and configure keystores/truststores) and SASL authentication (SASL_SSL) for client/broker security. Always enable topic replication (minimum replication.factor=3) and set min.insync.replicas=2 to ensure durability during node failures.

1.3. Monitoring and Alerting

Deploy tools like Prometheus and Grafana for cluster metrics, and configure Kafka’s JMX metrics exporter. Set up alerting on under-replicated partitions, disk usage, and consumer lag.

Key insight: A production Kafka setup must prioritize multi-AZ deployment, encrypted transport, authentication, and real-time monitoring.

Step 2: Deploying Debezium Connectors for High Availability

2.1. Running Kafka Connect in Distributed Mode

For resilience, run Kafka Connect in distributed mode (at least 3 nodes), not standalone. On Kubernetes, deploy as a StatefulSet or via the Strimzi operator. Use health checks and configure autoscaling. Sample deployment for GKE:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-connect
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kafka-connect
  template:
    metadata:
      labels:
        app: kafka-connect
    spec:
      containers:
      - name: kafka-connect
        image: debezium/connect:2.5
        ports:
        - containerPort: 8083
        env:
        - name: BOOTSTRAP_SERVERS
          value: my-kafka-cluster:9092

2.2. Configuring Connector Offsets and State

Configure offset.storage.topic and status.storage.topic on highly-replicated Kafka topics. This ensures the connector can resume from the last processed event even after failure.

2.3. Securing Database Connections

For PostgreSQL, create a dedicated, minimally-privileged user for Debezium with replication rights only on required tables. Always use TLS/SSL connections (sslmode=require).

Key insight: Running Debezium connectors in distributed mode with resilient offset storage and secure database access is non-negotiable for production.

Step 3: Managing Schema Evolution and Data Contracts

3.1. Using Schema Registry

Kafka topics benefit from a central schema registry (e.g., Confluent Schema Registry v7.5, AWS Glue Schema Registry) to manage Avro, Protobuf, or JSON schemas. This enables forward/backward compatibility and prevents schema-breaking changes from reaching consumers. Example curl to register a schema:

curl -X POST http://schema-registry:8081/subjects/orders-value/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\"type\":\"record\",\"name\":\"Order\",...}"}'

3.2. Enforcing Schema Compatibility

Set compatibility levels to BACKWARD or FULL to ensure consumers don’t break as producers evolve. This is crucial when downstream analytics (e.g., Spark, Flink) expect fixed field sets.

3.3. Data Lineage and Auditing

Deploy tools like OpenLineage or Marquez to track field-level lineage from source tables to Kafka topics and downstream consumers. This is vital for compliance and debugging.

Key insight: Managing schema evolution and lineage with a schema registry prevents silent data corruption and enables safe, incremental pipeline changes.

Step 4: Scaling and Optimizing Latency in Streaming Pipelines

4.1. Partitioning and Throughput

Kafka topics should be partitioned based on expected throughput and the number of parallel consumers (e.g., orders topic with 12 partitions for 50k TPS systems). Set num.partitions accordingly and monitor partition skew.

4.2. Tuning Consumer Lag and Batch Size

Configure consumer group settings like fetch.min.bytes, max.poll.records, and adjust auto.offset.reset for replay behavior. For near-real-time, keep batch sizes low (e.g., max.poll.records=100) and poll intervals short. Debezium’s poll.interval.ms can be set to 500ms for sub-second latency.

4.3. Downstream Processing: Stream vs. Batch

Downstream, use Apache Flink (v1.17) or Apache Spark Structured Streaming (v3.5) for real-time transformation and enrichment. Prefer stateless or windowed aggregations for lowest latency; batch sinks (like S3, BigQuery) should be tuned for micro-batching.

4.4. Cloud-Native Scaling

On cloud, leverage autoscaling (Kubernetes HPA/VPA), managed Kafka scaling (Confluent Cloud, MSK Serverless), and serverless ETL (AWS Glue Streaming, Azure Stream Analytics) for elastic throughput.

Key insight: Achieving sub-second latency and high throughput requires tuning partitioning, batching, and scaling dynamically to match workload spikes.

Tool Comparison: Kafka, Pulsar, Kinesis, and Event Hubs

Feature	Apache Kafka	Apache Pulsar	AWS Kinesis	Azure Event Hubs
Maturity	2011+ (Very High)	2016+ (High)	2013+ (High)	2014+ (High)
Message Retention	Configurable	Tiered, Infinite	7 days (std)	7 days (std)
Multi-Tenancy	Weak	Strong	Medium	Strong
Partition Scaling	Manual	Dynamic	Shard-based	Partition-based
Cloud-Native	Self/cloud/managed	Managed/cloud	Fully managed	Fully managed
Schema Registry	Yes (Confluent)	Yes (builtin)	Yes	Yes
Geo-Replication	MirrorMaker, CC	Built-in	Cross-region	Cross-region
Ecosystem (CDC, ETL)	Extensive	Moderate	AWS-focused	Azure-focused
Latency (P99, 1KB msg)	~5ms-50ms	~10ms-50ms	~70ms-200ms	~80ms-200ms

Key insight: Apache Kafka leads in ecosystem and lowest latency, but Pulsar, Kinesis, and Event Hubs may be preferable for multi-tenancy, dynamic scaling, or cloud-native simplicity.

Frequently Asked Questions

Q: How do I minimize end-to-end latency in a Debezium/Kafka pipeline? A: Optimize database WAL settings (e.g., wal_level=logical in Postgres), set Debezium poll.interval.ms to 500ms or lower, use low-latency Kafka settings (e.g., acks=all, linger.ms=5), and ensure consumers process messages promptly with small batch sizes. Monitor consumer lag and use sufficient partitions to avoid bottlenecks.

Q: What are the most common failure points in near-real-time data pipelines? A: The most frequent failures I see are Kafka broker outages (mitigated by multi-AZ deployment), connector crashes (mitigated by distributed mode and topic-based offset storage), schema mismatches (prevented with schema registry), and downstream consumer lag. Proactive monitoring and alerting are essential.

Q: Can I use Debezium and Kafka for non-relational sources or SaaS APIs? A: Yes, Debezium supports MongoDB and there are Kafka Connectors for many SaaS APIs (Salesforce, S3, etc.). For less common sources, custom Kafka Connect connectors or open-source projects like StreamSets or Airbyte can bridge the gap.

Key Takeaways

• Deploy Kafka in multi-AZ mode, enable TLS/SASL, and monitor with Prometheus for production reliability.
• Always run Debezium in distributed mode and secure its offset storage on replicated Kafka topics.
• Register and enforce schemas with a schema registry to catch breaking changes before they hit consumers.
• Partition Kafka topics and tune consumer batch settings to achieve sub-second pipeline latency.
• Choose the streaming platform that matches your team’s expertise, latency, and scaling needs—Kafka is best for ecosystem breadth and low latency, Pulsar for multi-tenancy, Kinesis/Event Hubs for cloud simplicity.
• Continuously monitor lag, throughput, and failures—near-real-time is only as good as your weakest link.