Brief Introduction and Summarization

• Kafka is a distributed messaging and event streaming system designed for high throughput, fault tolerant and horizontal scalable messaging.
• Producer sends messages → Broker Stores messages → Consumer Receives Messages
• Kafka VS RabbitMQ

• Organized into topics and partitions | - Message broker
• Routes messages through exchanges with routing keys into different queues | | Persistency and replay | Old messages can be replayed, messages and processes are saved in logs. | Queues can be durable, but messages are typically deleted after consumption | | Suitable for | - Real-time analytics and event streaming
• Activity feeds and log aggregation
• High-volume data pipelines | - Instant Message Delivery
• Real time notifications
• precise and rapid messaging |

Basic Components

Data Storage Architecture

Connecting and Processing

Lower Level Concepts and problem

Metadata Management - KRaft vs Zookeeper

Producers

Consumer

Brokers