Apache Kafka

Apache Kafka is an open-source distributed event streaming platform.

Kafka is Distributed Commit Log Service:

A single topic is distributed across multiple brokers. Each message is called a log. As the messages are consumed by the consumers, the logs are committed. The committed logs will not be resent to the same consumer again.

Data Ingest

Data Ingest means importing data from external data sources to a storage medium. Data Ingest is achieved very easily in kafka with ETL(Extract Transform Load) pipelines.

Characteristics of Kafka

• Scalable: Kafka is a distributed system that supports multiple nodes
• Fault-tolerant: Data is persisted to disk and can be replicated throughout the cluster
• High throughput: Each broker can process hundreds of thousands of messages per second
• Low latency: Data is delivered in a fraction of a second
• Flexible: Decouples the production of data from its consumption

It is similar to a publish-subscribe messaging system. Originally created at LinkedIn, now an open source Apache project. Widely used for data ingest.