Confluent Kafka

It is a platform which includes apache kafka with additional eco system components. Confluent kafka is available only for unix flavours.
Components of Confluent Kafka:

• zookeeper
• kafka
• connect
• schema registry ----------------provides registry for publishing avro schema
• rest proxy -------------------provides restful webservices for interacting with kafka
• ksql server ----------------------provides a sql based interface for interacting with kafka
• control center-------------------web based ui for monitoring and administering kafka

subdirectories of confluent:

• bin --------- binary scripts
• share -------libararies in jar format
• etc---------config files
• lib-----------system services
• src---------source code

Avro Format:
It is an opensource data exchange and serialization format. It stores the schema in json format making it easy to interpret and stores the data in binary format making it more compact and efficient. Sample avro schema:

{
"namespace":"com.boa.training",
"type":"record",
"name":"customer",
"fields":[
{"name":"id","type":"int"},
{"name":"name","type":"string"},
{"name":"email","type":"string"}
]
}

some of the types include record,enum,array and etc. Schema Registry: It is the storage layer for avro schema. It stores a versioned history of all schemas based on the id.

	avro-data--->byte[]		      byte[]----->avro-data	
producer----------------------------------->test-topic---------------------------->consumer
	AvroSerializer			AvroDeserailzer

First time , when you send the avro data , the schema is registered in the schema registry and it is assigned an id. Subsequent times, the producer and consumer can refer the schema using the id itself and does not need to refer the entire schema definition. The schema registry by default runs at port number 8081, Avro schema file is saved with the extension .avsc.

Start Zookeeper


command to start confluent ./confluent start


check the server status