Apache Kafka Interview Questions and Answers
Freshers / Beginner level questions & answers
Ques 1. What is Apache Kafka?
Apache Kafka is a publish-subscribe open source message broker application. This messaging application was coded in “Scala”. Basically, this project was started by the Apache software. Kafka’s design pattern is mainly based on the transactional logs design.
Ques 2. Enlist the several components in Kafka.
- Topic: Kafka Topic is the bunch or a collection of messages.
- Producer: In Kafka, Producers issue communications as well as publishes messages to a Kafka topic.
- Consumer: Kafka Consumers subscribes to a topic(s) and also reads and processes messages from the topic(s).
- Brokers: While it comes to manage storage of messages in the topic(s) we use Kafka Brokers.
Ques 3. What is a Consumer Group?
The concept of Consumer Groups is exclusive to Apache Kafka. Basically, every Kafka consumer group consists of one or more consumers that jointly consume a set of subscribed topics.
Ques 4. What is the role of the ZooKeeper in Kafka?
Apache Kafka is a distributed system is built to use Zookeeper. Although, Zookeeper’s main role here is to build coordination between different nodes in a cluster. However, we also use Zookeeper to recover from previously committed offset if any node fails because it works as periodically commit offset.
Ques 5. Is it possible to use Kafka without ZooKeeper?
No. It is impossible to bypass Zookeeper and connect directly to the Kafka server. If somehow, ZooKeeper is down, then it is impossible to service any client request.
Ques 6. What do you know about Partition in Kafka?
In every Kafka broker, there are few partitions available. And, here each partition in Kafka can be either a leader or a replica of a topic.
Ques 7. Why is Kafka technology significant to use?
There are some advantages of Kafka, which makes it significant to use:
- High-throughput: We do not need any large hardware in Kafka, because it is capable of handling high-velocity and high-volume data. Moreover, it can also support message throughput of thousands of messages per second.
- Low Latency: Kafka can easily handle these messages with the very low latency of the range of milliseconds, demanded by most of the new use cases.
- Fault-Tolerant: Kafka is resistant to node/machine failure within a cluster.
- Durability: As Kafka supports messages replication, so, messages are never lost. It is one of the reasons behind durability.
- Scalability: Kafka can be scaled-out, without incurring any downtime on the fly by adding additional nodes.
Ques 8. What are main APIs of Kafka?
- Producer API
- Consumer API
- Streams API
- Connector API
Ques 9. What are consumers or users?
Mainly, Kafka Consumer subscribes to a topic(s), and also reads and processes messages from the topic(s). Moreover, with a consumer group name, Consumers label themselves.
Ques 10. What are the types of traditional method of message transfer?
- Queuing: It is a method in which a pool of consumers may read a message from the server and each message goes to one of them.
- Publish-Subscribe: Whereas in Publish-Subscribe, messages are broadcasted to all consumers.
Ques 11. Describe partitioning key in apache kafka.
Its role is to specify the target divider of the memo within the producer. Usually, a hash-oriented divider concludes the divider ID according to the given factors. Consumers also use tailored partitions.
Intermediate / 1 to 5 years experienced level questions & answers
Ques 12. Explain the role of the offset.
There is a sequential ID number given to the messages in the partitions what we call, an offset. So, to identify each message in the partition uniquely, we use these offsets.
Ques 13. Explain the concept of Leader and Follower.
In every partition of Kafka, there is one server which acts as the Leader, and none or more servers plays the role as a Followers.
Ques 14. What is the process for starting a Kafka server?
- In order to start the ZooKeeper server: > bin/zookeeper-server-start.sh config/zookeeper.properties
- Next, to start the Kafka server: > bin/kafka-server-start.sh config/server.properties
Ques 15. Explain the role of the Kafka Producer API.
An API which permits an application to publish a stream of records to one or more Kafka topics is what we call Producer API.
Ques 16. What can you do with Kafka?
It can perform in several ways, such as:
- In order to transmit data between two systems, we can build a real-time stream of data pipelines with it.
- Also, we can build a real-time streaming platform with Kafka, that can actually react to the data.
Ques 17. What does ISR stand in Kafka environment?
ISR refers to In Sync Replicas. These are generally classified as a set of message replicas which are synced to be leaders.
Ques 18. What is the role of Consumer API?
An API which permits an application to subscribe to one or more topics and also to process the stream of records produced to them is what we call Consumer API.
Ques 19. Explain the role of Streams API in Kafka?
An API which permits an application to act as a stream processor, and also consuming an input stream from one or more topics and producing an output stream to one or more output topics, moreover, transforming the input streams to output streams effectively, is what we call Streams API.
Ques 20. What is the role of Connector API?
An API which permits to run as well as build the reusable producers or consumers which connect Kafka topics to existing applications or data systems is what we call the Connector API.
Ques 21. Why Should we use Apache Kafka Cluster?
- It is possible to track web activities just by storing/sending the events for real-time processes.
- Through this, we can Alert as well as report the operational metrics.
- Also, we can transform data into the standard format.
- Moreover, it allows continuous processing of streaming data to the topics.
Ques 22. What is Data Log in Kafka?
As we know, messages are retained for a considerable amount of time in Kafka. Moreover, there is flexibility for consumers that they can read as per their convenience. Although, there is a possible case that if Kafka is configured to keep messages for 24 hours and possibly that time consumer is down for time greater than 24 hours, then the consumer may lose those messages. However, still, we can read those messages from last known offset, but only at a condition that the downtime on part of the consumer is just 60 minutes. Moreover, on what consumers are reading from a topic Kafka doesn’t keep state.
Ques 23. Explain how to Tune Kafka for Optimal Performance.
- Tuning Kafka Producers
- Kafka Brokers Tuning
- Tuning Kafka Consumers
Ques 24. State Disadvantages of Apache Kafka.
- No Complete Set of Monitoring Tools.
- Issues with Message Tweaking.
- Not support wildcard topic selection.
- Lack of Pace.
Ques 25. Enlist all Apache Kafka Operations.
Apache Kafka Operations are:
- Addition and Deletion of Kafka Topics
- How to modify the Kafka Topics
- Distinguished Turnoff
- Mirroring Data between Kafka Clusters
- Finding the position of the Consumer
- Expanding Your Kafka Cluster
- Migration of Data Automatically
- Retiring Servers
- Datacenters
Ques 26. Explain Apache Kafka Use Cases?
- Kafka Metrics: It is possible to use Kafka for operational monitoring data. Also, to produce centralized feeds of operational data, it involves aggregating statistics from distributed applications.
- Kafka Log Aggregation: Moreover, to gather logs from multiple services across an organization.
- Stream Processing: While stream processing, Kafka’s strong durability is very useful.
Ques 27. What role does ZooKeeper play in a cluster of Kafka?
Apache ZooKeeper acts as a distributed, open-source configuration and synchronization service, along with being a naming registry for distributed applications. It keeps track of the status of the Kafka cluster nodes, as well as of Kafka topics, partitions, etc.
Since the data is divided across collections of nodes within ZooKeeper, it exhibits high availability and consistency. When a node fails, ZooKeeper performs an instant failover migration.
ZooKeeper is used in Kafka for managing service discovery for Kafka brokers, which form the cluster. ZooKeeper communicates with Kafka when a new broker joins, when a broker dies, when a topic gets removed, or when a topic is added so that each node in the cluster knows about these changes. Thus, it provides an in-sync view of the Kafka cluster configuration.
Ques 28. Elaborate the architecture of Kafka.
In Kafka, a cluster contains multiple brokers since it is a distributed system. Topic in the system will get divided into multiple partitions, and each broker stores one or more of those partitions so that multiple producers and consumers can publish and retrieve messages at the same time.
Experienced / Expert level questions & answers
Ques 29. What ensures load balancing of the server in Kafka?
As the main role of the Leader is to perform the task of all read and write requests for the partition, whereas Followers passively replicate the leader. Hence, at the time of Leader failing, one of the Followers takeover the role of the Leader. Basically, this entire process ensures load balancing of the servers.
Ques 30. What roles do Replicas and the ISR play?
- Basically, a list of nodes that replicate the log is Replicas. Especially, for a particular partition. However, they are irrespective of whether they play the role of the Leader.
- In addition, ISR refers to In-Sync Replicas. On defining ISR, it is a set of message replicas that are synced to the leaders.
Ques 31. Why are Replications critical in Kafka?
Because of Replication, we can be sure that published messages are not lost and can be consumed in the event of any machine error, program error or frequent software upgrades.
Ques 32. If a Replica stays out of the ISR for a long time, what does it signify?
Simply, it implies that the Follower cannot fetch data as fast as data accumulated by the Leader.
Ques 33. In the Producer, when does QueueFullException occur?
Whenever the Kafka Producer attempts to send messages at a pace that the Broker cannot handle at that time QueueFullException typically occurs. However, to collaboratively handle the increased load, users will need to add enough brokers, since the Producer doesn’t block.
Ques 34. What is the main difference between Kafka and Flume?
Ques 35. Is Apache Kafka is a distributed streaming platform and what you can do with it?
- To push records easily
- Also, can store a lot of records without giving any storage problems
- Moreover, it can process the records as they come in.
Ques 36. What is the purpose of retention period in Kafka cluster?
However, retention period retains all the published records within the Kafka cluster. It doesn’t check whether they have been consumed or not. Moreover, the records can be discarded by using a configuration setting for the retention period. And, it results as it can free up some space.
Ques 37. What is Geo-Replication in Kafka?
For our cluster, Kafka MirrorMaker offers geo-replication. Basically, messages are replicated across multiple data centers or cloud regions, with MirrorMaker. So, it can be used in active/passive scenarios for backup and recovery; or also to place data closer to our users, or support data locality requirements.
Ques 38. Explain Multi-tenancy in Kafka?
We can easily deploy Kafka as a multi-tenant solution. However, by configuring which topics can produce or consume data, Multi-tenancy is enabled. Also, it provides operations support for quotas.
Most helpful rated by users:
- Explain the concept of Leader and Follower.
- Enlist the several components in Kafka.
- What are main APIs of Kafka?