Comparison of Kafka Monitoring Tools

To answer this question, I investigated four different Kafka monitoring tools and tried to compare them with each other: Confluent Control Center, Lenses, KaDeck and Jaeger. First: There is no “best” tool, it always depends on the type of your project and the focus of needed functionalities. All of them provide a wide range of features, so in this blog post, I am trying to evaluate the advantages and disadvantages which might help you with your decision. At the end of this article, you will find a table shortly summarizing the most important points.

If you are not familiar with Apache Kafka, you are welcome to read our other blog posts as an introduction first: Kafka 101 Series – Part 1: Introduction to Kafka or Kafka 101 Series – Part 2: Stream Processing.
For a quick start, you are welcome to test our implementation and the different tools on Github. Note that you need to get a test license beforehand to be able to use Lenses.

Confluent Control Center

The commercial version of the Confluent Platform comes along with the Control Center. Unfortunately, it is not included in the community version of the Confluent Platform, but a 30-day-trial version is available.
Integration is straightforward when Kafka was set up with Docker: simply download the official image and include the configuration in your docker-compose file. Besides Docker, other local installations are supported while running the system on cloud is only supported within Confluent Cloud and not Confluent Platform.

The tool gives you a quick overview of your system: the number of healthy and unhealthy clusters and the number of brokers, partitions, topics and connected services in each cluster. It is possible to inspect each cluster to get more details, like the production and consumption of your brokers and topics, disk usage and system load. Most of this information can not only be displayed in plain numbers, but also be visualized in a time series diagram.
Role-based access, authorization and other security options can be configured but are not provided within the user interface by default. You first have to enable this functionality via the command line to then manage the users through the Control Center.

Advantages:

• quick but detailed overview of all important cluster data in numbers and as time series diagram
• the creation of stream processors is possible with KSQL
• detailed documentation, lots of tutorials, blog articles and a wide community
• other enterprise features of the Confluent Platform can be used as well (e.g. MQTT Proxy, Schema Validation)
• multi-cluster management is enabled by default
• configuration of alerts, e.g. when your broker’s production/consumption reaches unusual numbers

Disadvantages:

• role-based access control for Control Center has to be set up via the command line first
• enterprise version needed

Screenshot: Confluent Control Center

Lenses

Lenses is very similar to the Confluent Control Center as there are only small differences between the functionalities of them. Lenses was developed by Lenses.io and was formerly named Landoop.  A 30-day-trial version is available and afterward, the price depends on the size of your developer team.

Some functionalities are named differently than in the Confluent tool or one tool uses another kind of visualization compared to the other – but basically both provide the same features.  While the Control Center is using the open-source streaming SQL engine KSQL, Lenses uses “Lenses SQL” to enable the creation of stream processors.
When starting the application you will quickly notice that a lot can be done and configured through the user interface, like the creation of user accounts, the configuration of role-based access and the management of quotas for users and clients. With the first startup of the application, you simply log in with the default credentials and can then directly manage your user groups and access rights. This can be helpful especially if you do not feel comfortable with the command line or want to quickly test around a bit.

During my research, I also noticed that it seems like there are a smaller community, fewer articles and tutorials around Lenses compared to the Control Center. This could be a drawback at first, but especially when it is about security, the Control Center is more dependent on other components of the Kafka ecosystem. This is the reason why the setup of the Control Center could seem more “complicated” (and therefore more tutorials and help might be required). Retrospectively, Lenses was easy to use and larger documentation is rarely needed

Advantages:

• quick but detailed overview of all important cluster data in numbers and as time series diagram
• creation of stream processors with Lenses SQL
• role-based access, authentication and authorization can easily and directly be set up via the user interface
• low price for only one developer or a small team
• Kafka quotas can be configured via the user interface
• configuration of alerts, e.g. when your broker’s production/consumption reaches unusual numbers

Disadvantages:

• multi-cluster management is not enabled by default
• enterprise version needed

Screenshot: Lenses

KaDeck

KaDeck was developed by Xeotek and has its focus on filtering the messages within the topics and the creation of custom views. In contrast to the other two discussed monitoring tools, KaDeck offers a free full-featured version for single developers or small teams. It also offers some additional functionalities, that are not included in the other two options.

KaDeck provides the possibility to filter data without any knowledge of SQL. Those filters can then be saved as a view to being reproducible quickly and without the need to directly edit the topic configuration. In case you are searching for messages that contain a certain substring, the option for asterisk- and regex-search is offered.  While the Control Center and Lenses are using SQL queries to process data, KaDeck offers a JavaScript processor instead. This can be an advantage if you prefer JavaScript to SQL or want to create more complicated queries.

The tool does not offer the possibility to set up alerts and get notified when e.g. the production or consumption of your brokers reaches numbers which should be investigated. The reason for this could be that KaDeck focuses more on business monitoring than on technical monitoring like Control Center or Lenses do. Therefore, KaDeck could be used in addition to the other two options to cover all use cases.

Advantages:

• free full-featured version for single developers and small teams
• custom creation of multiple views to easily re-apply filters
• almost no or no SQL-knowledge needed
• transformation and filtering of records with JavaScript

Disadvantages:

• no possibility to set up alerts

Screenshot: KaDeck

Jaeger

Jaeger is an end-to-end distributed tracing tool and has a completely different focus when it is about the monitoring of your Kafka cluster. It was initially developed by Uber and is available as an open-source project.

Compared to the other discussed tools, Jaeger does not provide the possibility to ingest your brokers, topics or records. In return, it offers the functionality to monitor your microservices that are connected to your cluster to better understand the data flow between them and to identify potential bottlenecks. In Jaeger, one trace is subdivided into spans, each containing one executed operation with its start and end time. It is also possible to compare traces with each other to further analyze differences between them. Because of those completely different strengths, it could be a good idea to run Jaeger in addition to another monitoring tool when your system scales and the services add up. Especially when you are already using Jaeger to monitor your microservices, it will be a suitable complement to also analyze Kafka with it.

Advantages:

• open-source, completely free
• provides a “microservice-point-of-view” for troubleshooting or bottlenecks
• comparison of traces and operations

Disadvantages:

• probably another monitoring tool, in addition, makes sense to cover all needs

Screenshot: Jaeger

The Kafka Monitoring Tools in Summary

Conclusion

As already mentioned in the beginning, the choice of the best-suited monitoring tool always depends on the type of project, the time you have for setup, the money you are prepared to invest, the users and the needed functionalities.  On the one hand, the Confluent Control Center and Lenses focus more on a high-level overview of your clusters and brokers. On the other hand, KaDeck offers functionalities to get a better insight into your actual data in your topics. Additionally, Jaeger Tracing provides another point of view of your architecture for troubleshooting and system improvements. Therefore, a combination of two or maybe more tools seems reasonable to highlight different aspects of your system.

Hopefully, the above evaluation will help you to choose the right Kafka monitoring tool for your project or maybe to consider alternatives. Feel free to leave a comment: what were your experiences with the discussed or other tools? Do you have recommendations or alternatives?