Kafka Monitoring Tools

Desgin docs for Kafka Monitoring Tools

  5 minute read  

Some of these date back to older versions but efforts are made to keep the most important - sometimes :)

KAFKA MONITORING TOOLS - RESEARCH RESULTS

  • Commercial feature, only trial version for free
  • Out of the box UI
  • Managing and monitoring Kafka cluster (including view consumer offset)
  • Possibility to set up alerts
  • Detailed documentation, lots of tutorials, blog articles and a wide community
  • All-in-one solution with additional features through Confluent Platform/Cloud
  • Commercial feature, only trial version for free
  • Out of the box UI
  • Deliver monitoring of Kafka data pipelines
  • Managing and monitoring Kafka cluster (including view consumer offset)
  • Possibility to set up alerts
  • Smaller community, fewer articles and tutorials around Lenses compared to the Control Center
  • Commercial feature, only trial version for free
  • ChatOps integrations
  • Out of the box UI
  • Built-in anomaly detection, threshold, and heartbeat alerts
  • Managing and monitoring Kafka cluster (including view consumer offset)
  • Possibility to set up alerts
  • Commercial feature, only trial version for free
  • Out of the box Kafka monitoring dashboards
  • Monitoring tool (including view consumer offset). Displays key metrics for Kafka brokers, producers, consumers and Apache Zookeeper. Less focused on cluster state
  • Possibility to set up alerts
  • Commercial feature, only trial version for free
  • Less rich monitoring tool compared to Confluent, Lenses and Datadog but is very convenient for companies that are already customers of Cloudera and need their monitoring mechanisms under the same platform
  • Commercial feature, only trial version for free
  • Out of the box UI
  • Monitoring tool (including view consumer offset)
  • Poor documentation
  • In latest changelogs, only support for kafka 2.1 mentioned
  • Some of opensource projects looks much more better that this one
  • Commercial feature, only trial version for free
  • Out of the box UI
  • Focused on filtering the messages within the topics and the creation of custom views
  • No possibility to set up alerts
  • Focuses more on business monitoring than on technical monitoring like Control Center or Lenses
  • KaDeck could be used in addition to the other monitoring tools
  • Opensource project, Apache-2.0 License
  • Managing and monitoring Kafka cluster (including view consumer offset)
  • Out of the box UI
  • No possibility to set up alerts
  • Opensource project, BSD 2-Clause "Simplified" License
  • Managing and monitoring Kafka cluster (not possible to view consumer offset :warning:)
  • Possible to track resource utilization for brokers, topics, and partitions, query cluster state, to view the status of partitions, to monitor server capacity (i.e. CPU, network IO, etc.)
  • Anomaly Detection and self-healing and rebalancing
  • No possibility to set up alerts
  • UI as seperated component link
  • It can use metrics reporter from LinkedIn (necessary to add jar file to kafka lib directory) but it is also possible to uses Prometheus for metric aggregation
  • Opensource project, Apache-2.0 License
  • Provides consumer lag checking as a service without the need for specifying thresholds. It monitors committed offsets for all consumers and calculates the status of those consumers on demand
  • It does not monitor anything related to the health of the brokers
  • Possibility to set up alerts
  • Opensource project, Apache-2.0 License, reboot of Kafdrop 2.x
  • Monitoring tool (including view consumer offset)
  • Out of the box UI
  • No possibility to set up alerts
  • Opensource project, Apache-2.0 License
  • Kafka monitor is a framework to implement and execute long-running Kafka system tests in a real cluster
  • It plays a role as a passive observer and reports what it observes (broker availability, produce/consume latency, etc) by emitting metrics. In other words, it pretends to be a Kafka user and keeps reporting metrics from the user's PoV
  • It is more a load generation and reporting tool
  • UI does not exist
  • No possibility to set up alerts

13. OTHERS

Things like on the list below are there as well, but usually such smaller projects and have little or no development activity:

14. CONCLUSIONS

Currently in LambdaStack monitoring and getting metrics from Kafka are based on:

In real scenarios, based on some use cases and opinions from internal teams:

  • Kafka Exporter is used in order to get consumer offset and lag
  • JMX Exporter is used in order to get some standard broker's metrics such as cpu, memory utilization and so on

If it is possible to pay for a commercial license, Confluent, Lenses and Sematext offer more rich functionality compared to the other monitoring tools and they are very similar.

As far as the open source project is considered:

  • LinkedIn Cruise Control looks like the winner. Provides not only managing and monitoring Kafka cluster but also some extra features such as rebalancing, anomaly detection or self-healing
  • Yahoo Cluster Manager looks like good competitor but only for managing and monitoring Kafka cluster. However in compare to Cruise Control, during the installation some issues were met and it was not able to recieve some consumer data and a few issues are already reported in official repository related to the problem link. The project does not have good spirit of open source software at all.
  • LinkedIn Burrow looks like good additional tool for LinkedIn Cruise Control when it comes to consumer lag checking service and can be used instead of Kafka exporter plugin which cause some outstanding issues