What is ELK?
The Elastic Stack (also known as ELK) is a robust solution for search, logging and data analysis. The ELK Stack consists of three programs that, when used together, act as a fully centralized logging and protocol analysis solution. These programs are ElasticSearch, Logstash and Kibana. ELK is the most widely used open source and protocol management platform in the world.
The ELK-Stack with the Business Bot Platform as follows:
ElasticSearch (ES): is a full-text search engine similar to Solr (both use Apache Lucene as underlying storage) that stores logs and allows you to query log entries. As soon as a log entry is received in ES, the log entry is marked with the timestamp and indexed. Later, the log entry can be retrieved almost immediately thanks to the Lucene Index.
Kibana: performs queries on ElasticSearch and visualizes the logs of the Business Bot platform. The web-based application is responsible for providing easy-to-use dashboard tools.
Logstash: acts as a log router. In addition to the Business Bot platform, various applications (e.g. MySQL, Apache web server) can send their logs to Logstash, where they are filtered, transformed and sent to one or more outputs according to the Logstash configuration file. In order to call it ELK stack, Logstash must pass the logs to ElasticSearch. Thus, Logstash is responsible for collecting the data, transformations such as parsing – with regular expressions – adding fields, formatting as structures such as JSON, etc. and finally sending the data to various destinations such as an ElasticSearch cluster.
Filebeat: is a transporter for forwarding and centralizing log data. Filebeat is installed as an agent on the servers and monitors the specified log files or locations, collects log events, and forwards them to Logstash for indexing.
Log4j: is a fast and flexible framework for logging application debugging messages. With Log4j it is possible to enable logging at runtime without changing the application code. The Log4j package is designed to keep the delivered program code without high performance costs. Logging behavior can be controlled by editing the configuration file without touching the application binary. All log events of the Business Bot Platform are processed within Log4j. This provides you with an easy way to hook into the logs.
Send Chatbot-Logs from Business Bot Platform to ELK-Stack
If you have already installed an ELK stack, only the log4j configuration of the Business Bot platform needs to be adapted. Subsequently, Logstash needs the information which log file they have to look. This is all you need to configure the ELK stack with the Business Bot Platform. To monitor the BBP logs, go to Discover in the Kibana web interface and use the fields to filter the desired log messages.
What are the benefits of the solution?
1 – Sophisticated user interface for data analysis
ELK has an elegant user interface that optimizes data analysis tasks so you can spend time examining and reviewing data without having to figure out how to navigate through the user interface. The only thing you need to worry about is the results of your search.
2 – Data visualization for free
With ELK, you can start exploring your data at the moment of inspiration. Kibana provides out-of-the-box visualizations, including histograms, line charts, pie charts, and more. You can also use the Vega grammar to create custom visualizations.
3 – Expandable source and aggregation
You can configure Logstash to accept data from a variety of sources. Logash is essentially a streamlined ETL pipeline that provides a number of options for filtering and transforming data according to your needs.
The Business Bot platform can be easily connected to the ELK stack so that you can monitor all chatbot logs via the centralized ELK platform. Contact us, we are proud to support you around chatbots in corporate environment.