Monitoring Docker Containers with Metricbeat, Elasticsearch, and Kibana

Information about Monitoring Docker Containers with Metricbeat, Elasticsearch, and Kibana

Published on February 18, 2019

Author: Qbox-io



1. Monitoring Docker Containers With Metricbeat, Elasticsearch, and Kibana

2. In this presentation, you’ll learn how use Metricbeat to monitor bare Docker containers and shipping container data to Elasticsearch and Kibana.

3. Prerequisites Examples in this tutorial were tested in the following environment: • Ubuntu 16.04 (Xenial Xerus) • Metricbeat 6.3.2 download from the apt repository • Docker 18.03.1-ce

4. Prerequisites NOTE: We assume that you already have a working Docker environment on your system and a few containers running. If not, see the official Docker installation guide and learn how to run Docker containers as daemons. You’ll need to have at least one container running in Docker to ship some useful data to Elasticsearch and Kibana.

5. Install Metricbeat wget -qO - elasticsearch | sudo apt-key add - To install Metricbeat, you first need to add Elastic’s signing key used to verify the downloaded package: Next, add the Elastic repository to your repository source list: echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

6. Install Metricbeat sudo apt-get update && sudo apt-get install metricbeat Finally, update the repos on your system and install Metricbeat using apt-get:

7. Metricbeat General Configuration To run Metricbeat, you should configure input (metrics sources like Docker), output (a remote service or database to send these metrics to), and various modules if needed. This configuration is located in the inside the Metricbeat folder. Take a look at the edits we've made: metricbeat.yml

8. #================ Modules configuration ============================ metricbeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes reload.period: 10s #==========Elasticsearch template setting ========================== setup.template.settings: index.number_of_shards: 1 index.codec: best_compression #_source.enabled: false #===================Dashboards ===================================== # These settings control loading the sample dashboards to the Kibana #index. Loading the dashboards is disabled by default and can be # enabled either by setting the options here, or by using the `-#setup` CLI flag or the `setup` command. setup.dashboards.enabled: true #=======================Kibana ===================================== # Starting with Beats version 6.0.0, the dashboards are loaded via #the Kibana API. This requires a Kibana endpoint configuration. "YOUR_KIBANA_HOST" setup.kibana.protocol: "https" setup.kibana.username: "YOUR_KIBANA_USERNAME" setup.kibana.password: "YOUR_KIBANA_PASSWORD" #====================Outputs ===================================== # Configure what output to use when sending the data collected by # the beat. #-------------------Elasticsearch output --------------------------- output.elasticsearch: hosts: ["YOUR_ELASTICSEARCH_HOST"] username: "YOUR_ELASTICSEARCH_USERNAME" password: "YOUR_ELASTICSEARCH_PASSWORD"

9. — we load module configurations from external files to keep things isolated. All module configuration files are located under the folder so to target them we used glob pattern. We have also enabled config reloading. Metricbeat will periodically monitor our configuration files, and, if any changes are detected, it will reload the entire configuration.   Metricbeat General Configuration metricbeat.config.modules.path /modules.d/ *.yml There are several configuration options to pay attention to:

10. specifies the index template for Metricbeat. Our Metricbeat index will have 1 shard and will be compressed using type based on high compression ratio.  Metricbeat General Configuration setup.template.setting best_compression setup.dashboards.enabled — we will be loading Kibana example dashboards for Metricbeat. These dashboards include visualization and searches examples for our metrics.

11. — for the dashboards to work, we need to specify the Kibana endpoint. You'll need to enter the URL of your Kibana host and any credentials (username/ password) if needed. Metricbeat General Configuration setup.kibana output.elasticsearch — specifies the output to which we send Metricbeat metrics. We are using Elasticsearch, so you'll need to provide Elasticsearch host, protocol, and credentials if needed.

12. Metricbeat Docker Module To fetch metrics from Docker containers, we are going to use Metricbeat Docker module. It comes with a number of default metricsets we, such as container cpu diskio healthcheck info memory network&, , , , ,

13. Metricbeat Docker Module First, we need to manually enable the Docker module because we load external configuration files into our general configuration. The default Docker module configuration sits in the directory. We can enable or disable any module configuration under by running commands. For example, to enable the  config in the directory, you can run: modules.d modules.d modules enable modules disable docker modules.d /metricbeat modules enable docker

14. Metricbeat Docker Module Now, as the module is enabled, let’s tweak its configuration. This is how it looks like in the file:docker.yml metricsets: - "container" - "cpu" - "diskio" - "healthcheck" - "info" - "image" - "memory" - "network" hosts: ["unix:///var/run/docker.sock"] period: 10s enabled: true

15. Metricbeat Docker Module This is a minimal configuration suffice to get Metricbeat going. We have specified 8 metricsets including “image” metricset not included by default. Also, your Docker module needs access to the Docker daemon.  By default, Docker listens on the Unix socket We can use this socket to communicate with the daemon from within a container. Through this endpoint, Docker exposes the Docker API which can be used to get a stream of all events and statistics generated by Docker. "unix:///var/run/docker.sock"

16. Metricbeat Docker Module The next important configuration field we need to mention is This field defines how often Metricbeat accesses the Docker API. According to the official Metricbeat documentation, it is strongly recommended to run a Docker module with a period of at least 3 seconds or longer. That is because the request to Docker API takes up to 2 seconds itself, so specifying less than 3 seconds can cause request timeouts and no data returned to those requests. period

17. Metricbeat Docker Module Great! Now, everything is ready to run Metricbeat. One last advice is to have your Metricbeat instance as close to Docker as possible (preferably on the same host) to minimize network latency. On Linux, you can run Metricbeat in the shell specifying the config file as a parameter. sudo ./metricbeat -e -c metricbeat.yml

18. Metricbeat Docker Module Alternatively, you can start Metricbeat as a service to run at startup: sudo service metricbeat start Almost immediately after being started, Metricbeat will begin sending Docker metrics to Elasticsearch index. You can verify this by curling your Elasticsearch host: curl -XGET 'localhost:9200/_cat/indices?v&pretty' health status index uuid pri rep docs.count docs.deleted store.size yellow open metricbeat-6.3.2-2018.08.09 BikpOgqQR-pU_SuKrm5vw 1 1 2374 0 1.4mb 1.3mb

19. Metricbeat Docker Module As you see, our index was created! Now, you can create an index pattern for this index in Kibana and access metrics directly from the Kibana dashboard. Log in to your Kibana dashboard and follow the simple steps for creating a new Index Pattern under Management -> Index Patterns -> Create Index Pattern. If everything is ok, you’ll see the index mapping similar to the screenshot in the next slide.

20. Metricbeat Docker Module You can see that Metricbeat created fields for various Docker module metricsets such as container, CPU, etc. You can also find batches of metrics shipped to Elasticsearch by clicking on Discovery tab in Kibana dashboard. You should see something like this:

21. Metricbeat Docker Module Awesome! As you remember, we loaded Kibana dashboards, so you can access example visualizations of the Metricbeat data in Kibana.

22. Metricbeat Docker Module Pretty cool, isn’t it? In the dashboard, we have the statistics of running containers as well some data about their memory and CPU usage. You can experiment with Docker data by creating your custom visualizations. Check out our tutorial about building visualizations for Metricbeat data in Kibana.

23. Conclusion That’s it! You have learned how to monitor Docker containers using Metricbeat, Elasticsearch, and Kibana. Metricbeat Docker module exposes key metrics and events provided by the Docker API. Making them accessible for subsequent processing and analysis in Kibana is really a great monitoring solution that requires minimal configuration.

24. Conclusion Installing Metricbeat and configuring Docker module should not take more than half an hour. Once the Docker container data is available in Elasticsearch, you can use powerful Kibana visualization tools like Visual Builder and Timelion to analyze your Docker containers. This presentation originally appeared as an article on Qbox blog: Monitoring Docker Containers With Metricbeat, Elasticsearch, and Kibana

25. Resources Read the Qbox blog Watch our YouTube channel Read our Medium tutorials We have a wealth of resources to help you with your Elasticsearch journey. Here are just a few sites for you check out:

26. Visit the Qbox website to learn about our hosted Elasticsearch service

#the presentations

Related presentations